The 'unknown error' is most likely because Hawkular Metrics cannot resolve the
'hawkular-cassandra' hostname.
You should be able to exec into the Hawkular Metrics pod before it get shutdown
and try to dig, curl or ping the 'hawkular-cassandra' hostname. If you get an
error message here about the hostname not being resolveable then that is the
problem.
----- Original Message -----
> From: "Matt Wringe" <mwri...@redhat.com>
> To: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> Cc: users@lists.openshift.redhat.com
> Sent: Wednesday, June 15, 2016 11:36:15 AM
> Subject: Re: Metrics deployment
>
> Can you please list the secrets used and the deployment options used when
> deploying metrics?
>
> If you redeploy everything, does the error still exist?
>
> I am trying to check a few error conditions to see if I can reproduce the
> 'unknown error' here, but not displaying the full error is a bug in Hawkular
> Metrics which needs to be fixed.
>
> ----- Original Message -----
> > From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > To: "Matt Wringe" <mwri...@redhat.com>
> > Cc: users@lists.openshift.redhat.com
> > Sent: Tuesday, June 14, 2016 7:14:46 PM
> > Subject: Re: Metrics deployment
> >
> > 18:34:03,414 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:03,532 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
> > error
> > 18:34:03,533 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying
> > connecting
> > to Cassandra cluster in [1]s...
> > 18:34:04,534 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:04,534 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > 18:34:04,535 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying
> > connecting
> > to Cassandra cluster in [2]s...
> > 18:34:06,535 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:06,535 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > 18:34:06,536 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying
> > connecting
> > to Cassandra cluster in [3]s...
> > 18:34:09,536 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:09,537 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > 18:34:09,537 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying
> > connecting
> > to Cassandra cluster in [4]s...
> > 18:34:13,537 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:13,653 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra: unknown
> > error
> > 18:34:13,653 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying
> > connecting
> > to Cassandra cluster in [1]s...
> > 18:34:14,654 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> > 18:34:14,654 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to
> > Cassandra cluster - assuming its not up yet: hawkular-cassandra
> > 18:34:14,654 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying
> > connecting to Cassandra cluster in [2]s...
> > 18:34:16,655 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics
> > service
> >
> >
> >
> > --
> > Srinivas Kotaru
> >
> > On 6/14/16, 4:02 PM, "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > wrote:
> >
> > >9042 firewall created across all nodes to node communication. But issue
> > >still persist. Port 53 already open between all nodes to masters.
> > >
> > >Almost dead struck, appreciated any pointers to get fresh thoughts
> > >
> > >--
> > >Srinivas Kotaru
> > >
> > >On 6/14/16, 3:16 PM, "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > >wrote:
> > >
> > >>Ok. Let me open the port 9043. This port should be open between node to
> > >>node right?
> > >>
> > >>
> > >>--
> > >>Srinivas Kotaru
> > >>
> > >>On 6/14/16, 3:02 PM, "Matt Wringe" <mwri...@redhat.com> wrote:
> > >>
> > >>>----- Original Message -----
> > >>>> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > >>>> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> Cc: users@lists.openshift.redhat.com
> > >>>> Sent: Tuesday, June 14, 2016 5:40:36 PM
> > >>>> Subject: Re: Metrics deployment
> > >>>>
> > >>>> Matt.
> > >>>>
> > >>>> Sure, let us figure it out Hawkular side. Am here pasting 2 logs
> > >>>>
> > >>>> 1. oc logs –f
> > >>>> 2. cat /opt/eap/standalone/log/server.log
> > >>>
> > >>>Hmm, its getting an 'unknown error' when trying to connect to Cassandra,
> > >>>which doesn't really tell us anything :/
> > >>>
> > >>>The port that Hawkular Metrics uses to connect to Cassandra is 9042, you
> > >>>may also want to make sure that the DNS port is also open.
> > >>>
> > >>>>
> > >>>> Srinivas Kotaru
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Srinivas Kotaru
> > >>>>
> > >>>> On 6/14/16, 2:28 PM, "Matt Wringe" <mwri...@redhat.com> wrote:
> > >>>>
> > >>>> >
> > >>>> >
> > >>>> >----- Original Message -----
> > >>>> >> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > >>>> >> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> Cc: users@lists.openshift.redhat.com
> > >>>> >> Sent: Tuesday, June 14, 2016 4:44:04 PM
> > >>>> >> Subject: Re: Metrics deployment
> > >>>> >>
> > >>>> >>
> > >>>> >> I’m still struck with this issue. It is kind of chicken and egg
> > >>>> >> problem.
> > >>>> >> Heapster health probes failing since it is waiting for hawkular to
> > >>>> >> start.
> > >>>> >> Hawkular health probes are failing since it is unable to connect
> > >>>> >> Casandra.
> > >>>> >> Cansandra health probe also failing.
> > >>>> >
> > >>>> >The first step is to get Cassandra running, ignore Hawkular Metrics
> > >>>> >and
> > >>>> >Heapster until you have Cassandra running properly. Without Cassandra
> > >>>> >being
> > >>>> >able to run, those other components will not fully start.
> > >>>> >
> > >>>> >>
> > >>>> >> @Matt: Internal DNS looks ups working. I’m able to create apps,
> > >>>> >> build and
> > >>>> >> deploy code. Router and registry components also working as
> > >>>> >> expected
> > >>>> >>
> > >>>> >>
> > >>>> >> oc get pods
> > >>>> >> NAME READY STATUS RESTARTS AGE
> > >>>> >> hawkular-cassandra-1-mxd2m 1/1 Running 0 1h
> > >>>> >> hawkular-metrics-gvp9k 0/1 Running 4 11m
> > >>>> >> heapster-uleul 0/1 Running 4 11m
> > >>>> >> metrics-deployer-2z75w 0/1 Completed 0 1h
> > >>>> >
> > >>>> >Cassandra being in "READY 1/1" means that it started up properly. So
> > >>>> >Cassandra is running. Why do you think its not running? Things like
> > >>>> >readiness probes are expected to fail until the pod is ready. Just
> > >>>> >because
> > >>>> >there is a failure in the events doesn't mean its an error condition.
> > >>>> >
> > >>>> >Hawkular Metrics is not running here. So lets figure out why. Ignore
> > >>>> >Heapster until Hawkular Metrics is started.
> > >>>> >
> > >>>> >>
> > >>>> >> heapster events:
> > >>>> >> ------------------------
> > >>>> >>
> > >>>> >>
> > >>>> >> Events:
> > >>>> >> FirstSeen LastSeen Count From
> > >>>> >> SubobjectPath Type Reason
> > >>>> >> Message
> > >>>> >> --------- -------- ----- ----
> > >>>> >> ------------- -------- ------
> > >>>> >> -------
> > >>>> >> 1m 1m 1 {default-scheduler }
> > >>>> >> Normal
> > >>>> >> Scheduled
> > >>>> >> Successfully
> > >>>> >> assigned heapster-uleul to l3inpn-id2-004.cisco.com
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{heapster}
> > >>>> >> Normal Pulling pulling image
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-heapster:latest"
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{heapster}
> > >>>> >> Normal Pulled Successfully pulled image
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-heapster:latest"
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{heapster}
> > >>>> >> Normal Created Created container with docker
> > >>>> >> id a22bb9a246ca
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{heapster}
> > >>>> >> Normal Started Started container with docker
> > >>>> >> id a22bb9a246ca
> > >>>> >> 1m 5s 10 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{heapster}
> > >>>> >> Warning Unhealthy Readiness probe failed: The
> > >>>> >> heapster process
> > >>>> >> is
> > >>>> >> not
> > >>>> >> yet
> > >>>> >> started, it is waiting for the Hawkular Metrics to start.
> > >>>> >
> > >>>> >Readiness probe failing here is expected, and we know why from the
> > >>>> >error
> > >>>> >message:
> > >>>> >"Readiness probe failed: The heapster process is not yet started, it
> > >>>> >is
> > >>>> >waiting for the Hawkular Metrics to start."
> > >>>> >
> > >>>> >Once Hawkular Metrics is running then Heapster should automatically
> > >>>> >start
> > >>>> >functioning.
> > >>>> >
> > >>>> >>
> > >>>> >> Hawkular events :
> > >>>> >> ===============
> > >>>> >>
> > >>>> >> Events:
> > >>>> >> FirstSeen LastSeen Count From
> > >>>> >> SubobjectPath Type
> > >>>> >> Reason
> > >>>> >> Message
> > >>>> >> --------- -------- ----- ----
> > >>>> >> ------------- --------
> > >>>> >> ------
> > >>>> >> -------
> > >>>> >> 1m 1m 1 {default-scheduler }
> > >>>> >> Normal
> > >>>> >> Scheduled
> > >>>> >> Successfully
> > >>>> >> assigned hawkular-metrics-gvp9k to l3inpn-id2-004.cisco.com
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Normal Pulling
> > >>>> >> pulling image
> > >>>> >>
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Normal Pulled
> > >>>> >> Successfully
> > >>>> >> pulled
> > >>>> >> image
> > >>>> >>
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-hawkular-metrics:latest"
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Normal Created
> > >>>> >> Created
> > >>>> >> container
> > >>>> >> with
> > >>>> >> docker id 6e32459624d7
> > >>>> >> 1m 1m 1 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Normal Started
> > >>>> >> Started
> > >>>> >> container
> > >>>> >> with
> > >>>> >> docker id 6e32459624d7
> > >>>> >> 1m 57s 4 {kubelet
> > >>>> >> l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Warning
> > >>>> >> Unhealthy Readiness
> > >>>> >> probe
> > >>>> >> failed: Failed to access the status endpoint : <urlopen error
> > >>>> >> [Errno
> > >>>> >> 111]
> > >>>> >> Connection refused>.
> > >>>> >>
> > >>>> >> 46s 7s 5 {kubelet l3inpn-id2-004.cisco.com}
> > >>>> >> spec.containers{hawkular-metrics} Warning Unhealthy
> > >>>> >> Readiness
> > >>>> >> probe
> > >>>> >> failed:
> > >>>> >>
> > >>>> >>
> > >>>> >> Casandra events
> > >>>> >> ==============
> > >>>> >>
> > >>>> >> Events:
> > >>>> >> FirstSeen LastSeen Count From
> > >>>> >> SubobjectPath Type
> > >>>> >> Reason
> > >>>> >> Message
> > >>>> >> --------- -------- ----- ----
> > >>>> >> ------------- --------
> > >>>> >> ------
> > >>>> >> -------
> > >>>> >> 1h 1h 1 {default-scheduler }
> > >>>> >> Normal
> > >>>> >> Scheduled
> > >>>> >> Successfully
> > >>>> >> assigned hawkular-cassandra-1-mxd2m to l3inpn-id2-003.cisco.com
> > >>>> >> 1h 1h 1 {kubelet
> > >>>> >> l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Normal Pulling
> > >>>> >> pulling
> > >>>> >> image
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> > >>>> >> 1h 1h 1 {kubelet
> > >>>> >> l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Normal Pulled
> > >>>> >> Successfully
> > >>>> >> pulled
> > >>>> >> image
> > >>>> >> "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> > >>>> >> 1h 1h 1 {kubelet
> > >>>> >> l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Normal Created
> > >>>> >> Created
> > >>>> >> container
> > >>>> >> with docker id 2a1390edd4f1
> > >>>> >> 1h 1h 1 {kubelet
> > >>>> >> l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Normal Started
> > >>>> >> Started
> > >>>> >> container
> > >>>> >> with docker id 2a1390edd4f1
> > >>>> >> 1h 1h 1 {kubelet
> > >>>> >> l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Warning
> > >>>> >> Unhealthy
> > >>>> >> Readiness
> > >>>> >> probe
> > >>>> >> failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> > >>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> > >>>> >> 'Connection refused'.
> > >>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
> > >>>> >> unary
> > >>>> >> operator expected
> > >>>> >> Cassandra not in the up and normal state. Current state is
> > >>>> >>
> > >>>> >> 1h 1h 2 {kubelet l3inpn-id2-003.cisco.com}
> > >>>> >> spec.containers{hawkular-cassandra-1} Warning Unhealthy
> > >>>> >> Readiness
> > >>>> >> probe
> > >>>> >> failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
> > >>>> >> nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
> > >>>> >> 'Connection refused'.
> > >>>> >> Cassandra not in the up and normal state. Current state is
> > >>>> >> /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =:
> > >>>> >> unary
> > >>>> >> operator expected
> > >>>> >>
> > >>>> >>
> > >>>> >> Logs:
> > >>>> >> ====
> > >>>> >>
> > >>>> >> # oc logs -f heapster-uleul
> > >>>> >> Endpoint Check in effect. Checking
> > >>>> >> https://hawkular-metrics:443/hawkular/metrics/status
> > >>>> >> Could not connect to
> > >>>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> Curl exit code: 6. Status Code 000
> > >>>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is not
> > >>>> >> accessible
> > >>>> >> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > >>>> >
> > >>>> >Heapster is not starting, as expected, the problem is that it cannot
> > >>>> >access
> > >>>> >that URL because it cannot resolve the host (error code 6). This is
> > >>>> >expected, that hostname will not be resolvable until Hawkular Metrics
> > >>>> >starts.
> > >>>> >
> > >>>> >> # oc logs -f hawkular-metrics-gvp9k
> > >>>> >>
> > >>>> >> 16:42:43,201 INFO [org.jboss.as.server] (Controller Boot Thread)
> > >>>> >> JBAS015888:
> > >>>> >> Creating http management service using socket-binding
> > >>>> >> (management-http)
> > >>>> >> 16:42:43,202 INFO [org.xnio] (MSC service thread 1-1) XNIO Version
> > >>>> >> 3.0.14.GA-redhat-1
> > >>>> >> 16:42:43,208 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO
> > >>>> >> Implementation Version 3.0.14.GA-redhat-1
> > >>>> >> 16:42:43,223 INFO [org.jboss.remoting] (MSC service thread 1-1)
> > >>>> >> JBoss
> > >>>> >> Remoting version 3.3.5.Final-redhat-1
> > >>>> >
> > >>>> >I need the full logs for Hawkular Metrics, if you could get me the
> > >>>> >logs
> > >>>> >for
> > >>>> >the last container that was run that would be helpful (eg oc get logs
> > >>>> >-p
> > >>>> >....)
> > >>>> >
> > >>>> >>
> > >>>> >> # oc logs -f hawkular-cassandra-1-mxd2m
> > >>>> >>
> > >>>> >> INFO 20:15:20 Compacted (b7840c40-326c-11e6-a5f7-694864f768a1) 4
> > >>>> >> sstables
> > >>>> >> to
> > >>>> >> [/cassandra_data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/la-5-big,]
> > >>>> >> to level=0. 1,643 bytes to 1,155 (~70% of original) in 207ms =
> > >>>> >> 0.005321MB/s. 0 total partitions merged to 11. Partition merge
> > >>>> >> counts
> > >>>> >> were
> > >>>> >> {1:11, }
> > >>>> >> INFO 20:15:20 Compacting (b7a3f050-326c-11e6-a5f7-694864f768a1)
> > >>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-4-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-3-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-2-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-1-big-Data.db:level=0,
> > >>>> >> ]
> > >>>> >> INFO 20:15:20 Writing Memtable-schema_columns@672348153(49.749KiB
> > >>>> >> serialized
> > >>>> >> bytes, 967 ops, 0%/0% of on/off-heap limit)
> > >>>> >> INFO 20:15:20 Completed flushing
> > >>>> >> /cassandra_data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/tmp-la-7-big-Data.db
> > >>>> >> (11.212KiB) for commitlog position
> > >>>> >> ReplayPosition(segmentId=1465931718336,
> > >>>> >> position=159511)
> > >>>> >> INFO 20:15:20 Compacted (b797bb50-326c-11e6-a5f7-694864f768a1) 4
> > >>>> >> sstables
> > >>>> >> to
> > >>>> >> [/cassandra_data/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/la-5-big,]
> > >>>> >> to level=0. 128 bytes to 32 (~25% of original) in 146ms =
> > >>>> >> 0.000209MB/s.
> > >>>> >> 0
> > >>>> >> total partitions merged to 1. Partition merge counts were {4:1, }
> > >>>> >> INFO 20:15:20 Compacting (b7ae77a0-326c-11e6-a5f7-694864f768a1)
> > >>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-22-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-21-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-19-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-20-big-Data.db:level=0,
> > >>>> >> ]
> > >>>> >> INFO 20:15:20 Compacted (b7a3f050-326c-11e6-a5f7-694864f768a1) 4
> > >>>> >> sstables
> > >>>> >> to
> > >>>> >> [/cassandra_data/data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/la-5-big,]
> > >>>> >> to level=0. 128 bytes to 32 (~25% of original) in 155ms =
> > >>>> >> 0.000197MB/s.
> > >>>> >> 0
> > >>>> >> total partitions merged to 1. Partition merge counts were {4:1, }
> > >>>> >> INFO 20:15:20 Compacting (b7bca870-326c-11e6-a5f7-694864f768a1)
> > >>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-3-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-2-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-1-big-Data.db:level=0,
> > >>>> >> /cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-4-big-Data.db:level=0,
> > >>>> >> ]
> > >>>> >> INFO 20:15:20 Compacted (b7bca870-326c-11e6-a5f7-694864f768a1) 4
> > >>>> >> sstables
> > >>>> >> to
> > >>>> >> [/cassandra_data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/la-5-big,]
> > >>>> >> to level=0. 128 bytes to 32 (~25% of original) in 83ms =
> > >>>> >> 0.000368MB/s. 0
> > >>>> >> total partitions merged to 1. Partition merge counts were {4:1, }
> > >>>> >> INFO 20:15:20 Compacted (b7ae77a0-326c-11e6-a5f7-694864f768a1) 4
> > >>>> >> sstables
> > >>>> >> to
> > >>>> >> [/cassandra_data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/la-23-big,]
> > >>>> >> to level=0. 2,538 bytes to 527 (~20% of original) in 176ms =
> > >>>> >> 0.002856MB/s.
> > >>>> >> 0 total partitions merged to 3. Partition merge counts were {4:3,
> > >>>> >> }
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> almost struck here. Any points to look for? Any ports need to be
> > >>>> >> open
> > >>>> >> explicitly ?
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >>
> > >>>> >> --
> > >>>> >> Srinivas Kotaru
> > >>>> >>
> > >>>> >> On 6/14/16, 11:27 AM, "Matt Wringe" <mwri...@redhat.com> wrote:
> > >>>> >>
> > >>>> >> >----- Original Message -----
> > >>>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > >>>> >> >> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> Cc: users@lists.openshift.redhat.com
> > >>>> >> >> Sent: Tuesday, June 14, 2016 2:09:49 PM
> > >>>> >> >> Subject: Re: Metrics deployment
> > >>>> >> >>
> > >>>> >> >> Not sure what do you mean by installation. Am just running oc
> > >>>> >> >> new-app
> > >>>> >> >> -f
> > >>>> >> >> metrics-deployer.yaml with default values except HOST_NAME and
> > >>>> >> >> PV
> > >>>> >> >> storage.
> > >>>> >> >
> > >>>> >> >I would suspect something wrong with your node or cluster
> > >>>> >> >installation.
> > >>>> >> >Or
> > >>>> >> >you have firewall rules blocking connections between your nodes so
> > >>>> >> >that
> > >>>> >> >pod
> > >>>> >> >cannot connect with each other or access the OpenShift DNS server.
> > >>>> >> >
> > >>>> >> >The lifecycle hooks exist to make sure that components only enter
> > >>>> >> >the
> > >>>> >> >ready
> > >>>> >> >state when they are fully started and ready.
> > >>>> >> >
> > >>>> >> >Can you check the Hawkular Metrics status page and see what that
> > >>>> >> >outputs?
> > >>>> >> >eg
> > >>>> >> >https://${HAWKULAR_METRICS_HOSTNAME}/hawkular/metrics/status
> > >>>> >> >
> > >>>> >> >>
> > >>>> >> >> I just deleted entire metrics setup and re running. But not sure
> > >>>> >> >> this
> > >>>> >> >> will
> > >>>> >> >> fix the issue.
> > >>>> >> >>
> > >>>> >> >> $ ./delete_metrics-infra.sh
> > >>>> >> >> replicationcontroller "hawkular-cassandra-1" deleted
> > >>>> >> >> replicationcontroller "hawkular-metrics" deleted
> > >>>> >> >> replicationcontroller "heapster" deleted
> > >>>> >> >> route "hawkular-metrics" deleted
> > >>>> >> >> service "hawkular-cassandra" deleted
> > >>>> >> >> service "hawkular-cassandra-nodes" deleted
> > >>>> >> >> service "hawkular-metrics" deleted
> > >>>> >> >> service "heapster" deleted
> > >>>> >> >> pod "heapster-lyf65" deleted
> > >>>> >> >> serviceaccount "cassandra" deleted
> > >>>> >> >> serviceaccount "hawkular" deleted
> > >>>> >> >> serviceaccount "heapster" deleted
> > >>>> >> >> template "hawkular-cassandra-node-emptydir" deleted
> > >>>> >> >> template "hawkular-cassandra-node-pv" deleted
> > >>>> >> >> template "hawkular-cassandra-services" deleted
> > >>>> >> >> template "hawkular-heapster" deleted
> > >>>> >> >> template "hawkular-metrics" deleted
> > >>>> >> >> template "hawkular-support" deleted
> > >>>> >> >> secret "hawkular-cassandra-certificate" deleted
> > >>>> >> >> secret "hawkular-cassandra-secrets" deleted
> > >>>> >> >> secret "hawkular-metrics-account" deleted
> > >>>> >> >> secret "hawkular-metrics-certificate" deleted
> > >>>> >> >> secret "hawkular-metrics-secrets" deleted
> > >>>> >> >> secret "heapster-secrets" deleted
> > >>>> >> >>
> > >>>> >> >> --
> > >>>> >> >> Srinivas Kotaru
> > >>>> >> >>
> > >>>> >> >> On 6/14/16, 10:53 AM, "Matt Wringe" <mwri...@redhat.com> wrote:
> > >>>> >> >>
> > >>>> >> >> >----- Original Message -----
> > >>>> >> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>
> > >>>> >> >> >> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> Cc: users@lists.openshift.redhat.com
> > >>>> >> >> >> Sent: Tuesday, June 14, 2016 1:37:01 PM
> > >>>> >> >> >> Subject: Re: Metrics deployment
> > >>>> >> >> >>
> > >>>> >> >> >> I removed readiness probes from both hawkular-cassandra-1 &
> > >>>> >> >> >> hawkular-metrics
> > >>>> >> >> >> as both status shows probes failed.
> > >>>> >> >> >
> > >>>> >> >> >You should not have to remove the probes, this indicates that
> > >>>> >> >> >something
> > >>>> >> >> >is
> > >>>> >> >> >wrong with your installation.
> > >>>> >> >> >
> > >>>> >> >> >>
> > >>>> >> >> >> It looks good now. Both containers looks and running
> > >>>> >> >> >> (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however
> > >>>> >> >> >> heapster-7yl34
> > >>>> >> >> >> logs still shows Could not connect to
> > >>>> >> >> >> https://hawkular-metrics:443/hawkular/metrics/status. Curl
> > >>>> >> >> >> exit
> > >>>> >> >> >> code:
> > >>>> >> >> >> 6.
> > >>>> >> >> >> Status Code 000.
> > >>>> >> >> >>
> > >>>> >> >> >> Are we good or still had issues?
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >> # oc get pods
> > >>>> >> >> >> NAME READY STATUS RESTARTS
> > >>>> >> >> >> AGE
> > >>>> >> >> >> hawkular-cassandra-1-kr8ka 1/1 Running 0
> > >>>> >> >> >> 6m
> > >>>> >> >> >> hawkular-metrics-vhe3u 1/1 Running 2
> > >>>> >> >> >> 5m
> > >>>> >> >> >> heapster-7yl34 0/1 Running 2
> > >>>> >> >> >> 5m
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >> >> --
> > >>>> >> >> >> Srinivas Kotaru
> > >>>> >> >> >>
> > >>>> >> >> >> On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> <skot...@cisco.com>
> > >>>> >> >> >> wrote:
> > >>>> >> >> >>
> > >>>> >> >> >> >Matt
> > >>>> >> >> >> >
> > >>>> >> >> >> >Just want to share more info by running describe pod.
> > >>>> >> >> >> >
> > >>>> >> >> >> >It seems to be health probe failing. Do you think it is the
> > >>>> >> >> >> >issue?
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> ># oc describe pod hawkular-cassandra-1-it5uh
> > >>>> >> >> >> >Name: hawkular-cassandra-1-it5uh
> > >>>> >> >> >> >Namespace: openshift-infra
> > >>>> >> >> >> >Node: l3inpn-id2-003.cisco.com/173.36.96.16
> > >>>> >> >> >> >Start Time: Tue, 14 Jun 2016 16:36:21 +0000
> > >>>> >> >> >> >Labels:
> > >>>> >> >> >> >
> > >>>> >> >> >> > metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra
> > >>>> >> >> >> >Status: Running
> > >>>> >> >> >> >IP: 10.1.9.2
> > >>>> >> >> >> >Controllers: ReplicationController/hawkular-cassandra-1
> > >>>> >> >> >> >Containers:
> > >>>> >> >> >> > hawkular-cassandra-1:
> > >>>> >> >> >> > Container ID:
> > >>>> >> >> >> >
> > >>>> >> >> >> > docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d
> > >>>> >> >> >> > Image:
> > >>>> >> >> >> >
> > >>>> >> >> >> > registry.access.redhat.com/openshift3/metrics-cassandra:latest
> > >>>> >> >> >> > Image ID:
> > >>>> >> >> >> >
> > >>>> >> >> >> > docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53
> > >>>> >> >> >> > Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
> > >>>> >> >> >> > Command:
> > >>>> >> >> >> > /opt/apache-cassandra/bin/cassandra-docker.sh
> > >>>> >> >> >> > --cluster_name=hawkular-metrics
> > >>>> >> >> >> > --data_volume=/cassandra_data
> > >>>> >> >> >> > --internode_encryption=all
> > >>>> >> >> >> > --require_node_auth=true
> > >>>> >> >> >> > --enable_client_encryption=true
> > >>>> >> >> >> > --require_client_auth=true
> > >>>> >> >> >> > --keystore_file=/secret/cassandra.keystore
> > >>>> >> >> >> >
> > >>>> >> >> >> > --keystore_password_file=/secret/cassandra.keystore.password
> > >>>> >> >> >> > --truststore_file=/secret/cassandra.truststore
> > >>>> >> >> >> >
> > >>>> >> >> >> > --truststore_password_file=/secret/cassandra.truststore.password
> > >>>> >> >> >> > --cassandra_pem_file=/secret/cassandra.pem
> > >>>> >> >> >> > QoS Tier:
> > >>>> >> >> >> > cpu: BestEffort
> > >>>> >> >> >> > memory: BestEffort
> > >>>> >> >> >> > State: Running
> > >>>> >> >> >> > Started: Tue, 14 Jun 2016 16:37:01 +0000
> > >>>> >> >> >> > Ready: True
> > >>>> >> >> >> > Restart Count: 0
> > >>>> >> >> >> > Readiness: exec
> > >>>> >> >> >> > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh]
> > >>>> >> >> >> > delay=0s timeout=1s period=10s #success=1 #failure=3
> > >>>> >> >> >> > Environment Variables:
> > >>>> >> >> >> > CASSANDRA_MASTER: true
> > >>>> >> >> >> > POD_NAMESPACE: openshift-infra (v1:metadata.namespace)
> > >>>> >> >> >> >Conditions:
> > >>>> >> >> >> > Type Status
> > >>>> >> >> >> > Ready True
> > >>>> >> >> >> >Volumes:
> > >>>> >> >> >> > cassandra-data:
> > >>>> >> >> >> > Type: PersistentVolumeClaim (a reference to a
> > >>>> >> >> >> > PersistentVolumeClaim
> > >>>> >> >> >> > in
> > >>>> >> >> >> > the same namespace)
> > >>>> >> >> >> > ClaimName: metrics-cassandra-1
> > >>>> >> >> >> > ReadOnly: false
> > >>>> >> >> >> > hawkular-cassandra-secrets:
> > >>>> >> >> >> > Type: Secret (a volume populated by a Secret)
> > >>>> >> >> >> > SecretName: hawkular-cassandra-secrets
> > >>>> >> >> >> > cassandra-token-4urfd:
> > >>>> >> >> >> > Type: Secret (a volume populated by a Secret)
> > >>>> >> >> >> > SecretName: cassandra-token-4urfd
> > >>>> >> >> >> >Events:
> > >>>> >> >> >> > FirstSeen LastSeen Count From
> > >>>> >> >> >> > SubobjectPath Type
> > >>>> >> >> >> > Reason
> > >>>> >> >> >> > Message
> > >>>> >> >> >> > --------- -------- ----- ----
> > >>>> >> >> >> > -------------
> > >>>> >> >> >> > --------
> > >>>> >> >> >> > ------
> > >>>> >> >> >> > -------
> > >>>> >> >> >> > 27m 27m 1
> > >>>> >> >> >> > {default-scheduler }
> > >>>> >> >> >> > Normal Scheduled
> > >>>> >> >> >> > Successfully
> > >>>> >> >> >> > assigned hawkular-cassandra-1-it5uh to
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com
> > >>>> >> >> >> > 27m 27m 1 {kubelet
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com}
> > >>>> >> >> >> > spec.containers{hawkular-cassandra-1} Normal
> > >>>> >> >> >> > Pulling
> > >>>> >> >> >> > pulling
> > >>>> >> >> >> > image
> > >>>> >> >> >> >
> > >>>> >> >> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> > >>>> >> >> >> > 27m 27m 1 {kubelet
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com}
> > >>>> >> >> >> > spec.containers{hawkular-cassandra-1} Normal
> > >>>> >> >> >> > Pulled
> > >>>> >> >> >> > Successfully
> > >>>> >> >> >> > pulled
> > >>>> >> >> >> > image
> > >>>> >> >> >> >
> > >>>> >> >> >> > "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
> > >>>> >> >> >> > 27m 27m 1 {kubelet
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com}
> > >>>> >> >> >> > spec.containers{hawkular-cassandra-1} Normal
> > >>>> >> >> >> > Created
> > >>>> >> >> >> > Created
> > >>>> >> >> >> > container
> > >>>> >> >> >> > with docker id 17a9575eb655
> > >>>> >> >> >> > 27m 27m 1 {kubelet
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com}
> > >>>> >> >> >> > spec.containers{hawkular-cassandra-1} Normal
> > >>>> >> >> >> > Started
> > >>>> >> >> >> > Started
> > >>>> >> >> >> > container
> > >>>> >> >> >> > with docker id 17a9575eb655
> > >>>> >> >> >> > 27m 26m 3 {kubelet
> > >>>> >> >> >> > l3inpn-id2-003.cisco.com}
> > >>>> >> >> >> > spec.containers{hawkular-cassandra-1} Warning
> > >>>> >> >> >> > Unhealthy
> > >>>> >> >> >> > Readiness
> > >>>> >> >> >> > probe
> > >>>> >> >> >> > failed: cat: /etc/ld.so.conf.d/*.conf: No such file or
> > >>>> >> >> >> > directory
> > >>>> >> >> >> >nodetool: Failed to connect to '127.0.0.1:7199' -
> > >>>> >> >> >> >ConnectException:
> > >>>> >> >> >> >'Connection refused'.
> > >>>> >> >> >> >Cassandra not in the up and normal state. Current state is
> > >>>> >> >> >> >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line
> > >>>> >> >> >> >28:
> > >>>> >> >> >> >[: =:
> > >>>> >> >> >> >unary
> > >>>> >> >> >> >operator expected
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >
> > >>>> >> >> >> >--
> > >>>> >> >> >> >Srinivas Kotaru
> > >>>> >> >> >> >
> > >>>> >> >> >> >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> ><skot...@cisco.com>
> > >>>> >> >> >> >wrote:
> > >>>> >> >> >> >
> > >>>> >> >> >> >>Matt
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>DNS service is working
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>~ dig +short @master-01
> > >>>> >> >> >> >>kubernetes.default.svc.cluster.local
> > >>>> >> >> >> >>172.30.0.1
> > >>>> >> >> >> >>~ dig +short @master-01
> > >>>> >> >> >> >>jenkins.alln-test.svc.cluster.local
> > >>>> >> >> >> >>172.30.85.148
> > >>>> >> >> >> >>~ dig +short @master-01
> > >>>> >> >> >> >>cakephp-example.alln-test.svc.cluster.local
> > >>>> >> >> >> >>172.30.31.6
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>I captured hawkular-metrics, it shows the problem. It seems
> > >>>> >> >> >> >>to be
> > >>>> >> >> >> >>unable
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>connect Cassandra cluster
> > >>>> >> >> >> >>
> > >>>> >> >> >> >># oc exec hawkular-metrics-qldd8 cat
> > >>>> >> >> >> >>/opt/eap/standalone/log/server.log
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>12:40:58,848 INFO [org.jboss.as.clustering.infinispan]
> > >>>> >> >> >> >>(ServerService
> > >>>> >> >> >> >>Thread Pool -- 31) JBAS010280: Activating Infinispan
> > >>>> >> >> >> >>subsystem.
> > >>>> >> >> >> >>12:40:58,898 INFO [org.jboss.as.webservices]
> > >>>> >> >> >> >>(ServerService
> > >>>> >> >> >> >>Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>48) JBAS015537: Activating WebServices Extension
> > >>>> >> >> >> >>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>46)
> > >>>> >> >> >> >>JBAS010153: Node identifier property is set to the default
> > >>>> >> >> >> >>value.
> > >>>> >> >> >> >>Please
> > >>>> >> >> >> >>make sure it is unique.
> > >>>> >> >> >> >>12:40:58,902 INFO [org.jboss.as.security] (ServerService
> > >>>> >> >> >> >>Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>44) JBAS013371: Activating Security Subsystem
> > >>>> >> >> >> >>12:40:58,912 INFO [org.jboss.as.naming] (ServerService
> > >>>> >> >> >> >>Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>39)
> > >>>> >> >> >> >>JBAS011800: Activating Naming Subsystem
> > >>>> >> >> >> >>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>37)
> > >>>> >> >> >> >>JBAS012615: Activated the following JSF Implementations:
> > >>>> >> >> >> >>[main,
> > >>>> >> >> >> >>1.2]
> > >>>> >> >> >> >>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar
> > >>>> >> >> >> >>1.0.32.Final-redhat-1)
> > >>>> >> >> >> >>12:40:58,981 INFO [org.jboss.as.security] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-2)
> > >>>> >> >> >> >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1
> > >>>> >> >> >> >>12:40:58,961 INFO
> > >>>> >> >> >> >>[org.jboss.as.connector.subsystems.datasources]
> > >>>> >> >> >> >>(ServerService Thread Pool -- 27) JBAS010403: Deploying
> > >>>> >> >> >> >>JDBC-compliant
> > >>>> >> >> >> >>driver class org.h2.Driver (version 1.3)
> > >>>> >> >> >> >>12:40:59,039 INFO [org.jboss.as.naming] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-2)
> > >>>> >> >> >> >>JBAS011802: Starting Naming Service
> > >>>> >> >> >> >>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-4)
> > >>>> >> >> >> >>JBAS015400: Bound mail session [java:jboss/mail/Default]
> > >>>> >> >> >> >>12:40:59,614 INFO
> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on :
> > >>>> >> >> >> >>http-/0.0.0.0:8444
> > >>>> >> >> >> >>12:40:59,620 INFO
> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on :
> > >>>> >> >> >> >>http-/0.0.0.0:8080
> > >>>> >> >> >> >>12:40:59,621 INFO
> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on:
> > >>>> >> >> >> >>http-/0.0.0.0:8444
> > >>>> >> >> >> >>12:40:59,627 INFO
> > >>>> >> >> >> >>[org.apache.coyote.http11.Http11Protocol]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on:
> > >>>> >> >> >> >>http-/0.0.0.0:8080
> > >>>> >> >> >> >>12:40:59,654 INFO
> > >>>> >> >> >> >>[org.jboss.as.connector.subsystems.datasources]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service thread 1-3) JBAS010400: Bound data source
> > >>>> >> >> >> >>[java:jboss/datasources/ExampleDS]
> > >>>> >> >> >> >>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar"
> > >>>> >> >> >> >>(runtime-name:
> > >>>> >> >> >> >>"activemq-rar.rar")
> > >>>> >> >> >> >>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1) JBAS015876: Starting deployment of
> > >>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>(runtime-name: "hawkular-metrics-api-jaxrs.war")
> > >>>> >> >> >> >>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread 1-3) JBAS015012: Started FileSystemDeploymentService
> > >>>> >> >> >> >>for
> > >>>> >> >> >> >>directory
> > >>>> >> >> >> >>/opt/eap/standalone/deployments
> > >>>> >> >> >> >>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-2)
> > >>>> >> >> >> >>JBAS017100: Listening on 127.0.0.1:9999
> > >>>> >> >> >> >>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-3)
> > >>>> >> >> >> >>JBAS015102: Unable to load console module for slot eap,
> > >>>> >> >> >> >>disabling
> > >>>> >> >> >> >>console
> > >>>> >> >> >> >>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF
> > >>>> >> >> >> >>Server
> > >>>> >> >> >> >>4.3.5.Final-redhat-3
> > >>>> >> >> >> >>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-4)
> > >>>> >> >> >> >>JBAS017100: Listening on 0.0.0.0:4447
> > >>>> >> >> >> >>12:41:01,361 INFO
> > >>>> >> >> >> >>[org.jboss.as.connector.deployers.RADeployer]
> > >>>> >> >> >> >>(MSC
> > >>>> >> >> >> >>service thread 1-3) IJ020001: Required license terms for
> > >>>> >> >> >> >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/
> > >>>> >> >> >> >>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>is
> > >>>> >> >> >> >>using a private module
> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
> > >>>> >> >> >> >>which may be changed or removed in future versions without
> > >>>> >> >> >> >>notice.
> > >>>> >> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>is
> > >>>> >> >> >> >>using a private module
> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-core-asl:main")
> > >>>> >> >> >> >>which may be changed or removed in future versions without
> > >>>> >> >> >> >>notice.
> > >>>> >> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>is
> > >>>> >> >> >> >>using a private module
> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
> > >>>> >> >> >> >>which may be changed or removed in future versions without
> > >>>> >> >> >> >>notice.
> > >>>> >> >> >> >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1) JBAS015867: Deployment
> > >>>> >> >> >> >>"deployment.hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>is
> > >>>> >> >> >> >>using a private module
> > >>>> >> >> >> >>("org.codehaus.jackson.jackson-mapper-asl:main")
> > >>>> >> >> >> >>which may be changed or removed in future versions without
> > >>>> >> >> >> >>notice.
> > >>>> >> >> >> >>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1)
> > >>>> >> >> >> >>JBAS016002: Processing weld deployment
> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
> > >>>> >> >> >> >>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1)
> > >>>> >> >> >> >>JBAS016005: Starting Services for CDI deployment:
> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
> > >>>> >> >> >> >>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-1)
> > >>>> >> >> >> >>WELD-000900 1.1.31 (redhat)
> > >>>> >> >> >> >>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service
> > >>>> >> >> >> >>thread
> > >>>> >> >> >> >>1-3)
> > >>>> >> >> >> >>JBAS016008: Starting weld service for deployment
> > >>>> >> >> >> >>hawkular-metrics-api-jaxrs.war
> > >>>> >> >> >> >>12:41:02,820 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:02,835 INFO [org.jboss.web] (ServerService Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>51)
> > >>>> >> >> >> >>JBAS018210: Register web context: /hawkular/metrics
> > >>>> >> >> >> >>12:41:02,958 INFO [org.jboss.as.server] (ServerService
> > >>>> >> >> >> >>Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>28)
> > >>>> >> >> >> >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war"
> > >>>> >> >> >> >>(runtime-name :
> > >>>> >> >> >> >>"hawkular-metrics-api-jaxrs.war")
> > >>>> >> >> >> >>12:41:02,959 INFO [org.jboss.as.server] (ServerService
> > >>>> >> >> >> >>Thread
> > >>>> >> >> >> >>Pool
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>28)
> > >>>> >> >> >> >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name :
> > >>>> >> >> >> >>"activemq-rar.rar")
> > >>>> >> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> > >>>> >> >> >> >>JBAS015961:
> > >>>> >> >> >> >>Http
> > >>>> >> >> >> >>management interface listening on
> > >>>> >> >> >> >>http://127.0.0.1:9990/management
> > >>>> >> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> > >>>> >> >> >> >>JBAS015951:
> > >>>> >> >> >> >>Admin console listening on http://127.0.0.1:9990
> > >>>> >> >> >> >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread)
> > >>>> >> >> >> >>JBAS015874:
> > >>>> >> >> >> >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in
> > >>>> >> >> >> >>6593ms -
> > >>>> >> >> >> >>Started
> > >>>> >> >> >> >>267 of 303 services (62 services are lazy, passive or
> > >>>> >> >> >> >>on-demand)
> > >>>> >> >> >> >>12:41:02,999 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra:
> > >>>> >> >> >> >>unknown
> > >>>> >> >> >> >>error
> > >>>> >> >> >> >>12:41:02,999 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
> > >>>> >> >> >> >>12:41:04,000 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:04,001 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:04,001 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
> > >>>> >> >> >> >>12:41:06,002 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:06,002 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:06,003 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
> > >>>> >> >> >> >>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2)
> > >>>> >> >> >> >>RESTEASY006050: Found BeanManager at java:comp/BeanManager
> > >>>> >> >> >> >>12:41:08,299 INFO
> > >>>> >> >> >> >>[org.hibernate.validator.internal.util.Version]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator
> > >>>> >> >> >> >>4.3.2.Final-redhat-2
> > >>>> >> >> >> >>12:41:08,418 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
> > >>>> >> >> >> >>starting
> > >>>> >> >> >> >>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying
> > >>>> >> >> >> >>javax.ws.rs.core.Application: class
> > >>>> >> >> >> >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy
> > >>>> >> >> >> >>12:41:08,421 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics
> > >>>> >> >> >> >>starting
> > >>>> >> >> >> >>12:41:08,422 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> > >>>> >> >> >> >>12:41:08,422 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp]
> > >>>> >> >> >> >>(http-/0.0.0.0:8080-2) Virtual clock is disabled
> > >>>> >> >> >> >>12:41:09,003 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:09,004 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:09,004 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
> > >>>> >> >> >> >>12:41:13,004 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:13,123 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra:
> > >>>> >> >> >> >>unknown
> > >>>> >> >> >> >>error
> > >>>> >> >> >> >>12:41:13,124 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
> > >>>> >> >> >> >>12:41:14,124 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:14,124 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:14,125 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
> > >>>> >> >> >> >>12:41:16,125 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:16,126 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:16,126 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
> > >>>> >> >> >> >>12:41:19,126 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:19,127 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:19,127 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
> > >>>> >> >> >> >>12:41:23,128 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:23,249 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra:
> > >>>> >> >> >> >>unknown
> > >>>> >> >> >> >>error
> > >>>> >> >> >> >>12:41:23,249 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
> > >>>> >> >> >> >>12:41:24,250 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:24,250 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:24,250 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
> > >>>> >> >> >> >>12:41:26,251 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:26,251 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:26,251 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
> > >>>> >> >> >> >>12:41:29,252 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:29,252 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:29,253 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
> > >>>> >> >> >> >>12:41:33,253 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:33,376 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra:
> > >>>> >> >> >> >>unknown
> > >>>> >> >> >> >>error
> > >>>> >> >> >> >>12:41:33,377 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
> > >>>> >> >> >> >>12:41:34,377 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:34,378 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:34,378 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
> > >>>> >> >> >> >>12:41:36,378 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:36,379 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:36,379 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [3]s...
> > >>>> >> >> >> >>12:41:39,379 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:39,380 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:39,380 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [4]s...
> > >>>> >> >> >> >>12:41:43,380 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:43,503 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra:
> > >>>> >> >> >> >>unknown
> > >>>> >> >> >> >>error
> > >>>> >> >> >> >>12:41:43,504 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [1]s...
> > >>>> >> >> >> >>12:41:44,504 INFO
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200002:
> > >>>> >> >> >> >>Initializing
> > >>>> >> >> >> >>metrics
> > >>>> >> >> >> >>service
> > >>>> >> >> >> >>12:41:44,505 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could
> > >>>> >> >> >> >>not
> > >>>> >> >> >> >>connect
> > >>>> >> >> >> >>to
> > >>>> >> >> >> >>Cassandra cluster - assuming its not up yet:
> > >>>> >> >> >> >>hawkular-cassandra
> > >>>> >> >> >> >>12:41:44,505 WARN
> > >>>> >> >> >> >>[org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle]
> > >>>> >> >> >> >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18]
> > >>>> >> >> >> >>Retrying
> > >>>> >> >> >> >>connecting to Cassandra cluster in [2]s...
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>--
> > >>>> >> >> >> >>Srinivas Kotaru
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> >>wrote:
> > >>>> >> >> >> >>
> > >>>> >> >> >> >>>----- Original Message -----
> > >>>> >> >> >> >>>> From: "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> >>>> <skot...@cisco.com>
> > >>>> >> >> >> >>>> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> >>>> Cc: users@lists.openshift.redhat.com
> > >>>> >> >> >> >>>> Sent: Monday, June 13, 2016 7:26:06 PM
> > >>>> >> >> >> >>>> Subject: Re: Metrics deployment
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> Matt
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> PV issue resolved. Was able to to see PV successfully
> > >>>> >> >> >> >>>> bounded
> > >>>> >> >> >> >>>> and
> > >>>> >> >> >> >>>> Casandra
> > >>>> >> >> >> >>>> container has been running. However, it seems puzzle not
> > >>>> >> >> >> >>>> fully
> > >>>> >> >> >> >>>> yet
> > >>>> >> >> >> >>>> solved.
> > >>>> >> >> >> >>>
> > >>>> >> >> >> >>>Are you sure the OpenShift DNS server is running?
> > >>>> >> >> >> >>>
> > >>>> >> >> >> >>>If you are running OSE 3.1, can you please follow this
> > >>>> >> >> >> >>>https://access.redhat.com/solutions/2329131 and see if you
> > >>>> >> >> >> >>>are
> > >>>> >> >> >> >>>now
> > >>>> >> >> >> >>>seeing
> > >>>> >> >> >> >>>errors in the Hawkular Metrics logs (essentially just run
> > >>>> >> >> >> >>>`oc
> > >>>> >> >> >> >>>exec
> > >>>> >> >> >> >>>hawkular-metrics-xxxxx cat
> > >>>> >> >> >> >>>/opt/eap/standalone/log/server.log`)
> > >>>> >> >> >> >>>
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> I could see other container(heapster) not coming up, and
> > >>>> >> >> >> >>>> seeing
> > >>>> >> >> >> >>>> below
> > >>>> >> >> >> >>>> errors
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f
> > >>>> >> >> >> >>>> heapster-fnkdc
> > >>>> >> >> >> >>>> Endpoint Check in effect. Checking
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
> > >>>> >> >> >> >>>> Could not connect to
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> is
> > >>>> >> >> >> >>>> not
> > >>>> >> >> >> >>>> accessible
> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > >>>> >> >> >> >>>> Could not connect to
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> is
> > >>>> >> >> >> >>>> not
> > >>>> >> >> >> >>>> accessible
> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> # oc get pv
> > >>>> >> >> >> >>>> pv-5gb-0011 5Gi RWO Bound
> > >>>> >> >> >> >>>> openshift-infra/metrics-cassandra-1 22m
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> $ oc get pods
> > >>>> >> >> >> >>>> NAME READY STATUS
> > >>>> >> >> >> >>>> RESTARTS
> > >>>> >> >> >> >>>> AGE
> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7 1/1 Running 0
> > >>>> >> >> >> >>>> 20m
> > >>>> >> >> >> >>>> hawkular-metrics-mf5qf 0/1 Running 7
> > >>>> >> >> >> >>>> 20m
> > >>>> >> >> >> >>>> heapster-fnkdc 0/1 Error 6
> > >>>> >> >> >> >>>> 20m
> > >>>> >> >> >> >>>> metrics-deployer-cvep0 0/1 Completed 0
> > >>>> >> >> >> >>>> 21m
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> # oc logs -f hawkular-metrics-mf5qf
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2)
> > >>>> >> >> >> >>>> XNIO
> > >>>> >> >> >> >>>> Version
> > >>>> >> >> >> >>>> 3.0.14.GA-redhat-1
> > >>>> >> >> >> >>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller
> > >>>> >> >> >> >>>> Boot
> > >>>> >> >> >> >>>> Thread)
> > >>>> >> >> >> >>>> JBAS015888:
> > >>>> >> >> >> >>>> Creating http management service using socket-binding
> > >>>> >> >> >> >>>> (management-http)
> > >>>> >> >> >> >>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread
> > >>>> >> >> >> >>>> 1-2)
> > >>>> >> >> >> >>>> XNIO
> > >>>> >> >> >> >>>> NIO
> > >>>> >> >> >> >>>> Implementation Version 3.0.14.GA-redhat-1
> > >>>> >> >> >> >>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service
> > >>>> >> >> >> >>>> thread
> > >>>> >> >> >> >>>> 1-2)
> > >>>> >> >> >> >>>> JBoss
> > >>>> >> >> >> >>>> Remoting version 3.3.5.Final-redhat-1
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> $ oc logs -f heapster-fnkdc
> > >>>> >> >> >> >>>> Endpoint Check in effect. Checking
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status
> > >>>> >> >> >> >>>> Could not connect to
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> is
> > >>>> >> >> >> >>>> not
> > >>>> >> >> >> >>>> accessible
> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > >>>> >> >> >> >>>> Could not connect to
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> is
> > >>>> >> >> >> >>>> not
> > >>>> >> >> >> >>>> accessible
> > >>>> >> >> >> >>>> [HTTP status code: 000. Curl exit code 6]. Retrying.
> > >>>> >> >> >> >>>> Could not connect to
> > >>>> >> >> >> >>>> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> $ oc logs -f hawkular-cassandra-1-2pzd7
> > >>>> >> >> >> >>>> INFO 23:00:24 Starting listening for CQL clients on
> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042...
> > >>>> >> >> >> >>>> INFO 23:00:24 Binding thrift service to
> > >>>> >> >> >> >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160
> > >>>> >> >> >> >>>> INFO 23:00:24 enabling encrypted thrift connections
> > >>>> >> >> >> >>>> between
> > >>>> >> >> >> >>>> client
> > >>>> >> >> >> >>>> and
> > >>>> >> >> >> >>>> server
> > >>>> >> >> >> >>>> INFO 23:00:24 Listening for thrift clients...
> > >>>> >> >> >> >>>> INFO 23:00:26 Created default superuser role
> > >>>> >> >> >> >>>> 'cassandra'
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> # oc get svc
> > >>>> >> >> >> >>>> NAME CLUSTER-IP EXTERNAL-IP
> > >>>> >> >> >> >>>> PORT(S)
> > >>>> >> >> >> >>>> AGE
> > >>>> >> >> >> >>>> hawkular-cassandra 172.30.2.13 <none>
> > >>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
> > >>>> >> >> >> >>>> hawkular-cassandra-nodes None <none>
> > >>>> >> >> >> >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m
> > >>>> >> >> >> >>>> hawkular-metrics 172.30.117.176 <none>
> > >>>> >> >> >> >>>> 443/TCP
> > >>>> >> >> >> >>>> 25m
> > >>>> >> >> >> >>>> heapster 172.30.107.135 <none>
> > >>>> >> >> >> >>>> 80/TCP
> > >>>> >> >> >> >>>> 25m
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> HTTP/1.1 504 Gateway Timeout
> > >>>> >> >> >> >>>> Mime-Version: 1.0
> > >>>> >> >> >> >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT
> > >>>> >> >> >> >>>> Content-Type: text/html
> > >>>> >> >> >> >>>> Connection: keep-alive
> > >>>> >> >> >> >>>> Proxy-Connection: keep-alive
> > >>>> >> >> >> >>>> Content-Length: 1572
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> --
> > >>>> >> >> >> >>>> Srinivas Kotaru
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> >>>> <skot...@cisco.com>
> > >>>> >> >> >> >>>> wrote:
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>> >Matt
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >That is good catch. I ran without
> > >>>> >> >> >> >>>> >USE_PERSISTENT_STORAGE=false
> > >>>> >> >> >> >>>> >and
> > >>>> >> >> >> >>>> >working
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >I adjusted PV to 5Gi and reran. Will update progress.
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >Thanks you for your help so far.
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >--
> > >>>> >> >> >> >>>> >Srinivas Kotaru
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> >>>> >wrote:
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>> >>
> > >>>> >> >> >> >>>> >>
> > >>>> >> >> >> >>>> >>----- Original Message -----
> > >>>> >> >> >> >>>> >>> From: "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> >>>> >>> <skot...@cisco.com>
> > >>>> >> >> >> >>>> >>> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> >>>> >>> Cc: users@lists.openshift.redhat.com
> > >>>> >> >> >> >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM
> > >>>> >> >> >> >>>> >>> Subject: Re: Metrics deployment
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> Oh ok
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> Am using PV for metrics
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> description: "The persistent volume size for each of
> > >>>> >> >> >> >>>> >>> the
> > >>>> >> >> >> >>>> >>> Cassandra
> > >>>> >> >> >> >>>> >>> nodes"
> > >>>> >> >> >> >>>> >>> name: CASSANDRA_PV_SIZE
> > >>>> >> >> >> >>>> >>> value: "10Gi"
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> oc get pv
> > >>>> >> >> >> >>>> >>> NAME CAPACITY ACCESSMODES STATUS
> > >>>> >> >> >> >>>> >>> CLAIM
> > >>>> >> >> >> >>>> >>> REASON
> > >>>> >> >> >> >>>> >>> AGE
> > >>>> >> >> >> >>>> >>> pv-1gb-001 1Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-1gb-002 1Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-1gb-003 1Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-1gb-004 1Gi RWO Bound
> > >>>> >> >> >> >>>> >>> thlatt/mongodb
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-1gb-005 1Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-2gb-0010 2Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-2gb-006 2Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-2gb-007 2Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-2gb-008 2Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-2gb-009 2Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-5gb-0011 5Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-5gb-0012 5Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-5gb-0013 5Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-5gb-0014 5Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>> pv-5gb-0015 5Gi RWO Available
> > >>>> >> >> >> >>>> >>> 4d
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> am running with below command
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded
> > >>>> >> >> >> >>>> >>> HOSTNAME,
> > >>>> >> >> >> >>>> >>> MASTER_API
> > >>>> >> >> >> >>>> >>> and
> > >>>> >> >> >> >>>> >>> PV info so not passing any parameters)
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>
> > >>>> >> >> >> >>>> >>I would suspect that Cassandra is blocked because its
> > >>>> >> >> >> >>>> >>waiting
> > >>>> >> >> >> >>>> >>for
> > >>>> >> >> >> >>>> >>10Gi
> > >>>> >> >> >> >>>> >>PV
> > >>>> >> >> >> >>>> >>to become available, and none of the PV listed above
> > >>>> >> >> >> >>>> >>are
> > >>>> >> >> >> >>>> >>big
> > >>>> >> >> >> >>>> >>enough.
> > >>>> >> >> >> >>>> >>
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> --
> > >>>> >> >> >> >>>> >>> Srinivas Kotaru
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe"
> > >>>> >> >> >> >>>> >>> <mwri...@redhat.com>
> > >>>> >> >> >> >>>> >>> wrote:
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>> >----- Original Message -----
> > >>>> >> >> >> >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> >>>> >>> >> <skot...@cisco.com>
> > >>>> >> >> >> >>>> >>> >> To: "Matt Wringe" <mwri...@redhat.com>
> > >>>> >> >> >> >>>> >>> >> Cc: users@lists.openshift.redhat.com
> > >>>> >> >> >> >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM
> > >>>> >> >> >> >>>> >>> >> Subject: Re: Metrics deployment
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> Matt
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> Thanks for looking into. I rerun the setup, but
> > >>>> >> >> >> >>>> >>> >> had
> > >>>> >> >> >> >>>> >>> >> the
> > >>>> >> >> >> >>>> >>> >> same
> > >>>> >> >> >> >>>> >>> >> issue
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> # oc get pods
> > >>>> >> >> >> >>>> >>> >> NAME READY STATUS
> > >>>> >> >> >> >>>> >>> >> RESTARTS
> > >>>> >> >> >> >>>> >>> >> AGE
> > >>>> >> >> >> >>>> >>> >> hawkular-cassandra-1-y2egy 0/1
> > >>>> >> >> >> >>>> >>> >> ContainerCreating
> > >>>> >> >> >> >>>> >>> >> 0
> > >>>> >> >> >> >>>> >>> >> 5m
> > >>>> >> >> >> >>>> >>> >> hawkular-metrics-4b16f 0/1 Running
> > >>>> >> >> >> >>>> >>> >> 1
> > >>>> >> >> >> >>>> >>> >> 4m
> > >>>> >> >> >> >>>> >>> >> heapster-x2gj2 0/1 Running
> > >>>> >> >> >> >>>> >>> >> 2
> > >>>> >> >> >> >>>> >>> >> 4m
> > >>>> >> >> >> >>>> >>> >> metrics-deployer-9v7vc 0/1 Completed
> > >>>> >> >> >> >>>> >>> >> 0
> > >>>> >> >> >> >>>> >>> >> 6m
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy
> > >>>> >> >> >> >>>> >>> >> Error from server: container
> > >>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1"
> > >>>> >> >> >> >>>> >>> >> in
> > >>>> >> >> >> >>>> >>> >> pod
> > >>>> >> >> >> >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start:
> > >>>> >> >> >> >>>> >>> >> ContainerCreating
> > >>>> >> >> >> >>>> >>> >
> > >>>> >> >> >> >>>> >>> >Ok, so it looks like something is blocking the
> > >>>> >> >> >> >>>> >>> >Cassandra
> > >>>> >> >> >> >>>> >>> >pod
> > >>>> >> >> >> >>>> >>> >from
> > >>>> >> >> >> >>>> >>> >starting.
> > >>>> >> >> >> >>>> >>> >
> > >>>> >> >> >> >>>> >>> >If you are using persistent storage, Cassandra will
> > >>>> >> >> >> >>>> >>> >not
> > >>>> >> >> >> >>>> >>> >start
> > >>>> >> >> >> >>>> >>> >until
> > >>>> >> >> >> >>>> >>> >the
> > >>>> >> >> >> >>>> >>> >PV
> > >>>> >> >> >> >>>> >>> >is available. There may be some more information
> > >>>> >> >> >> >>>> >>> >about
> > >>>> >> >> >> >>>> >>> >Cassandra
> > >>>> >> >> >> >>>> >>> >in
> > >>>> >> >> >> >>>> >>> >the
> > >>>> >> >> >> >>>> >>> >pod
> > >>>> >> >> >> >>>> >>> >section of the console under events.
> > >>>> >> >> >> >>>> >>> >
> > >>>> >> >> >> >>>> >>> >What command did you use when deploying the
> > >>>> >> >> >> >>>> >>> >deployer?
> > >>>> >> >> >> >>>> >>> >
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC
> > >>>> >> >> >> >>>> >>> >> service
> > >>>> >> >> >> >>>> >>> >> thread
> > >>>> >> >> >> >>>> >>> >> 1-4)
> > >>>> >> >> >> >>>> >>> >> VM
> > >>>> >> >> >> >>>> >>> >> Arguments: -Duser.home=/home/jboss
> > >>>> >> >> >> >>>> >>> >> -Duser.name=jboss
> > >>>> >> >> >> >>>> >>> >> -D[Standalone]
> > >>>> >> >> >> >>>> >>> >> -XX:+UseCompressedOops -verbose:gc
> > >>>> >> >> >> >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log
> > >>>> >> >> >> >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > >>>> >> >> >> >>>> >>> >> -XX:+UseGCLogFileRotation
> > >>>> >> >> >> >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M
> > >>>> >> >> >> >>>> >>> >> -XX:-TraceClassUnloading
> > >>>> >> >> >> >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m
> > >>>> >> >> >> >>>> >>> >> -Djava.net.preferIPv4Stack=true
> > >>>> >> >> >> >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager
> > >>>> >> >> >> >>>> >>> >> -Djava.awt.headless=true
> > >>>> >> >> >> >>>> >>> >> -Djboss.modules.policy-permissions=true
> > >>>> >> >> >> >>>> >>> >> -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar
> > >>>> >> >> >> >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager
> > >>>> >> >> >> >>>> >>> >> -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false
> > >>>> >> >> >> >>>> >>> >> -Djava.security.egd=file:/dev/./urandom
> > >>>> >> >> >> >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log
> > >>>> >> >> >> >>>> >>> >> -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties
> > >>>> >> >> >> >>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread
> > >>>> >> >> >> >>>> >>> >> 1-3)
> > >>>> >> >> >> >>>> >>> >> XNIO
> > >>>> >> >> >> >>>> >>> >> Version
> > >>>> >> >> >> >>>> >>> >> 3.0.14.GA-redhat-1
> > >>>> >> >> >> >>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service
> > >>>> >> >> >> >>>> >>> >> thread
> > >>>> >> >> >> >>>> >>> >> 1-3)
> > >>>> >> >> >> >>>> >>> >> XNIO
> > >>>> >> >> >> >>>> >>> >> NIO
> > >>>> >> >> >> >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1
> > >>>> >> >> >> >>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server]
> > >>>> >> >> >> >>>> >>> >> (Controller
> > >>>> >> >> >> >>>> >>> >> Boot
> > >>>> >> >> >> >>>> >>> >> Thread)
> > >>>> >> >> >> >>>> >>> >> JBAS015888:
> > >>>> >> >> >> >>>> >>> >> Creating http management service using
> > >>>> >> >> >> >>>> >>> >> socket-binding
> > >>>> >> >> >> >>>> >>> >> (management-http)
> > >>>> >> >> >> >>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC
> > >>>> >> >> >> >>>> >>> >> service
> > >>>> >> >> >> >>>> >>> >> thread
> > >>>> >> >> >> >>>> >>> >> 1-3)
> > >>>> >> >> >> >>>> >>> >> JBoss
> > >>>> >> >> >> >>>> >>> >> Remoting version 3.3.5.Final-redhat-1
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> $ oc logs -f heapster-x2gj2
> > >>>> >> >> >> >>>> >>> >> Endpoint Check in effect. Checking
> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status
> > >>>> >> >> >> >>>> >>> >> Could not connect to
> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> >>> >> is
> > >>>> >> >> >> >>>> >>> >> not
> > >>>> >> >> >> >>>> >>> >> accessible
> > >>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6].
> > >>>> >> >> >> >>>> >>> >> Retrying.
> > >>>> >> >> >> >>>> >>> >> Could not connect to
> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> >>> >> is
> > >>>> >> >> >> >>>> >>> >> not
> > >>>> >> >> >> >>>> >>> >> accessible
> > >>>> >> >> >> >>>> >>> >> [HTTP status code: 000. Curl exit code 6].
> > >>>> >> >> >> >>>> >>> >> Retrying.
> > >>>> >> >> >> >>>> >>> >> Could not connect to
> > >>>> >> >> >> >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> >>> >> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> $ oc logs -f metrics-deployer-9v7vc
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> ++ oc create -f -
> > >>>> >> >> >> >>>> >>> >> serviceaccount "heapster" created
> > >>>> >> >> >> >>>> >>> >> service "heapster" created
> > >>>> >> >> >> >>>> >>> >> replicationcontroller "heapster" created
> > >>>> >> >> >> >>>> >>> >> + echo 'Success!'
> > >>>> >> >> >> >>>> >>> >> Success!
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> --
> > >>>> >> >> >> >>>> >>> >> Srinivas Kotaru
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe"
> > >>>> >> >> >> >>>> >>> >> <mwri...@redhat.com>
> > >>>> >> >> >> >>>> >>> >> wrote:
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >----- Original Message -----
> > >>>> >> >> >> >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)"
> > >>>> >> >> >> >>>> >>> >> >> <skot...@cisco.com>
> > >>>> >> >> >> >>>> >>> >> >> To: users@lists.openshift.redhat.com
> > >>>> >> >> >> >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM
> > >>>> >> >> >> >>>> >>> >> >> Subject: Metrics deployment
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Hi
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Am trying to configure metrics in our newly
> > >>>> >> >> >> >>>> >>> >> >> installed
> > >>>> >> >> >> >>>> >>> >> >> clusters. Am
> > >>>> >> >> >> >>>> >>> >> >> seeing
> > >>>> >> >> >> >>>> >>> >> >> below errors once metrics-deploy script was
> > >>>> >> >> >> >>>> >>> >> >> successful.
> > >>>> >> >> >> >>>> >>> >> >> I
> > >>>> >> >> >> >>>> >>> >> >> used
> > >>>> >> >> >> >>>> >>> >> >> our
> > >>>> >> >> >> >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME
> > >>>> >> >> >> >>>> >>> >> >> and
> > >>>> >> >> >> >>>> >>> >> >> MASTER_URL
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL,
> > >>>> >> >> >> >>>> >>> >> >> and
> > >>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
> > >>>> >> >> >> >>>> >>> >> >> (
> > >>>> >> >> >> >>>> >>> >> >> hard coded as values)
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> template "hawkular-heapster" created
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Deploying the Heapster component
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> ++ echo 'Deploying the Heapster component'
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> ++ '[' -n '' ']'
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> ++ oc create -f -
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> ++ oc process hawkular-heapster -v
> > >>>> >> >> >> >>>> >>> >> >> IMAGE_PREFIX=registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> serviceaccount "heapster" created
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> service "heapster" created
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> replicationcontroller "heapster" created
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> + echo 'Success!'
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Success!
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> # oc get pods
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> NAME READY STATUS RESTARTS AGE
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1
> > >>>> >> >> >> >>>> >>> >> >> ContainerCreating
> > >>>> >> >> >> >>>> >>> >> >> 0 4m
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0
> > >>>> >> >> >> >>>> >>> >> >> 3s
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >It looks like none of your containers are fully
> > >>>> >> >> >> >>>> >>> >> >up
> > >>>> >> >> >> >>>> >>> >> >and
> > >>>> >> >> >> >>>> >>> >> >running
> > >>>> >> >> >> >>>> >>> >> >yet.
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >Without Cassandra running, Hawkular Metrics will
> > >>>> >> >> >> >>>> >>> >> >not
> > >>>> >> >> >> >>>> >>> >> >run,
> > >>>> >> >> >> >>>> >>> >> >and
> > >>>> >> >> >> >>>> >>> >> >Heapster
> > >>>> >> >> >> >>>> >>> >> >will
> > >>>> >> >> >> >>>> >>> >> >wait until Hawkular Metrics is fully running.
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >Do you see anything in the Cassandra logs? The
> > >>>> >> >> >> >>>> >>> >> >first
> > >>>> >> >> >> >>>> >>> >> >step
> > >>>> >> >> >> >>>> >>> >> >will
> > >>>> >> >> >> >>>> >>> >> >be to
> > >>>> >> >> >> >>>> >>> >> >get
> > >>>> >> >> >> >>>> >>> >> >Cassandra running properly.
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> $ oc logs -f heapster-e8gbu
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Endpoint Check in effect. Checking
> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Could not connect to
> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status'
> > >>>> >> >> >> >>>> >>> >> >> is
> > >>>> >> >> >> >>>> >>> >> >> not
> > >>>> >> >> >> >>>> >>> >> >> accessible
> > >>>> >> >> >> >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6].
> > >>>> >> >> >> >>>> >>> >> >> Retrying.
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Could not connect to
> > >>>> >> >> >> >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status.
> > >>>> >> >> >> >>>> >>> >> >> Curl exit code: 6. Status Code 000
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >Heapster waits until Hawkular Metrics is started
> > >>>> >> >> >> >>>> >>> >> >before
> > >>>> >> >> >> >>>> >>> >> >trying
> > >>>> >> >> >> >>>> >>> >> >to
> > >>>> >> >> >> >>>> >>> >> >push
> > >>>> >> >> >> >>>> >>> >> >metrics to it. The issue that you are seeing is
> > >>>> >> >> >> >>>> >>> >> >because
> > >>>> >> >> >> >>>> >>> >> >Heapster
> > >>>> >> >> >> >>>> >>> >> >could
> > >>>> >> >> >> >>>> >>> >> >not
> > >>>> >> >> >> >>>> >>> >> >properly connect to Hawkular Metrics. Until the
> > >>>> >> >> >> >>>> >>> >> >Hawkular
> > >>>> >> >> >> >>>> >>> >> >Metrics
> > >>>> >> >> >> >>>> >>> >> >service
> > >>>> >> >> >> >>>> >>> >> >is
> > >>>> >> >> >> >>>> >>> >> >fully up, Heapster will not be able to connect
> > >>>> >> >> >> >>>> >>> >> >to
> > >>>> >> >> >> >>>> >>> >> >it.
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> What is the wrong? Why it checking just
> > >>>> >> >> >> >>>> >>> >> >> hawkular-metrics
> > >>>> >> >> >> >>>> >>> >> >> rather
> > >>>> >> >> >> >>>> >>> >> >> full
> > >>>> >> >> >> >>>> >>> >> >> routing
> > >>>> >> >> >> >>>> >>> >> >> URL which was provided as
> > >>>> >> >> >> >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >The Hawkular Metrics service has two hostnames:
> > >>>> >> >> >> >>>> >>> >> >the
> > >>>> >> >> >> >>>> >>> >> >internal
> > >>>> >> >> >> >>>> >>> >> >hostname
> > >>>> >> >> >> >>>> >>> >> >used
> > >>>> >> >> >> >>>> >>> >> >by the internal components (eg
> > >>>> >> >> >> >>>> >>> >> >'hawkular-metrics')
> > >>>> >> >> >> >>>> >>> >> >and
> > >>>> >> >> >> >>>> >>> >> >the
> > >>>> >> >> >> >>>> >>> >> >external
> > >>>> >> >> >> >>>> >>> >> >hostname (eg what is configured via
> > >>>> >> >> >> >>>> >>> >> >HAWKULAR_METRICS_HOSTNAME).
> > >>>> >> >> >> >>>> >>> >> >The
> > >>>> >> >> >> >>>> >>> >> >OpenShift dns server will resolve hostnames to
> > >>>> >> >> >> >>>> >>> >> >the
> > >>>> >> >> >> >>>> >>> >> >name
> > >>>> >> >> >> >>>> >>> >> >of
> > >>>> >> >> >> >>>> >>> >> >services,
> > >>>> >> >> >> >>>> >>> >> >which
> > >>>> >> >> >> >>>> >>> >> >is where the internal 'hawkular-metrics' comes
> > >>>> >> >> >> >>>> >>> >> >from.
> > >>>> >> >> >> >>>> >>> >> >
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> --
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> Srinivas Kotaru
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >> >> _______________________________________________
> > >>>> >> >> >> >>>> >>> >> >> users mailing list
> > >>>> >> >> >> >>>> >>> >> >> users@lists.openshift.redhat.com
> > >>>> >> >> >> >>>> >>> >> >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> > >>>> >> >> >> >>>> >>> >> >>
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>> >>
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >>>
> > >>>> >> >> >> >>>> >
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>>>
> > >>>> >> >> >> >>
> > >>>> >> >> >> >
> > >>>> >> >> >>
> > >>>> >> >> >>
> > >>>> >> >>
> > >>>> >> >>
> > >>>> >>
> > >>>> >>
> > >>>>
> > >>>>
> > >>
> > >
> >
> >
>
_______________________________________________
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
