Hi Punnadi,
Seems like your Kubernetes Service is not up and running or your Stratos
host cannot reach the Kubernetes EP.
Were you able to access
http://192.168.19.51:8080/api/v1/namespaces/default/services from your
browser?
Thanks,
On Tue, Aug 25, 2015 at 3:05 PM, Punnadi Gunarathna <punn...@wso2.com>
wrote:
>
> Hi All,
>
> I got the below exception while trying to spawn multi tenant WSO2AS
> cartridge:
>
> TID: [0] [SCC] [2015-08-25 14:53:00,707] INFO
> {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} -
> Creating kubernetes service: [cluster]
> asdevelopment.punasdevelopment.punasdevelopment.domain [service] service-1
> [protocol] http [service-port] 30000 [container-port] 9763
> TID: [0] [SCC] [2015-08-25 14:53:01,098] WARN
> {org.apache.cxf.phase.PhaseInterceptorChain} - Interceptor for
> {http://192.168.19.51:8080}WebClient has thrown exception, unwinding now
> org.apache.cxf.interceptor.Fault: Could not send Message.
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
> at com.sun.proxy.$Proxy87.createService(Unknown Source)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430)
> at
> org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.ConnectException: ConnectException invoking
> http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection
> refused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
> at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
> ... 16 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
> at
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210)
> at
> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)
> at
> org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320)
> ... 19 more
> TID: [0] [SCC] [2015-08-25 14:53:01,102] ERROR
> {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Could not
> create kubernetes service: [service-id] service-1 [service-name]
> asdevelopment-punasdevelopment-punasdevelopment-domain [node-port] 30000
> [container-port-name] http-9763 [container-port] 9763
> javax.ws.rs.ProcessingException: java.net.ConnectException:
> ConnectException invoking
> http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection
> refused
> at
> org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:676)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
> at com.sun.proxy.$Proxy87.createService(Unknown Source)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430)
> at
> org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.ConnectException: ConnectException invoking
> http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection
> refused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
> at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
> ... 13 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
> at
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210)
> at
> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)
> at
> org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320)
> ... 19 more
> TID: [0] [SCC] [2015-08-25 14:53:01,191] ERROR
> {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} -
> Could not start container: [application] asdevelopment [cartridge]
> punasdevelopment [member]
> asdevelopment.punasdevelopment.punasdevelopment.domain445288ff-b028-42cd-97c3-327245226876
> org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException:
> Could not create kubernetes service: [service-id] service-1 [service-name]
> asdevelopment-punasdevelopment-punasdevelopment-domain [node-port] 30000
> [container-port-name] http-9763 [container-port] 9763
> at
> org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:229)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223)
> at
> org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109)
> at
> org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException:
> ConnectException invoking
> http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection
> refused
> at
> org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:676)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
> at com.sun.proxy.$Proxy87.createService(Unknown Source)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440)
> at
> io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430)
> at
> org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217)
> ... 8 more
> Caused by: java.net.ConnectException: ConnectException invoking
> http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection
> refused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
> at
> org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
> at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
> at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
> at
> org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674)
> ... 13 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
> at
> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
> at
> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
> at
> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
> at
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267)
> at
> org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210)
> at
> org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)
> at
> org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320)
> ... 19 more
>
> Kubernetes cluster definition and stratos log are attached herewith.
>
> Please help me to fix this issue.
> --
>
> Thanks and Regards,
>
> Punnadi Gunarathna
> Senior Software Engineer,
> WSO2, Inc.; http://wso2.com <http://wso2>
> Blog: http://hi-my-world.blogspot.com/
> Tel : 94 11 214 5345
> Fax :94 11 2145300
>
> <http://lalajisureshika.blogspot.com/>
>
--
*Lasindu Charith*
Software Engineer, WSO2 Inc.
Committer & PMC Member, Apache Stratos
Mobile: +94714427192 | Web: blog.lasindu.com
