Punnadi Gunarathna created STRATOS-1530: -------------------------------------------
Summary: Exception while Creating kubernetes service on kubernetes cluster installed on bare-metal ubuntu machine where master and minion IPs are same Key: STRATOS-1530 URL: https://issues.apache.org/jira/browse/STRATOS-1530 Project: Stratos Issue Type: Bug Components: Kubernetes API Client Affects Versions: 4.1.1 Reporter: Punnadi Gunarathna I got the below exception while trying to spawn multi tenant WSO2AS cartridge: TID: [0] [SCC] [2015-08-25 14:53:00,707] INFO {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Creating kubernetes service: [cluster] asdevelopment.punasdevelopment.punasdevelopment.domain [service] service-1 [protocol] http [service-port] 30000 [container-port] 9763 TID: [0] [SCC] [2015-08-25 14:53:01,098] WARN {org.apache.cxf.phase.PhaseInterceptorChain} - Interceptor for {http://192.168.19.51:8080}WebClient has thrown exception, unwinding now org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224) at com.sun.proxy.$Proxy87.createService(Unknown Source) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430) at org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.net.ConnectException: ConnectException invoking http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection refused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ... 16 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210) at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320) ... 19 more TID: [0] [SCC] [2015-08-25 14:53:01,102] ERROR {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Could not create kubernetes service: [service-id] service-1 [service-name] asdevelopment-punasdevelopment-punasdevelopment-domain [node-port] 30000 [container-port-name] http-9763 [container-port] 9763 javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection refused at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:676) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224) at com.sun.proxy.$Proxy87.createService(Unknown Source) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430) at org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.net.ConnectException: ConnectException invoking http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection refused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674) ... 13 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210) at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320) ... 19 more TID: [0] [SCC] [2015-08-25 14:53:01,191] ERROR {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Could not start container: [application] asdevelopment [cartridge] punasdevelopment [member] asdevelopment.punasdevelopment.punasdevelopment.domain445288ff-b028-42cd-97c3-327245226876 org.apache.stratos.kubernetes.client.exceptions.KubernetesClientException: Could not create kubernetes service: [service-id] service-1 [service-name] asdevelopment-punasdevelopment-punasdevelopment-domain [node-port] 30000 [container-port-name] http-9763 [container-port] 9763 at org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:229) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.createKubernetesServices(KubernetesIaas.java:529) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:223) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:127) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection refused at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:676) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224) at com.sun.proxy.$Proxy87.createService(Unknown Source) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:440) at io.fabric8.kubernetes.api.KubernetesClient.createService(KubernetesClient.java:430) at org.apache.stratos.kubernetes.client.KubernetesApiClient.createService(KubernetesApiClient.java:217) ... 8 more Caused by: java.net.ConnectException: ConnectException invoking http://192.168.19.51:8080/api/v1/namespaces/default/services: Connection refused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1364) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:674) ... 13 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:183) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1307) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1267) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:210) at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1320) ... 19 more Kubernetes cluster definition and stratos log are attached herewith. -- This message was sent by Atlassian JIRA (v6.3.4#6332)