Dear all, I've been working on my ~3.0.2 ONAP instance 49 days. Policy was working fine until last week. Today, I tried to run a CL operation but alarms do not appear in network.log anymore (they are being published, checked from Postman). Seems policy is trying to get messages from Dmaap without success due to a SSLException.
Policy debug.log file is showing these errors periorically for several topics: [2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] DenialOfServiceTaf: Not processing this transaction: This Transaction is not denied, ms=0.003419 [2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] X509Taf: No Certificate Info on Transaction, ms=0.010566 [2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] BasicHttpTaf: d...@people.osaaf.org authenticated by cached AAF password, ms=0.041259 [2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 INFO [cadi] Authenticated: d...@people.osaaf.org authenticated by cached AAF password from 10.42.216.188:45922 [2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] isUserInRole: d...@people.osaaf.org has org.onap.policy.pdpd.healthcheck|dev-policy-drools-0|get, 0.038811 ms [2019-06-10T13:51:35.773+00:00|INFO|AuthorizationFilter|qtp1528923159-182] user Basic Authorization for d...@people.osaaf.org evaluated on Mon Jun 10 13:51:35 UTC 2019 in role org.onap.policy.pdpd.healthcheck|dev-policy-drools-0|get is authorized to GET [2019-06-10T13:51:35.788+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.788+0000 WARN [cadi] Trans: user=d...@people.osaaf.org[BAth],ip=10.42.216.188,ms=15.877652,validate=0.351069,code=15.510055 *[2019-06-10T13:51:35.901+00:00|ERROR|InlineBusTopicSink|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT]* SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=dcae.policy.shared, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-unauthenticated.DCAE_CL_OUTPUT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=unauthenticated.DCAE_CL_OUTPUT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch *because of* *javax.net.ssl.SSLException* : Received fatal alert: internal_error at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224) at java.lang.Thread.run(Thread.java:748) [2019-06-10T13:51:35.902+00:00|INFO|CambriaConsumerImpl|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] UEB GET /events/unauthenticated.DCAE_CL_OUTPUT/dcae.policy.shared/dev-policy-drools-0?timeout=15000&limit=100 [2019-06-10T13:51:35.902+00:00|WARN|HostSelector|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] All hosts were blacklisted; reverting to full set of hosts. [2019-06-10T13:51:35.902+00:00|INFO|HttpClient|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] GET https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT/dcae.policy.shared/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ... [2019-06-10T13:51:35.910+00:00|WARN|HttpClient|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes [2019-06-10T13:51:35.910+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms. [2019-06-10T13:51:36.165+00:00|ERROR|InlineBusTopicSink|DMAAP-source-SDNR-CL-RSP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=eb1159c8-7e19-4ca0-b49f-366464176e03, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-SDNR-CL-RSP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=SDNR-CL-RSP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224) at java.lang.Thread.run(Thread.java:748) [2019-06-10T13:51:36.165+00:00|INFO|CambriaConsumerImpl|DMAAP-source-SDNR-CL-RSP] UEB GET /events/SDNR-CL-RSP/eb1159c8-7e19-4ca0-b49f-366464176e03/dev-policy-drools-0?timeout=15000&limit=100 [2019-06-10T13:51:36.165+00:00|WARN|HostSelector|DMAAP-source-SDNR-CL-RSP] All hosts were blacklisted; reverting to full set of hosts. [2019-06-10T13:51:36.165+00:00|INFO|HttpClient|DMAAP-source-SDNR-CL-RSP] GET https://message-router:3905/events/SDNR-CL-RSP/eb1159c8-7e19-4ca0-b49f-366464176e03/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ... [2019-06-10T13:51:36.167+00:00|ERROR|InlineBusTopicSink|DMAAP-source-APPC-LCM-WRITE] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=e9a291f0-086d-4bca-b84e-0d9d93dbc180, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-APPC-LCM-WRITE,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=APPC-LCM-WRITE, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224) at java.lang.Thread.run(Thread.java:748) [2019-06-10T13:51:36.167+00:00|INFO|CambriaConsumerImpl|DMAAP-source-APPC-LCM-WRITE] UEB GET /events/APPC-LCM-WRITE/e9a291f0-086d-4bca-b84e-0d9d93dbc180/dev-policy-drools-0?timeout=15000&limit=100 [2019-06-10T13:51:36.167+00:00|WARN|HostSelector|DMAAP-source-APPC-LCM-WRITE] All hosts were blacklisted; reverting to full set of hosts. [2019-06-10T13:51:36.167+00:00|INFO|HttpClient|DMAAP-source-APPC-LCM-WRITE] GET https://message-router:3905/events/APPC-LCM-WRITE/e9a291f0-086d-4bca-b84e-0d9d93dbc180/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ... [2019-06-10T13:51:36.172+00:00|WARN|HttpClient|DMAAP-source-APPC-LCM-WRITE] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes [2019-06-10T13:51:36.172+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-APPC-LCM-WRITE] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms. [2019-06-10T13:51:36.173+00:00|WARN|HttpClient|DMAAP-source-SDNR-CL-RSP] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes [2019-06-10T13:51:36.173+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-SDNR-CL-RSP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms. [2019-06-10T13:51:39.929+00:00|ERROR|InlineBusTopicSink|DMAAP-source-PDPD-CONFIGURATION] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=12e77824-cba7-48dc-8251-b80b76529ead, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-PDPD-CONFIGURATION,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=PDPD-CONFIGURATION, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384) at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87) at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64) at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172) at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224) at java.lang.Thread.run(Thread.java:748) [2019-06-10T13:51:39.929+00:00|INFO|CambriaConsumerImpl|DMAAP-source-PDPD-CONFIGURATION] UEB GET /events/PDPD-CONFIGURATION/12e77824-cba7-48dc-8251-b80b76529ead/dev-policy-drools-0?timeout=15000&limit=100 [2019-06-10T13:51:39.929+00:00|WARN|HostSelector|DMAAP-source-PDPD-CONFIGURATION] All hosts were blacklisted; reverting to full set of hosts. [2019-06-10T13:51:39.929+00:00|INFO|HttpClient|DMAAP-source-PDPD-CONFIGURATION] GET https://message-router:3905/events/PDPD-CONFIGURATION/12e77824-cba7-48dc-8251-b80b76529ead/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ... [2019-06-10T13:51:39.936+00:00|WARN|HttpClient|DMAAP-source-PDPD-CONFIGURATION] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes [2019-06-10T13:51:39.936+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-PDPD-CONFIGURATION] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms. [2019-06-10T13:51:44.873+00:00|INFO|HttpClient|DMAAP-source-APPC-CL] --> HTTP/1.1 200 OK [2019-06-10T13:51:44.873+00:00|INFO|CambriaConsumerImpl|DMAAP-source-APPC-CL] UEB GET /events/APPC-CL/5d3db7a2-6611-475e-a259-a6c5e5dfc712/dev-policy-drools-0?timeout=15000&limit=100 [2019-06-10T13:51:44.873+00:00|INFO|HttpClient|DMAAP-source-APPC-CL] GET http://message-router:3904/events/APPC-CL/5d3db7a2-6611-475e-a259-a6c5e5dfc712/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) Are you aware of this issue? I would appreciate so much your kind help on this! Kind regards, Xoan -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17518): https://lists.onap.org/g/onap-discuss/message/17518 Mute This Topic: https://lists.onap.org/mt/32004642/21656 Mute #policyr: https://lists.onap.org/mk?hashtag=policyr&subid=2740164 Mute #casablanca: https://lists.onap.org/mk?hashtag=casablanca&subid=2740164 Mute #oom: https://lists.onap.org/mk?hashtag=oom&subid=2740164 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
