Hello,
nothing to do with this, Analytics worker again at 400% CPU this morning, I had
to kill -9 the process then restart. It never ran more than 3 days ;-(.
First I have
[2019-02-13 23:53:12,795] WARN
{org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream.
java.net.SocketException: Socket is closed
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at
org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
in the logs (often during the night —> purging ?), then
[2019-02-14 00:01:46,401] WARN {} - IOException occurred while connecting to
pgsql.uclouvain.be:5440<http://pgsql.uclouvain.be:5440> java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-14 00:01:48,660] WARN {} - IOException occurred while connecting to
pgsql.uclouvain.be:5440<http://pgsql.uclouvain.be:5440>
java.net.SocketTimeoutException
at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:69)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-14 00:05:25,337] ERROR {org.postgresql.Driver} - Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
My datasources config for all 4 DBs:
WSO2EI_ANALYTICS_WKR_PERMISSIONS_DB
WSO2EI_ANALYTICS_WKR_EI_ANALYTICS_DB
WSO2EI_ANALYTICS_METRICS_DB
WSO2EI_ANALYTICS_CARBON_DB
definition:
configuration:
jdbcUrl:
'jdbc:postgresql://pgsql.domain:5440/WSO2EI_ANALYTICS_WKR_CARBON_DB'
username: user
password: secret
driverClassName: org.postgresql.Driver
maxPoolSize: 20
idleTimeout: 20000
connectionTestQuery: SELECT 1
validationTimeout: 5000
isAutoCommit: false
type: RDBMS
description: "The datasource used for registry and user manager"
name: WSO2_CARBON_DB
We plan to switch to mySQL
Bernard
Le 7 févr. 2019 à 10:55, Dilini Muthumala
<[email protected]<mailto:[email protected]>> a écrit :
Hi Bernard,
Those are Hikari Connection Pool configurations. You can find documentation on
each of these parameters in 'Configurations' section in [1]. I have quoted
'idleTimeout' and 'validationTimeout' parameter descriptions below in this
email for your convenience.
As for the issue you are getting, it would be better if you could share the
complete log file. However from the looks of it, it seems like a connection
problem. Please see [2].
Thanks,
Dilini
[1] https://github.com/brettwooldridge/HikariCP/tree/HikariCP-2.4.1
[2]
https://stackoverflow.com/questions/15806083/org-postgresql-util-psqlexception-the-connection-attempt-failed
HikariCP Parameter descriptions:
⌚idleTimeout
This property controls the maximum amount of time that a connection is allowed
to sit idle in the pool. Whether a connection is retired as idle or not is
subject to a maximum variation of +30 seconds, and average variation of +15
seconds. A connection will never be retired as idle before this timeout. A
value of 0 means that idle connections are never removed from the pool.
Default: 600000 (10 minutes)
⌚validationTimeout
This property controls the maximum amount of time that a connection will be
tested for aliveness. This value must be less than the connectionTimeout. The
lowest accepted validation timeout is 1000ms (1 second). Default: 5000
On Wed, Feb 6, 2019 at 3:37 PM Bernard Paris
<[email protected]<mailto:[email protected]>> wrote:
Hello,
since this problem is still alive, could anyone give us an idea of what exactly
are
idleTimeout
and
validationTimeout
parameters we are to define in deployment.yaml worker file, or what are their
matching name in TOMCAT connection JDBC pool ?
Thanks
Bernard
[2019-02-01 02:45:23,770] WARN {} - IOException occurred while connecting to
pgsql.hifolks.be:5440<http://pgsql.uclouvain.be:5440/>
java.net.SocketTimeoutException
at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:69)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-01 02:45:31,458] WARN
{org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream.
java.net.SocketException: Socket is closed
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at
org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-01 02:45:31,459] ERROR {org.postgresql.Driver} - Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException
at java.net.SocksSocketImpl.remainingMillis(SocksSocketImpl.java:111)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:69)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
... 14 more
Le 28 janv. 2019 à 10:15, Bernard Paris
<[email protected]<mailto:[email protected]>> a écrit :
Hi devs,
we are evaluating wso2-EI 6.4 in pre-prod environment, running EI with
analytics worker.
I followed the guide at
https://docs.wso2.com/display/SP430/Configuring+Datasources
to configure postgres datasources in deployment.yaml, for all dbs:
METRICS_DB
CARBON_DB
PERMISSIONS_DB
EI_ANALYTICS_DB
Library is postgresql-42.2.2.jar
The system works fine for some days but after a while the analytics worker
stops responding.
I need to manually kill the (worker) process because "carbon.sh stop" doesn't
do the stuff, then restart the worker… that will be ok for 3 or 4 days.
In comparison to datasources of wso2-EI in master-datasources.xml, db connector
attributes are the same, except an extra one "testOnBorrow" set to true.
Unfortunately this attribute is not allowed in deployment.yaml.
Any help should be appreciate to get the system stable.
[2019-01-24 01:48:52,291] INFO {org.wso2.carbon.databridge.core.DataBridge} -
user admin connected
[2019-01-24 01:48:52,291] INFO {org.wso2.carbon.databridge.core.DataBridge} -
user admin connected
[2019-01-24 01:48:52,289] WARN
{org.apache.thrift.transport.TIOStreamTransport} - Error closing output stream.
java.net.SocketException: Socket is closed
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1524)
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1545)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at
org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
at org.apache.thrift.transport.TSocket.close(TSocket.java:196)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:304)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:17,423] WARN {} - IOException occurred while connecting to
pgsql.server.be:5440<http://pgsql.uclouvain.be:5440/> java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:54:22,204] ERROR {org.postgresql.Driver} - Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
... 14 more
[2019-01-24 01:55:53,456] INFO {org.wso2.carbon.databridge.core.DataBridge} -
user admin connected
[2019-01-24 01:55:31,190] WARN {} - IOException occurred while connecting to
pgsql.server.be<http://pgsql.uclouvain.be:5440/>:5440<http://pgsql.uclouvain.be:5440/>
java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-01-24 01:56:09,847] INFO {org.wso2.carbon.databridge.core.DataBridge} -
user admin connected
[2019-01-24 01:55:58,072] WARN {} - IOException occurred while connecting to
pgsql.server.be<http://pgsql.uclouvain.be:5440/>:5440<http://pgsql.uclouvain.be:5440/>
java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:295)
at
org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:453)
at
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:205)
at
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at
com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496)
at com.zaxxer.hikari.pool.HikariPool.access$300(HikariPool.java:69)
at com.zaxxer.hikari.pool.HikariPool$1.run(HikariPool.java:367)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
_______________________________________________
Dev mailing list
[email protected]<mailto:[email protected]>
http://wso2.org/cgi-bin/mailman/listinfo/dev
_______________________________________________
Dev mailing list
[email protected]<mailto:[email protected]>
http://wso2.org/cgi-bin/mailman/listinfo/dev
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
