Try increasing your keep alive for TCP. I have seen wait timeout issues that look similar to what you are seeing but your issue may be different.
> sysctl net.ipv4.tcp_keepalive_time > net.ipv4.tcp_keepalive_time = 7200 > sysctl -w net.ipv4.tcp_keepalive_time=60 > net.ipv4.tcp_keepalive_time = 60 On 8/20/12 6:20 PM, "Matt Goeke" <[email protected]> wrote: >(oozie-users is CC'ed due to this potentially being a general issue >unrelated to CDH) > >All, > >Recently I have been running into issues with our test Oozie server where >random DAGs will fail with the error: >OozieClientException: org.apache.oozie.DagEngineException: E0600: Could >not >get connection, Communications link failureThe last packet successfully >received from the server was 2,411,007 milliseconds ago. The last packet >sent successfully to the server was 23 milliseconds ago. > >Looking into the Oozie logs it looks like it is an issue with the JDBC >connection to the mysql oozie schema but I have tried several things >including swapping out the jdbc connector jar, swapping between >IP/hostname >on jdbc connection string and reinstalling but everytime the issue comes >back. Restarting the server seems to allieviate the issue for 20-30 mins >but obviously there is something I am missing that is either conflicting >on >the same box or a misconfiguration. Also, currently the mysql database and >the oozie server are on the same box so I am assuming this is not a >network >issue. > >oozie-site.xml: > <property> > <name>oozie.service.StoreService.jdbc.driver</name> > <value>com.mysql.jdbc.Driver</value> > </property> > > <property> > <name>oozie.service.StoreService.jdbc.url</name> > <value>jdbc:mysql://{ip}:3306/oozie</value> > </property> > > <property> > <name>oozie.service.StoreService.jdbc.username</name> > <value>{user}</value> > </property> > > <property> > <name>oozie.service.StoreService.jdbc.password</name> > <value>{password}</value> > </property> > >oozie.log: >The last packet successfully received from the server was 3,202,197 >milliseconds ago. The last packet sent successfully to the server was 0 >milliseconds ago. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at >sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc >essorImpl.java:39) > at >sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr >uctorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:409) > at >com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1122) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3056) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2942) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3485) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1960) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2114) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2690) > at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5093) > at >org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConne >ction.java:371) > at >org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAu >toCommit(PoolingDataSource.java:328) > at >org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingC >onnection.java:198) > at >org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingC >onnection.java:198) > at >org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConn >ection.setAutoCommit(ConfiguringConnectionDecorator.java:117) > at >org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConn >ection.<init>(ConfiguringConnectionDecorator.java:111) > at >org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(Config >uringConnectionDecorator.java:93) > at >org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSo >urce.java:100) > at >org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingD >ataSource.java:88) > at >org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreM >anager.java:941) > at >org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.j >ava:926) > ... 25 more >Caused by: java.io.EOFException: Can not read response from server. >Expected to read 4 bytes, read 0 bytes before connection was unexpectedly >lost. > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2503) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2953) > ... 42 more > >Any advice is greatly appreciated! > >-- >Matt
