RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
By the way, we also see a generous amount of warnings in Zookeeper's logs. Are these related? An indication of what? Thanks, Markus 2017-10-19 08:57:35,583 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@368] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15e1925fb7e3748, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:748) 2017-10-19 08:57:35,583 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1044] - Closed socket connection for client /xxx.xxx.xxx.xxx:41312 which had sessionid 0x15e1925fb7e3748 -Original message- > From:Markus Jelsma > Sent: Thursday 19th October 2017 13:45 > To: solr-user@lucene.apache.org > Subject: RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > Hello, > > We are having this problem again, now it affects the front-end too, the logs > are littered with Zookeeper connection log lines at WARN level. > > Is it expected that i have to deal with this problem myself? Isn't SolrJ or > HTTPClient even going to guarantee me that they will handle underlying > connection problems? > > If i have to deal with it myself, is it just a case of catching > IllegalStateException and closing and reconnecting SolrClient? > > Thanks, > Markus > > -Original message- > > From:Shawn Heisey > > Sent: Tuesday 18th July 2017 16:18 > > To: solr-user@lucene.apache.org > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > > > On 7/18/2017 5:10 AM, Markus Jelsma wrote: > > > The problem was never resolved but Shawn asked for the stack trace, here > > > it is: > > > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > > > As I suspected, it is the connection pool inside HttpClient that is shut > > down (closed). > > > > Earlier today before I came into the office, I asked the HttpClient user > > list whether this could ever happen for a reason other than an explicit > > close/shutdown. They looked at the code and found that the exception > > only is thrown if the "isShutDown" boolean flag is true, and the only > > place that ever gets set to true is when an explicit shutdown is called > > on the connection pool. > > > > When a solr client is built without an external HttpClient, calling > > close() on the solr client will shut down the internal HttpClient. If > > an external HttpClient is used, the user code would need to shut it down > > for this to happen. Recent versions of SolrJ are using > > CloseableHttpClient, which will shut down the connection pool if close() > > is called. > > > > It's looking like this error has happened because the HttpClient object > > inside the solr client has been shut down explicitly, which might have > > happened because one of the outer layers had close() called. > > > > Thanks, > > Shawn > > > > >
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Hello, We are having this problem again, now it affects the front-end too, the logs are littered with Zookeeper connection log lines at WARN level. Is it expected that i have to deal with this problem myself? Isn't SolrJ or HTTPClient even going to guarantee me that they will handle underlying connection problems? If i have to deal with it myself, is it just a case of catching IllegalStateException and closing and reconnecting SolrClient? Thanks, Markus -Original message- > From:Shawn Heisey > Sent: Tuesday 18th July 2017 16:18 > To: solr-user@lucene.apache.org > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > On 7/18/2017 5:10 AM, Markus Jelsma wrote: > > The problem was never resolved but Shawn asked for the stack trace, here it > > is: > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > As I suspected, it is the connection pool inside HttpClient that is shut > down (closed). > > Earlier today before I came into the office, I asked the HttpClient user > list whether this could ever happen for a reason other than an explicit > close/shutdown. They looked at the code and found that the exception > only is thrown if the "isShutDown" boolean flag is true, and the only > place that ever gets set to true is when an explicit shutdown is called > on the connection pool. > > When a solr client is built without an external HttpClient, calling > close() on the solr client will shut down the internal HttpClient. If > an external HttpClient is used, the user code would need to shut it down > for this to happen. Recent versions of SolrJ are using > CloseableHttpClient, which will shut down the connection pool if close() > is called. > > It's looking like this error has happened because the HttpClient object > inside the solr client has been shut down explicitly, which might have > happened because one of the outer layers had close() called. > > Thanks, > Shawn > >
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
On 7/18/2017 5:10 AM, Markus Jelsma wrote: > The problem was never resolved but Shawn asked for the stack trace, here it > is: > Caused by: java.lang.IllegalStateException: Connection pool shut down > at org.apache.http.util.Asserts.check(Asserts.java:34) As I suspected, it is the connection pool inside HttpClient that is shut down (closed). Earlier today before I came into the office, I asked the HttpClient user list whether this could ever happen for a reason other than an explicit close/shutdown. They looked at the code and found that the exception only is thrown if the "isShutDown" boolean flag is true, and the only place that ever gets set to true is when an explicit shutdown is called on the connection pool. When a solr client is built without an external HttpClient, calling close() on the solr client will shut down the internal HttpClient. If an external HttpClient is used, the user code would need to shut it down for this to happen. Recent versions of SolrJ are using CloseableHttpClient, which will shut down the connection pool if close() is called. It's looking like this error has happened because the HttpClient object inside the solr client has been shut down explicitly, which might have happened because one of the outer layers had close() called. Thanks, Shawn
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Hello Susheel, Yes, the closing happens only at the end of the checking cycle. I asked my colleague about the firewall and he is positive everything is allowed between those nodes. I also cannot completely drop the firewall between those nodes to be sure, because the problem is very hard to reproduce; it pops up once in a while, sometimes not for weeks, today already a couple of times. It is locally unreproducible but we're going to try to reproduce it in our development environment. So i have to get back to the problem in a few weeks from now. Number of requests is sometimes, briefly, very high. Usually very low. These specific checks are executed in order, not concurrently. Thanks, Markus -Original message- > From:Susheel Kumar > Sent: Tuesday 18th July 2017 15:17 > To: solr-user@lucene.apache.org > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > Then most likely its due to closing of connection as mentioned above though > you said it's not happening in that part of your code. To rule out > firewall possibility, you can test in some other/local env. Also how many > requests/client/connections happening concurrently. > > Thanks, > Susheel > > On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma > wrote: > > > Hello Susheel, > > > > No, nothing at all. I've check all six nodes, they are clean. > > > > Thanks, > > Markus > > > > > > > > -Original message- > > > From:Susheel Kumar > > > Sent: Tuesday 18th July 2017 14:30 > > > To: solr-user@lucene.apache.org > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > > > > > Do you see any errors etc. in solr.log during this time? > > > > > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma < > > markus.jel...@openindex.io> > > > wrote: > > > > > > > The problem was never resolved but Shawn asked for the stack trace, > > here > > > > it is: > > > > > > > > org.apache.solr.client.solrj.SolrServerException: java.lang. > > IllegalStateException: > > > > Connection pool shut down > > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > doRequest(LBHttpSolrClient.java:485) > > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > > LBHttpSolrClient.java:388) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > > sendRequest(CloudSolrClient.java:1383) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > > > > CloudSolrClient.java:1073) > > > > at org.apache.solr.client.solrj.SolrRequest.process( > > SolrRequest.java:160) > > > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1173) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1090) > > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > > java:1110) > > > > ..internal method calling getById().. > > > > at java.lang.Thread.run(Thread.java:748) > > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > > > at org.apache.http.pool.AbstractConnPool.lease( > > AbstractConnPool.java:184) > > > > at org.apache.http.pool.AbstractConnPool.lease( > > AbstractConnPool.java:217) > > > > at org.apache.http.impl.conn.PoolingClientConnectionManager > > > > .requestConnection(PoolingClientConnectionManager.java:184) > > > > at org.apache.http.impl.client.DefaultRequestDirector.execute( > > > > DefaultRequestDirector.java:415) > > > > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > > > > AbstractHttpClient.java:882) > > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > > CloseableHttpClient.java:82) > > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > > CloseableHttpClient.java:55) > > > > at org.apache.solr.client.solrj.impl.HttpSolrClient. > > > > executeMethod(HttpSolrClient.java:515) > > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > HttpSolrClient.java:279) > > > > at org.apache.solr.c
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Then most likely its due to closing of connection as mentioned above though you said it's not happening in that part of your code. To rule out firewall possibility, you can test in some other/local env. Also how many requests/client/connections happening concurrently. Thanks, Susheel On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma wrote: > Hello Susheel, > > No, nothing at all. I've check all six nodes, they are clean. > > Thanks, > Markus > > > > -Original message- > > From:Susheel Kumar > > Sent: Tuesday 18th July 2017 14:30 > > To: solr-user@lucene.apache.org > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > > > Do you see any errors etc. in solr.log during this time? > > > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma < > markus.jel...@openindex.io> > > wrote: > > > > > The problem was never resolved but Shawn asked for the stack trace, > here > > > it is: > > > > > > org.apache.solr.client.solrj.SolrServerException: java.lang. > IllegalStateException: > > > Connection pool shut down > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > doRequest(LBHttpSolrClient.java:485) > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > LBHttpSolrClient.java:388) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > sendRequest(CloudSolrClient.java:1383) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > > > CloudSolrClient.java:1073) > > > at org.apache.solr.client.solrj.SolrRequest.process( > SolrRequest.java:160) > > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1173) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1090) > > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient. > java:1110) > > > ..internal method calling getById().. > > > at java.lang.Thread.run(Thread.java:748) > > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > > at org.apache.http.pool.AbstractConnPool.lease( > AbstractConnPool.java:184) > > > at org.apache.http.pool.AbstractConnPool.lease( > AbstractConnPool.java:217) > > > at org.apache.http.impl.conn.PoolingClientConnectionManager > > > .requestConnection(PoolingClientConnectionManager.java:184) > > > at org.apache.http.impl.client.DefaultRequestDirector.execute( > > > DefaultRequestDirector.java:415) > > > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > > > AbstractHttpClient.java:882) > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > CloseableHttpClient.java:82) > > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > > CloseableHttpClient.java:55) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient. > > > executeMethod(HttpSolrClient.java:515) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:279) > > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:268) > > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > doRequest(LBHttpSolrClient.java:447) > > > ... 24 more > > > > > > So, to summarize, we have a program checking presence of documents in > Solr > > > using getById() and we don't want this exception to bubble up, we want > > > SolrJ to restore the connection pool just as CloudSolrClient would > move on > > > to another node if one went down in the mean time. > > > > > > Is this possible? How? > > > > > > Many thanks, > > > Markus > > > > > > -Original message- > > > > From:Markus Jelsma > > > > Sent: Thursday 29th June 2017 16:38 > > > > To: solr-user@lucene.apache.org > > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > Thanks. I probably should have mentioned there is no firewall > limiting > > > connections between those hosts. Actually, the processes run on the > same > > > hosts as the Solr cluster is running on. > > > > > > > > Thanks, > > > > M
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Hello Susheel, No, nothing at all. I've check all six nodes, they are clean. Thanks, Markus -Original message- > From:Susheel Kumar > Sent: Tuesday 18th July 2017 14:30 > To: solr-user@lucene.apache.org > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace > > Do you see any errors etc. in solr.log during this time? > > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma > wrote: > > > The problem was never resolved but Shawn asked for the stack trace, here > > it is: > > > > org.apache.solr.client.solrj.SolrServerException: > > java.lang.IllegalStateException: > > Connection pool shut down > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > doRequest(LBHttpSolrClient.java:485) > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > LBHttpSolrClient.java:388) > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > sendRequest(CloudSolrClient.java:1383) > > at org.apache.solr.client.solrj.impl.CloudSolrClient. > > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > > CloudSolrClient.java:1073) > > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173) > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090) > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110) > > ..internal method calling getById().. > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.IllegalStateException: Connection pool shut down > > at org.apache.http.util.Asserts.check(Asserts.java:34) > > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) > > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217) > > at org.apache.http.impl.conn.PoolingClientConnectionManager > > .requestConnection(PoolingClientConnectionManager.java:184) > > at org.apache.http.impl.client.DefaultRequestDirector.execute( > > DefaultRequestDirector.java:415) > > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > > AbstractHttpClient.java:882) > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > CloseableHttpClient.java:82) > > at org.apache.http.impl.client.CloseableHttpClient.execute( > > CloseableHttpClient.java:55) > > at org.apache.solr.client.solrj.impl.HttpSolrClient. > > executeMethod(HttpSolrClient.java:515) > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > HttpSolrClient.java:279) > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > HttpSolrClient.java:268) > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > doRequest(LBHttpSolrClient.java:447) > > ... 24 more > > > > So, to summarize, we have a program checking presence of documents in Solr > > using getById() and we don't want this exception to bubble up, we want > > SolrJ to restore the connection pool just as CloudSolrClient would move on > > to another node if one went down in the mean time. > > > > Is this possible? How? > > > > Many thanks, > > Markus > > > > -Original message- > > > From:Markus Jelsma > > > Sent: Thursday 29th June 2017 16:38 > > > To: solr-user@lucene.apache.org > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > > > > > Thanks. I probably should have mentioned there is no firewall limiting > > connections between those hosts. Actually, the processes run on the same > > hosts as the Solr cluster is running on. > > > > > > Thanks, > > > Markus > > > > > > > > > > > > -Original message- > > > > From:Alexandre Rafalovitch > > > > Sent: Thursday 29th June 2017 15:38 > > > > To: solr-user > > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > > > > One thing to check is whether there is a firewall between the client > > > > and the server. They - sometimes - cut the silent connections in the > > > > _middle_ (at the firewall). The usual solution is keepAlive request of > > > > some kind or not using the connection pool. > > > > > > > > One way to check is with network tracer like Wireshark and checking > > > > whether the actual hardware at the other end of the connection is a > > > > normal server or s
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Do you see any errors etc. in solr.log during this time? On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma wrote: > The problem was never resolved but Shawn asked for the stack trace, here > it is: > > org.apache.solr.client.solrj.SolrServerException: > java.lang.IllegalStateException: > Connection pool shut down > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > doRequest(LBHttpSolrClient.java:485) > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > LBHttpSolrClient.java:388) > at org.apache.solr.client.solrj.impl.CloudSolrClient. > sendRequest(CloudSolrClient.java:1383) > at org.apache.solr.client.solrj.impl.CloudSolrClient. > requestWithRetryOnStaleState(CloudSolrClient.java:1134) > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > CloudSolrClient.java:1073) > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173) > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090) > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110) > ..internal method calling getById().. > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.IllegalStateException: Connection pool shut down > at org.apache.http.util.Asserts.check(Asserts.java:34) > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217) > at org.apache.http.impl.conn.PoolingClientConnectionManager > .requestConnection(PoolingClientConnectionManager.java:184) > at org.apache.http.impl.client.DefaultRequestDirector.execute( > DefaultRequestDirector.java:415) > at org.apache.http.impl.client.AbstractHttpClient.doExecute( > AbstractHttpClient.java:882) > at org.apache.http.impl.client.CloseableHttpClient.execute( > CloseableHttpClient.java:82) > at org.apache.http.impl.client.CloseableHttpClient.execute( > CloseableHttpClient.java:55) > at org.apache.solr.client.solrj.impl.HttpSolrClient. > executeMethod(HttpSolrClient.java:515) > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > HttpSolrClient.java:279) > at org.apache.solr.client.solrj.impl.HttpSolrClient.request( > HttpSolrClient.java:268) > at org.apache.solr.client.solrj.impl.LBHttpSolrClient. > doRequest(LBHttpSolrClient.java:447) > ... 24 more > > So, to summarize, we have a program checking presence of documents in Solr > using getById() and we don't want this exception to bubble up, we want > SolrJ to restore the connection pool just as CloudSolrClient would move on > to another node if one went down in the mean time. > > Is this possible? How? > > Many thanks, > Markus > > -Original message- > > From:Markus Jelsma > > Sent: Thursday 29th June 2017 16:38 > > To: solr-user@lucene.apache.org > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > > > Thanks. I probably should have mentioned there is no firewall limiting > connections between those hosts. Actually, the processes run on the same > hosts as the Solr cluster is running on. > > > > Thanks, > > Markus > > > > > > > > -Original message- > > > From:Alexandre Rafalovitch > > > Sent: Thursday 29th June 2017 15:38 > > > To: solr-user > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > > > One thing to check is whether there is a firewall between the client > > > and the server. They - sometimes - cut the silent connections in the > > > _middle_ (at the firewall). The usual solution is keepAlive request of > > > some kind or not using the connection pool. > > > > > > One way to check is with network tracer like Wireshark and checking > > > whether the actual hardware at the other end of the connection is a > > > normal server or some sort of unexpected hardware piece of equipment > > > (firewall). Yes, that's using the hammer to swat a fly :-) > > > > > > Regards, > > >Alex. > > > > > > http://www.solr-start.com/ - Resources for Solr users, new and > experienced > > > > > > > > > On 29 June 2017 at 08:21, Markus Jelsma > wrote: > > > > Hi, > > > > > > > > Everything is 6.6.0. I could include a stack trace (i don't print > them in my program), but that would only be the the trace from getById() to > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that > what you're looking for? > > > > > &
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
The problem was never resolved but Shawn asked for the stack trace, here it is: org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException: Connection pool shut down at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:485) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090) at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110) ..internal method calling getById().. at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217) at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447) ... 24 more So, to summarize, we have a program checking presence of documents in Solr using getById() and we don't want this exception to bubble up, we want SolrJ to restore the connection pool just as CloudSolrClient would move on to another node if one went down in the mean time. Is this possible? How? Many thanks, Markus -Original message- > From:Markus Jelsma > Sent: Thursday 29th June 2017 16:38 > To: solr-user@lucene.apache.org > Subject: RE: SolrJ 6.6.0 Connection pool shutdown > > Thanks. I probably should have mentioned there is no firewall limiting > connections between those hosts. Actually, the processes run on the same > hosts as the Solr cluster is running on. > > Thanks, > Markus > > > > -Original message- > > From:Alexandre Rafalovitch > > Sent: Thursday 29th June 2017 15:38 > > To: solr-user > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > > > One thing to check is whether there is a firewall between the client > > and the server. They - sometimes - cut the silent connections in the > > _middle_ (at the firewall). The usual solution is keepAlive request of > > some kind or not using the connection pool. > > > > One way to check is with network tracer like Wireshark and checking > > whether the actual hardware at the other end of the connection is a > > normal server or some sort of unexpected hardware piece of equipment > > (firewall). Yes, that's using the hammer to swat a fly :-) > > > > Regards, > > Alex. > > > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > > > > On 29 June 2017 at 08:21, Markus Jelsma wrote: > > > Hi, > > > > > > Everything is 6.6.0. I could include a stack trace (i don't print them in > > > my program), but that would only be the the trace from getById() to > > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that > > > what you're looking for? > > > > > > We haven't called close() in that particular part of the program. > > > > > > Method requestWithRetryOnStaleState has some retry logic built-in but > > > doesn't seem to work for the exception i got. > > > > > > I'll let it print the stack trace and get back if it happens again. > > > > > > Thanks, > > > Markus > > > > > > -Original message- > > >> From:Shawn Heisey >
RE: SolrJ 6.6.0 Connection pool shutdown
Thanks. I probably should have mentioned there is no firewall limiting connections between those hosts. Actually, the processes run on the same hosts as the Solr cluster is running on. Thanks, Markus -Original message- > From:Alexandre Rafalovitch > Sent: Thursday 29th June 2017 15:38 > To: solr-user > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > One thing to check is whether there is a firewall between the client > and the server. They - sometimes - cut the silent connections in the > _middle_ (at the firewall). The usual solution is keepAlive request of > some kind or not using the connection pool. > > One way to check is with network tracer like Wireshark and checking > whether the actual hardware at the other end of the connection is a > normal server or some sort of unexpected hardware piece of equipment > (firewall). Yes, that's using the hammer to swat a fly :-) > > Regards, >Alex. > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 29 June 2017 at 08:21, Markus Jelsma wrote: > > Hi, > > > > Everything is 6.6.0. I could include a stack trace (i don't print them in > > my program), but that would only be the the trace from getById() to > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that what > > you're looking for? > > > > We haven't called close() in that particular part of the program. > > > > Method requestWithRetryOnStaleState has some retry logic built-in but > > doesn't seem to work for the exception i got. > > > > I'll let it print the stack trace and get back if it happens again. > > > > Thanks, > > Markus > > > > -Original message- > >> From:Shawn Heisey > >> Sent: Tuesday 27th June 2017 23:02 > >> To: solr-user@lucene.apache.org > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown > >> > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote: > >> > We have a proces checking presence of many documents in a collection, > >> > just a simple client.getById(id). It sometimes begins throwing lots of > >> > these exceptions in a row: > >> > > >> > org.apache.solr.client.solrj.SolrServerException: > >> > java.lang.IllegalStateException: Connection pool shut down > >> > > >> > Then, as suddenly as it appeared, it's gone again a no longer a problem. > >> > I would expect SolrJ not to throw this but to wait until it the > >> > connection pool, or whatever mechanism is there, to recover. > >> > > >> > Did i miss a magic parameter for SolrJ?\ > >> > >> That error message will be much longer than what you've provided here. > >> It will have a java stacktrace that's typically a dozen or so lines > >> long. There may also be one or more "Caused by" sections after the > >> stacktrace, each with a stacktrace of its own. Can you share the full > >> error message? Is the server also running 6.6.0, or a different version? > >> > >> It would also be helpful if you can share the SolrJ code you've written, > >> cleanly redacted to remove anything sensitive. > >> > >> That particular message ("Connection pool shut down") sounds like it > >> probably came from HttpClient, which SolrJ uses ... and I would expect > >> that to only happen if you close/shutdown the HttpClient or the > >> SolrClient. After closing, a client can't be used any more. Normally > >> the only time you should close a client is right before exiting the > >> program, although if the program's about to exit, it's generally > >> unnecessary, so in my opinion for *most* usages, closing the client > >> likely never needs to happen. > >> > >> Thanks, > >> Shawn > >> > >> >
Re: SolrJ 6.6.0 Connection pool shutdown
One thing to check is whether there is a firewall between the client and the server. They - sometimes - cut the silent connections in the _middle_ (at the firewall). The usual solution is keepAlive request of some kind or not using the connection pool. One way to check is with network tracer like Wireshark and checking whether the actual hardware at the other end of the connection is a normal server or some sort of unexpected hardware piece of equipment (firewall). Yes, that's using the hammer to swat a fly :-) Regards, Alex. http://www.solr-start.com/ - Resources for Solr users, new and experienced On 29 June 2017 at 08:21, Markus Jelsma wrote: > Hi, > > Everything is 6.6.0. I could include a stack trace (i don't print them in my > program), but that would only be the the trace from getById() to > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that what > you're looking for? > > We haven't called close() in that particular part of the program. > > Method requestWithRetryOnStaleState has some retry logic built-in but doesn't > seem to work for the exception i got. > > I'll let it print the stack trace and get back if it happens again. > > Thanks, > Markus > > -Original message- >> From:Shawn Heisey >> Sent: Tuesday 27th June 2017 23:02 >> To: solr-user@lucene.apache.org >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown >> >> On 6/27/2017 6:50 AM, Markus Jelsma wrote: >> > We have a proces checking presence of many documents in a collection, just >> > a simple client.getById(id). It sometimes begins throwing lots of these >> > exceptions in a row: >> > >> > org.apache.solr.client.solrj.SolrServerException: >> > java.lang.IllegalStateException: Connection pool shut down >> > >> > Then, as suddenly as it appeared, it's gone again a no longer a problem. I >> > would expect SolrJ not to throw this but to wait until it the connection >> > pool, or whatever mechanism is there, to recover. >> > >> > Did i miss a magic parameter for SolrJ?\ >> >> That error message will be much longer than what you've provided here. >> It will have a java stacktrace that's typically a dozen or so lines >> long. There may also be one or more "Caused by" sections after the >> stacktrace, each with a stacktrace of its own. Can you share the full >> error message? Is the server also running 6.6.0, or a different version? >> >> It would also be helpful if you can share the SolrJ code you've written, >> cleanly redacted to remove anything sensitive. >> >> That particular message ("Connection pool shut down") sounds like it >> probably came from HttpClient, which SolrJ uses ... and I would expect >> that to only happen if you close/shutdown the HttpClient or the >> SolrClient. After closing, a client can't be used any more. Normally >> the only time you should close a client is right before exiting the >> program, although if the program's about to exit, it's generally >> unnecessary, so in my opinion for *most* usages, closing the client >> likely never needs to happen. >> >> Thanks, >> Shawn >> >>
RE: SolrJ 6.6.0 Connection pool shutdown
Hi, Everything is 6.6.0. I could include a stack trace (i don't print them in my program), but that would only be the the trace from getById() to CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that what you're looking for? We haven't called close() in that particular part of the program. Method requestWithRetryOnStaleState has some retry logic built-in but doesn't seem to work for the exception i got. I'll let it print the stack trace and get back if it happens again. Thanks, Markus -Original message- > From:Shawn Heisey > Sent: Tuesday 27th June 2017 23:02 > To: solr-user@lucene.apache.org > Subject: Re: SolrJ 6.6.0 Connection pool shutdown > > On 6/27/2017 6:50 AM, Markus Jelsma wrote: > > We have a proces checking presence of many documents in a collection, just > > a simple client.getById(id). It sometimes begins throwing lots of these > > exceptions in a row: > > > > org.apache.solr.client.solrj.SolrServerException: > > java.lang.IllegalStateException: Connection pool shut down > > > > Then, as suddenly as it appeared, it's gone again a no longer a problem. I > > would expect SolrJ not to throw this but to wait until it the connection > > pool, or whatever mechanism is there, to recover. > > > > Did i miss a magic parameter for SolrJ?\ > > That error message will be much longer than what you've provided here. > It will have a java stacktrace that's typically a dozen or so lines > long. There may also be one or more "Caused by" sections after the > stacktrace, each with a stacktrace of its own. Can you share the full > error message? Is the server also running 6.6.0, or a different version? > > It would also be helpful if you can share the SolrJ code you've written, > cleanly redacted to remove anything sensitive. > > That particular message ("Connection pool shut down") sounds like it > probably came from HttpClient, which SolrJ uses ... and I would expect > that to only happen if you close/shutdown the HttpClient or the > SolrClient. After closing, a client can't be used any more. Normally > the only time you should close a client is right before exiting the > program, although if the program's about to exit, it's generally > unnecessary, so in my opinion for *most* usages, closing the client > likely never needs to happen. > > Thanks, > Shawn > >
Re: SolrJ 6.6.0 Connection pool shutdown
On 6/27/2017 6:50 AM, Markus Jelsma wrote: > We have a proces checking presence of many documents in a collection, just a > simple client.getById(id). It sometimes begins throwing lots of these > exceptions in a row: > > org.apache.solr.client.solrj.SolrServerException: > java.lang.IllegalStateException: Connection pool shut down > > Then, as suddenly as it appeared, it's gone again a no longer a problem. I > would expect SolrJ not to throw this but to wait until it the connection > pool, or whatever mechanism is there, to recover. > > Did i miss a magic parameter for SolrJ?\ That error message will be much longer than what you've provided here. It will have a java stacktrace that's typically a dozen or so lines long. There may also be one or more "Caused by" sections after the stacktrace, each with a stacktrace of its own. Can you share the full error message? Is the server also running 6.6.0, or a different version? It would also be helpful if you can share the SolrJ code you've written, cleanly redacted to remove anything sensitive. That particular message ("Connection pool shut down") sounds like it probably came from HttpClient, which SolrJ uses ... and I would expect that to only happen if you close/shutdown the HttpClient or the SolrClient. After closing, a client can't be used any more. Normally the only time you should close a client is right before exiting the program, although if the program's about to exit, it's generally unnecessary, so in my opinion for *most* usages, closing the client likely never needs to happen. Thanks, Shawn
SolrJ 6.6.0 Connection pool shutdown
Hi, We have a proces checking presence of many documents in a collection, just a simple client.getById(id). It sometimes begins throwing lots of these exceptions in a row: org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException: Connection pool shut down Then, as suddenly as it appeared, it's gone again a no longer a problem. I would expect SolrJ not to throw this but to wait until it the connection pool, or whatever mechanism is there, to recover. Did i miss a magic parameter for SolrJ? Thanks, Markus
Re: Connection pool shutdown error
I bet the problem is how the SolrServer instance is used within Spring Repository. I think somewhere you should alternatively - explicitly close the client each time. - reuse the same instance (and finally close that) But being a Spring newbie I cannot give you further information. Best, Andrea On 03/19/2015 02:18 PM, phi...@free.fr wrote: Hello, I am trying to use the 4.9.1 SOLR Core API and the 1.3.2.RELEASE version of the Spring Data SOLR API, to connect to a SOLR server, but to no avail. When I run Java application, I get the following errors: --- Exception in thread "main" org.springframework.data.solr.UncategorizedSolrException: Error executing query; nested exception is org.apache.solr.client.solrj.SolrServerException: Error executing query ... Caused by: java.lang.IllegalStateException: Connection pool shut down - I have tried changing Core API version (4.3.0, 4.4.0, ...) but to no avail. Any help would be much appreciated. Cheers, Philippe Here's my Solr Context: package com.myco.archives.SolrGuiMain; @Configuration @EnableSolrRepositories(basePackages = { "com.myco.archives" }, multicoreSupport = false) @ComponentScan public class SolrContext { private final StringHTTP_SEARCHARCHIVES = "http://mysolr.com:8990/solr/collection3";; @Bean public SolrServer solrServer() { SolrServer server = new HttpSolrServer(HTTP_SEARCHARCHIVES); return server; } @Bean public SolrOperations solrTemplate() { return new SolrTemplate(solrServer()); } } - Here's my Repository Class: import org.springframework.data.repository.CrudRepository; public interface ArchiveDocumentRepository extends CrudRepository { List findByText(String text); List findByYmd(Date ymd); } And here's my App: import org.springframework.context.annotation.AnnotationConfigApplicationContext; public class App { private ArchiveDocumentRepository archiveDocumentRepository; public App() { setContext(); processDocs(); } public static void main(String[] args) { new App(); } public void setContext() throws RuntimeException { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(SolrContext.class); if (context != null) { setArchiveDocumentRepository(context.getBean(ArchiveDocumentRepository.class)); } context.close(); } public final ArchiveDocumentRepository getArchiveDocumentRepository() { return archiveDocumentRepository; } public final void setArchiveDocumentRepository(ArchiveDocumentRepository archiveDocumentRepository) { this.archiveDocumentRepository = archiveDocumentRepository; } public void processDocs() { Iterable docs = getArchiveDocumentRepository().findAll(); for (Document doc : docs) { System.out.println("doc count = " + doc.getYmd()); } } } ---
Connection pool shutdown error
Hello, I am trying to use the 4.9.1 SOLR Core API and the 1.3.2.RELEASE version of the Spring Data SOLR API, to connect to a SOLR server, but to no avail. When I run Java application, I get the following errors: --- Exception in thread "main" org.springframework.data.solr.UncategorizedSolrException: Error executing query; nested exception is org.apache.solr.client.solrj.SolrServerException: Error executing query ... Caused by: java.lang.IllegalStateException: Connection pool shut down - I have tried changing Core API version (4.3.0, 4.4.0, ...) but to no avail. Any help would be much appreciated. Cheers, Philippe Here's my Solr Context: package com.myco.archives.SolrGuiMain; @Configuration @EnableSolrRepositories(basePackages = { "com.myco.archives" }, multicoreSupport = false) @ComponentScan public class SolrContext { private final StringHTTP_SEARCHARCHIVES = "http://mysolr.com:8990/solr/collection3";; @Bean public SolrServer solrServer() { SolrServer server = new HttpSolrServer(HTTP_SEARCHARCHIVES); return server; } @Bean public SolrOperations solrTemplate() { return new SolrTemplate(solrServer()); } } - Here's my Repository Class: import org.springframework.data.repository.CrudRepository; public interface ArchiveDocumentRepository extends CrudRepository { List findByText(String text); List findByYmd(Date ymd); } And here's my App: import org.springframework.context.annotation.AnnotationConfigApplicationContext; public class App { private ArchiveDocumentRepository archiveDocumentRepository; public App() { setContext(); processDocs(); } public static void main(String[] args) { new App(); } public void setContext() throws RuntimeException { AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(SolrContext.class); if (context != null) { setArchiveDocumentRepository(context.getBean(ArchiveDocumentRepository.class)); } context.close(); } public final ArchiveDocumentRepository getArchiveDocumentRepository() { return archiveDocumentRepository; } public final void setArchiveDocumentRepository(ArchiveDocumentRepository archiveDocumentRepository) { this.archiveDocumentRepository = archiveDocumentRepository; } public void processDocs() { Iterable docs = getArchiveDocumentRepository().findAll(); for (Document doc : docs) { System.out.println("doc count = " + doc.getYmd()); } } } ---
Re: DIH using a connection pool
Yes, this is the right way for the DIH. You might find it easier to write a separate local client that polls the DB and uploads changes. The DIH is oriented toward longer batch jobs. On Thu, May 24, 2012 at 7:29 AM, Esteban Donato wrote: > Hi community, > > I am using Solr with DIH to index content from a DB. The point is > that I have to configure DIH to check changes in the DB very > frequently (aprox 1 sec) to maintain the index almost up-to-date. I > noted that JDBCDataSource closes the DB connection after every > execution which is not acceptable with this update rate. Ideally I > would need DIH using a connection pool. Looking at DIH code and faq I > noticed that I can configure a connection pool and expose it via jndi > for the JDBCDataSource to use it. My question is: is this they way to > go for integrating a connection pool with DIH? > > Thanks > Esteban -- Lance Norskog goks...@gmail.com
DIH using a connection pool
Hi community, I am using Solr with DIH to index content from a DB. The point is that I have to configure DIH to check changes in the DB very frequently (aprox 1 sec) to maintain the index almost up-to-date. I noted that JDBCDataSource closes the DB connection after every execution which is not acceptable with this update rate. Ideally I would need DIH using a connection pool. Looking at DIH code and faq I noticed that I can configure a connection pool and expose it via jndi for the JDBCDataSource to use it. My question is: is this they way to go for integrating a connection pool with DIH? Thanks Esteban
Re: Connection Pool
Do multiple calls with your client program. So, curl _file1_ & curl _file2_ & curl _file3_ & curl _file4_ & wait; wait; wait; wait On Sun, May 16, 2010 at 8:20 AM, Monmohan Singh wrote: > Sorry for hijacking the thread, but I have an additional question > Is there a way to achieve similar performance (SUSS like) when targeting > extract request handler (/update/extract)? > I guess one way can be to extract content on the client side and then use > SUSS to send update request but then extraction needs to be taken care of > locally in an asynchronous/batch manner. > Regards > Monmohan > > On Sun, May 16, 2010 at 5:19 AM, Lance Norskog wrote: > >> Connection spooling is specified by the underlying apache commons >> connection manager when you create the Server. >> >> The SUSS does socket pooling by default and is the preferred way to do >> concurrent indexing. There are some quirks in the Server >> implementation set, and SUSS avoids them. Unless you are willing to >> root around in the SolrJ Server code and understand exactly how it >> works, stay with the SUSS. >> >> On Fri, May 14, 2010 at 6:44 AM, gabriele renzi wrote: >> > On Fri, May 14, 2010 at 3:35 PM, Anderson vasconcelos >> > wrote: >> >> Hi >> >> I wanna to know if has any connection pool client to manage the >> connections >> >> with solr. In my system, we have a lot of concurrency index request. I >> cant >> >> shared my connection, i need to create one per transaction. But if i >> create >> >> one per transaction, i think the performance will down. >> >> >> >> How you resolve this problem? >> > >> > The commonsHttpSolrServer class does connection pooling, and IIRC also >> > the StreamingUpdateSolrServer. >> > >> > >> > >> > -- >> > blog en: http://www.riffraff.info >> > blog it: http://riffraff.blogsome.com >> > >> >> >> >> -- >> Lance Norskog >> goks...@gmail.com >> > -- Lance Norskog goks...@gmail.com
Re: Connection Pool
Sorry for hijacking the thread, but I have an additional question Is there a way to achieve similar performance (SUSS like) when targeting extract request handler (/update/extract)? I guess one way can be to extract content on the client side and then use SUSS to send update request but then extraction needs to be taken care of locally in an asynchronous/batch manner. Regards Monmohan On Sun, May 16, 2010 at 5:19 AM, Lance Norskog wrote: > Connection spooling is specified by the underlying apache commons > connection manager when you create the Server. > > The SUSS does socket pooling by default and is the preferred way to do > concurrent indexing. There are some quirks in the Server > implementation set, and SUSS avoids them. Unless you are willing to > root around in the SolrJ Server code and understand exactly how it > works, stay with the SUSS. > > On Fri, May 14, 2010 at 6:44 AM, gabriele renzi wrote: > > On Fri, May 14, 2010 at 3:35 PM, Anderson vasconcelos > > wrote: > >> Hi > >> I wanna to know if has any connection pool client to manage the > connections > >> with solr. In my system, we have a lot of concurrency index request. I > cant > >> shared my connection, i need to create one per transaction. But if i > create > >> one per transaction, i think the performance will down. > >> > >> How you resolve this problem? > > > > The commonsHttpSolrServer class does connection pooling, and IIRC also > > the StreamingUpdateSolrServer. > > > > > > > > -- > > blog en: http://www.riffraff.info > > blog it: http://riffraff.blogsome.com > > > > > > -- > Lance Norskog > goks...@gmail.com >
Re: Connection Pool
Connection spooling is specified by the underlying apache commons connection manager when you create the Server. The SUSS does socket pooling by default and is the preferred way to do concurrent indexing. There are some quirks in the Server implementation set, and SUSS avoids them. Unless you are willing to root around in the SolrJ Server code and understand exactly how it works, stay with the SUSS. On Fri, May 14, 2010 at 6:44 AM, gabriele renzi wrote: > On Fri, May 14, 2010 at 3:35 PM, Anderson vasconcelos > wrote: >> Hi >> I wanna to know if has any connection pool client to manage the connections >> with solr. In my system, we have a lot of concurrency index request. I cant >> shared my connection, i need to create one per transaction. But if i create >> one per transaction, i think the performance will down. >> >> How you resolve this problem? > > The commonsHttpSolrServer class does connection pooling, and IIRC also > the StreamingUpdateSolrServer. > > > > -- > blog en: http://www.riffraff.info > blog it: http://riffraff.blogsome.com > -- Lance Norskog goks...@gmail.com
Re: Connection Pool
On Fri, May 14, 2010 at 3:35 PM, Anderson vasconcelos wrote: > Hi > I wanna to know if has any connection pool client to manage the connections > with solr. In my system, we have a lot of concurrency index request. I cant > shared my connection, i need to create one per transaction. But if i create > one per transaction, i think the performance will down. > > How you resolve this problem? The commonsHttpSolrServer class does connection pooling, and IIRC also the StreamingUpdateSolrServer. -- blog en: http://www.riffraff.info blog it: http://riffraff.blogsome.com
Connection Pool
Hi I wanna to know if has any connection pool client to manage the connections with solr. In my system, we have a lot of concurrency index request. I cant shared my connection, i need to create one per transaction. But if i create one per transaction, i think the performance will down. How you resolve this problem? Thanks