By placing some strategic debug messages, I have found that the JDBC
connections are not being closed until all <entity> elements have been
processed (in the entire config file). A simplified example would be:
<dataConfig>
<dataSource name="ds1" driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/db1" user="..." password="..." />
<dataSource name="ds2" driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/db2" user="..." password="..." />
<document>
<entity name="entity1" datasource="ds1" ...>
... field list ...
<entity name="entity1a" datasource="ds1" ...>
... field list ...
</entity>
</entity>
<entity name="entity2" datasource="ds2" ...>
... field list ...
<entity name="entity2a" datasource="ds2" ...>
... field list ...
</entity>
</entity>
</document>
</dataConfig>
The behavior is:
JDBC connection opened for entity1 and entity1a - Applicable queries run and
ResultSet objects processed
All open ResultSet and Statement objects closed for entity1 and entity1a
JDBC connection opened for entity2 and entity2a - Applicable queries run and
ResultSet objects processed
All open ResultSet and Statement objects closed for entity2 and entity2a
All JDBC connections (none are closed at this point) are closed.
In my instance, I have some 95 unique <entity> elements (19 parents with 5
children each), resulting in 95 open JDBC connections. If I understand the
process correctly, it should be safe to close the JDBC connection for a
"root" <entity> (immediate children of <document>) and all descendant
<entity> elements once the parent has been successfully completed. I have
been digging around the code, but due to my unfamiliarity with the code, I'm
not sure where this would occur.
Is this a valid solution? It's looking like I should probably open a defect
and I'm willing to do so along with submitting a patch, but need a little
more direction on where the fix would best reside.
Thanks,
Shane
On Mon, Jan 10, 2011 at 7:14 AM, Shane Perry <thry...@gmail.com> wrote:
> Gora,
>
> Thanks for the response. After taking another look, you are correct about
> the hasnext() closing the ResultSet object (1.4.1 as well as 1.4.0). I
> didn't recognize the case difference in the two function calls, so missed
> it. I'll keep looking into the original issue and reply if I find a
> cause/solution.
>
> Shane
>
>
> On Sat, Jan 8, 2011 at 4:04 AM, Gora Mohanty <g...@mimirtech.com> wrote:
>
>> On Sat, Jan 8, 2011 at 1:10 AM, Shane Perry <thry...@gmail.com> wrote:
>> > Hi,
>> >
>> > I am in the process of migrating our system from Postgres 8.4 to Solr
>> > 1.4.1. Our system is fairly complex and as a result, I have had to
>> define
>> > 19 base entities in the data-config.xml definition file. Each of these
>> > entities executes 5 queries. When doing a full-import, as each entity
>> > completes, the server hosting Postgres shows 5 "idle in transaction" for
>> the
>> > entity.
>> >
>> > In digging through the code, I found that the JdbcDataSource wraps the
>> > ResultSet object in a custom ResultSetIterator object, leaving the
>> ResultSet
>> > open. Walking through the code I can't find a close() call anywhere on
>> the
>> > ResultSet. I believe this results in the "idle in transaction"
>> processes.
>> [...]
>>
>> Have not examined the "idle in transaction" issue that you
>> mention, but the ResultSet object in a ResultSetIterator is
>> closed in the private hasnext() method, when there are no
>> more results, or if there is an exception. hasnext() is called
>> by the public hasNext() method that should be used in
>> iterating over the results, so I see no issue there.
>>
>> Regards,
>> Gora
>>
>> P.S. This is from Solr 1.4.0 code, but I would not think that
>> this part of the code would have changed.
>>
>
>
