[firebird-support] Firebird 2.5.x leaking memory on long living connections coming from ado.net provider clients

[Dominik Psenner dominik.psen...@topcontrol.it [firebird-support]]

Hi,

after a recent update of the firebird ado.net driver from 5.12.0 we observe 
that the firebird server 2.5.x leaks memory that is related to a connection 
that a client opened using the ado.net provider > 5.12.0.

We observe both an increase in the private memory set but also nonpaged pool 
increases significantly until the server is no longer able to handle any 
statements. Closing the connection in question however properly releases all 
related resources in the firebird server process and the firebird server 
process recovers, going back to normal. This may relates to changes implemented 
with  that before ado.net provider 5.12.1, connections from the ado.net 
provider to the firebird server were closed too early and therefore were always 
short-living. 5.12.1 fixes that and from then on connections are properly 
pooled and therefore connections are from then on long-living as long as there 
is activity. A connection is then taken from the pool, queries are run and the 
connection is returned to the connection pool.

We have observed fbserver.exe to have an np pool of way beyond 5000K and a 
working set way beyond 4GB for a database that is 500MB on disk (and probably 
fits into ram).

While attempting to implement a minimal sample project that reproduces the 
issue we have implemented a minimal application that populates several thousand 
records into a dummy database that contains only one table created as follows:

CREATE TABLE ITEM (
ID BIGINT NOT NULL,
MEASUREMENT DOUBLE PRECISION NOT NULL,
UNIT VARCHAR(10) NOT NULL);

ALTER TABLE ITEM ADD CONSTRAINT PK_ITEM PRIMARY KEY (ID);
CREATE UNIQUE DESCENDING INDEX PK_ITEM_DESC ON ITEM (ID);

CREATE SEQUENCE GEN_ITEM_ID;

SET TERM ^ ;

create trigger item_bi for item
active before insert position 0
as
begin
  if (new.id is null) then
new.id = gen_id(gen_item_id,1);
end
^

SET TERM ; ^

For simplicity I do not how we have populated the table with values. After 
having populated the database, we start the breaking application which works as 
follows:

The application does loop, selecting a value by the primary key in range [1, 
`select max(id) from item`] and then sleeps for a short period of time.

Even only a short period of time later the server begins to grow in np pool 
until it refuses to execute new queries. Soon the breaking application crashes 
with IscError 335544761 "too many open handles to database" and the SQLSTATE is 
"HY000".

Now, there may be two apparent causes for the symptom:


1.   Ado.net provider has been broken since ever and this bug has been 
hidden until the connection pool was fixed with 5.12.1.0

2.   The server code leaks handles, causing the np pool to grow until it 
can no longer handle any request.

Would you please guide us on where to go next? Should we report this to the 
maintainer of the firebird ado net provider? Is this bug in the firebird server 
2.5.x that causes it to leak in some circumstances?

Things that are related to this issue:

· 
https://github.com/cincuranet/FirebirdSql.Data.FirebirdClient/compare/5.12.0...0...5.12.1.0

· http://tracker.firebirdsql.org/browse/DNET/fixforversion/10856

Things that appear to be related to this issue (but might not be related at 
all):

· https://firebirder.ru/problemy-s-icu-podxod-2

· http://tracker.firebirdsql.org/browse/CORE-2642

Firebird Server Versions affected:

· 2.5.3 superserver on windows 7 x64

· 2.5.8 superserver on windows 7 x64

· 2.5.4 classic on windows 2016 x64

Mit freundlichen Grüßen - Distinti saluti - Best regards

Dominik Psenner
Software Developer

Tel:+39 0471 319 999
Fax:   +39 0471 319 990
Email: dominik.psen...@topcontrol.it

[cid:image001.jpg@01D450EF.A4F0FE50]
Enzenbergweg 24/A - Via Enzenberg 24/A
I- 39018 Terlan - Terlano (BZ)
www.topcontrol.it






Die unbefugte Verwendung dieser Mitteilung ist verboten und könnte 
strafrechtlich verfolgt werden. Wer diese Mitteilung irrtümlicherweise erhält 
wird gebeten uns umgehend zu informieren und anschließend die Mitteilung zu 
vernichten. Vielen Dank.
Il presente messaggio è diretto unicamente al destinatario sopra indicato. 
L'utilizzo non autorizzato del presente messaggio è vietato e potrebbe 
costituire reato. Chiunque altro riceva questa comunicazione per errore è 
invitato ad informarci immediatamente ed è tenuto a istruggere quanto ricevuto. 
Grazie per la collaborazione.
Caution: The unauthorized use of this message is prohibited and may be 
prosecuted by law. Anyone who receives this communication in error is requested 
to inform us immediately and then delete the message. Thank you very much for 
your collaboration!


[Non-text portions of this message have been removed]

Re: [firebird-support] Can not create backup of Firebird database because of the errors

[Ann Harrison aharri...@ibphoenix.com [firebird-support]]

On Thu, Sep 20, 2018 at 5:30 AM Georgiy Kovalov g...@liconic.com
[firebird-support]  wrote:

>
>
> At the time of backup firebird database (gbak -g -ig) I have the following
> error:
>
> gbak:writing data for table ORDERS
> gbak: ERROR:message length error (encountered 532, expected 528)
> gbak: ERROR:gds_$receive failed
> gbak:Exiting before completion due to errors
>
> OK.  The first thing to do is to run gbak with the -o and -v options.
That will tell you which table
is causing the error.  Message length errors can be caused by
compression/decompression
problems.  When you've identified the table with the error, you can probe
it to find the bad
record.  Probe with a sorted, non-indexed query and you'll eventually find
a record that will cause
the same error.
same error.  At that point,



> When I'm using *gfix* with different parameters (-v -full, -mend,
> -ignore), I have the message:
>
> Summary of validation errors
> Number of index page errors : 540
>
> Mend is a little dangerous, but essentially it clears invalid pointers to
pages and records, so you get
a null pointer instead of a bad one.  Whatever the bad one pointed to will
go from unavailable to
ready to be removed.  Certainly not the option I would use first.  Index
pages errors are generally
benign.  Firebird indexes can contain pointers to deleted records after a
crash or hard shutdown.
That's expected and eventually fixed if the entry is visited.


> In *firebird.log* file I see the lines:
>
> PC (Server) Thu Sep 20 08:37:01 2018
> Database: E:\...GDB
> Index 2 is corrupt on page 134706 level 1. File: 
> ..\..\..\src\jrd\validation.cpp, line: 1699
>  in table COMPONENTS (197)
>
>
>
That's a problem that can be fixed by rebuilding the index.  However, it's
unlikely to be the source
of your gbak problem because gbak doesn't backup or reference user defined
indexes.

Good luck,

Ann

ODP: [firebird-support] Question about multithread: readonlyquery ina single transaction

[Karol Bieniaszewski liviusliv...@poczta.onet.pl [firebird-support]]

Thank you very much for confirmation 

Performance is not always vital. 
Flexibility many times have bigger priority.

Regards,
Karol Bieniaszewski

Od: Dimitry Sibiryakov s...@ibphoenix.com [firebird-support]
Wysłano: środa, 19 września 2018 20:13
Temat: Re: ODP: [firebird-support] Question about multithread: readonlyquery 
ina single transaction

19.09.2018 13:26, Karol Bieniaszewski liviusliv...@poczta.onet.pl 
[firebird-support] wrote:
> Is something like this safe?

   Yes. But it has no performance difference from the case when everything 
happen in 
single thread.


-- 
   WBR, SD. 

Re: [firebird-support] Can not create backup of Firebird database because of the errors

[liviuslivius liviusliv...@poczta.onet.pl [firebird-support]]

Hi,
first, this was not good concept to use -mend, as you can loos data. If you 
have a copy before this operation bring it back and run gfix without it.
If this is realy index problem you can fix it with simple command Alter index 
indexName active;
This will rebuild your index
Regards,Karol Bieniaszewski
null

[firebird-support] Can not create backup of Firebird database because of the errors

[Georgiy Kovalov g...@liconic.com [firebird-support]]

Hello,

At the time of backup firebird database (gbak -g -ig) I have the 
following error:

|gbak: writing data for table ORDERS gbak: ERROR:message length error 
(encountered 532, expected 528) gbak: ERROR:gds_$receive failed 
gbak:Exiting before completion due to errors |

When I'm using*gfix*with different parameters (-v -full, -mend, 
-ignore), I have the message:

|Summary of validation errors Number of index page errors : 540 |

In*firebird.log*file I see the lines:

|PC (Server) Thu Sep 20 08:37:01 2018 Database: E:\...GDB Index 2 is 
corrupt on page 134706 level 1. File: ..\..\..\src\jrd\validation.cpp, 
line: 1699 in table COMPONENTS (197) |

However, the database works OK without problems.

Please help me to fix the error and make a backup. (I need the backup to 
migrate to on 64bit server).

Thank you!