Re: [Firebird-devel] Reference counters in API
> On the other hand I see no problems with adding that method to our > interfaces, specially if it's needed to make Delphi people life easier. > It does not conflict with our versioning support. Unfortunately it is not enough. To be binary compatible with IUnknown (not with Delphi itself but with well known stantard interface) we should use stdcall also. So, we can be compatible and abandon upgradeInterface, or not compatible and don't add confusion introducing queryInterface. Regards, Vlad -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> On 03/31/11 15:28, Vlad Khorsun wrote: >>> On the other hand I see no problems with adding that method to our >>> interfaces, specially if it's needed to make Delphi people life easier. >>> It does not conflict with our versioning support. >> Unfortunately it is not enough. To be binary compatible with IUnknown >> (not with Delphi itself but with well known stantard interface) we should use >> stdcall also. So, we can be compatible and abandon upgradeInterface, or >> not compatible and don't add confusion introducing queryInterface. > > Vlad, if really needed we can make 3 first functions in our interface to > use stdcall. This breaks nothing. And all other functions still will be cdecl ? Don't looks as beauty to me :( Regards, Vlad PS I'm not insisting on binary compatibility with IUnknown. I just want to make it clear - are we compatible or not. -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> On 03/31/11 15:00, Kjell Rilbe wrote:
>> Generally speaking, I feel it's completely wrong to add something to
>> an api only because some clients/users would expect it, unless it
>> actually does something useful in the api. it will only lead to extra
>> complexity and problems down the line.
>>
>> So, if the only reason to add queryInterface is to make Delphi users'
>> life easier I'd say no, even if I've been a huge Delphi fan ever since
>> version 1.
>
> BTW, what's a problem with writing using Delphi code RAII type for our
> interface without built-in into compiler support?
It is not a *big* problem, no. But it will be very convenient. See
simplified
example below :
1. Not compatible with IUnknown.
a) Our public header
class Interface
{
virtual int _cdecl addRef() = 0;
virtual int _cdecl release() = 0;
};
class IFoo : public Interface
{
virtual void _cdecl xxx() = 0;
};
b) Delphi declarations
type
IFBInterface = class
function addRef : Integer; abstract; cdecl;
function release : Integer; abstract; cdecl;
end;
IFoo = class(IFBInterface)
procedure xxx; abstract; cdecl;
end;
TFBInterface = class(TInterfacedObject, IFBInterface)
private:
// IInterface overloads
function _AddRef : Integer;
function _Release : Integer;
// IFBInterface methods
function addRef : Integer;
function release : Integer;
FFBIntf : IFBInterface;
public:
constructor Create(AFBIntf : IFBInterface);
end;
TFoo = class (TFBInterface, IFoo)
private:
FFoo : IFoo;
public
constructor Create(AFoo : IFoo);
// IFoo methods
procedure xxx;
end;
function TFBInterface._AddRef : Integer;
begin
return addRef;
end;
function TFBInterface._Release : Integer;
begin
return release;
end;
function TFBInterface.addRef : Integer;
begin
return FFBIntf.addRef;
end;
function TFBInterface.release : Integer;
begin
return FFBIntf.release;
end;
c) Delphi code example :
var
aFoo : IFoo;
begin
aFoo := TFoo.Create( fbGetMeFoo; )
aFoo.xxx;
end
2. If we became compatible with IUnknown
a) Our public header
class Interface
{
virtual int _stdcall queryInterface(...) = 0;
virtual int _stdcall addRef() = 0;
virtual int _stdcall release() = 0;
};
class IFoo : public Interface
{
virtual void _stdcall xxx() = 0;
};
b) Delphi declarations
IFoo = interface (IInterface)
procedure xxx; stdcall;
end;
c) Delphi code example :
var
aFoo : IFoo;
begin
aFoo := fbGetMeFoo;
aFoo.xxx;
end
Regards,
Vlad
--
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> 1. FB is multi platform, while IUnknown is inherently tied to Microsoft. > (Or am I wrong about this?) Absolutely wrong. > 2. IUnknown compatibility would imply a lot of other stuff that FB > doesn't want, re. COM etc. Wrong again. > Although that may not be required per se, > just because of IUnknown compatibility, it is in people's minds very > tightly coupled with COM. This is not good. FB is not a COM object and > shouldn't be. Ever. Please. ;-) Nobody going ever think about COM. Regards, Vlad -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
>> Nobody going ever think about COM. > > So, would you say that this article is misleading: > > http://en.wikipedia.org/wiki/IUnknown Article is not misleading. How you read it - probably. It said : In programming, the IUnknown interface is the fundamental interface in the Component Object Model (COM) and not said : COM is fundamental interface for IUnknown and not said COM is required for IUnknown and ever not said IUnknown is impossible without COM Regards, Vlad -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] [FB-Tracker] Created: (CORE-3418) Inactive DB-trigger after Create/Alter Is Active
>> Inactive DB-trigger after Create/Alter Is Active
>>
>>
>> Key: CORE-3418
>> URL: http://tracker.firebirdsql.org/browse/CORE-3418
>> Project: Firebird Core
>> Issue Type: Bug
>> Affects Versions: 2.5.0
>> Reporter: Maxim Kuzmin
>>
>>
>> After execute this query db-trigger is active:
>> CREATE OR ALTER TRIGGER TRG$START
>> INACTIVE ON TRANSACTION START POSITION 0 AS BEGIN
>> RDB$SET_CONTEXT('USER_SESSION', 'TRANS_ID',
>> CURRENT_TRANSACTION); END ^ SET TERM ; ^
>
> Can someone explain/elaborate on what the problem is. Maxim's entry is very
> "parse" on the details or the effect.
After creation of inactive database trigger it is really active.
As for effect - create such trigger and query value of contex variable
'TRANS_ID', it will be not null,
while should be NULL.
Regards,
Vlad
--
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] [FB-Tracker] Created: (CORE-3418) Inactive DB-trigger after Create/Alter Is Active
> Perhaps you could add this detail to the case? > > I'm sure it would help anyone reading the case through the tracker and with > the documentation (once it has been fixed). Done. Regards, Vlad -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] GEN_UUID() performance?
>> Currently, I re-run the test case mentioned in my initial email. Still >> running. I then will also run a test case where inserting GUIDs into a >> non-indexed column will happen. Perhaps adjusting the index tree for >> GUIDs is slow. > > Inserting into a table with a GUID column as primary key takes here ~62 > minutes. Inserting into table with an non-indexed GUID column takes ~10 > seconds. > > Shall I open a ticket with a proper test case? First, try to insert 1mln *random* INTs, i.e. not in sequential order. Else you compare apples (sequential numbers) with oranges (random numbers) :) Regards, Vlad -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Performance after CREATE DATABASE
Right after CREATE DATABASE and until disconnect database works with actual FW=OFF, despite of settings on header page. We specially didn't changed this to make restore fast and simple (no need to disconnect and no chance to lose exclusive attachment). Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Performance after CREATE DATABASE
> On 04/07/11 11:00, Vlad Khorsun wrote: >> Right after CREATE DATABASE and until disconnect database works with >> actual FW=OFF, >> despite of settings on header page. We specially didn't changed this to make >> restore fast and >> simple (no need to disconnect and no chance to lose exclusive attachment). > > Always thought that this works only for gbak attachment. What do you Hmm... i was wrong about gbak. It is explicitly set isc_dpb_force_write to 0 when created database. Later, after restore of data, it restored also FW flag to the state which was read in backup file. > think, is it a bug or a feature in non-gbak case? So, i think, this is the bug. Look at jrd8_create_database, it : a) create database using PIO_create(). PIO_create() creates file in FW=OFF mode. b) format header page using PAG_format_header(). It set FW=ON flag on header page. Note, current file is opened with FW=OFF mode and fil_flags have no FIL_force_write flag set c) then it call PAG_set_force_write(), if isc_dpb_force_write was set in DPB - if isc_dpb_force_write is present in DPB and set to zero then FW=OFF will be set at header page and file will not be re-opened with FW=OFF as it is already in this mode - if isc_dpb_force_write is present in DPB and set to non-zero value then FW=ON will be set at header page and file will be re-opened with FW=ON - if isc_dpb_force_write is NOT present in DPB nothing will be done and file will be left in FW=OFF mode while header page will sill have FW=ON flag. So, next attachment will work in correct FW=ON mode. We can fix engine (make PIO_create to create files in FW=ON mode, or add new parameter to it) but we can't fix old applications which creates database and expect fast work in this first attachment. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Performance after CREATE DATABASE
>> So, i think, this is the bug. > > Agreed. > >> We can fix engine (make PIO_create to create files in FW=ON mode, or add new >> parameter to it) but >> we can't fix old applications which creates database and expect fast work in >> this first attachment. > > As for me, a possibility to have a corrupted database while being in an > explicitly specified but fake forced writes mode is more important than > performance of those applications. If FW mode specified explicitly, than it worked as specified :) The only problem is when FW was not set at CREATE DATABASE time. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Performance after CREATE DATABASE
>>> We can fix engine (make PIO_create to create files in FW=ON mode, or add >>> new parameter to it) > > I think we must take isc_dpb_gbak_attach into an account. When it is > set, let's keep old behavior. gbak is OK. It set isc_dpb_force_write at least since FB 1.5. > And to minimize performance issues in classic, we may use bigger > (SS-like) cache during create time. It could be done at gbak level. Regarsd, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Providers, yvalve, external engines and object lifecycle
> 30.03.2011 21:45, Vlad Khorsun wrote:
>
>>> What would do a detach/commit/etc in an object when refcount> 1?
>>
>> Call corresponding method of real object. And not touch refcount.
>>
>>> What would do a detach/commit/etc in an object when refcount = 1?
>>
>> Same as above.
>
> Sorry for ignorance, but why refcount is not decremented in these cases?
To protect us from the more then one call of detach() for the same
instance.
Example :
IAttachment *att = provider->attach(...);
doSomething(att);
att->detach();
doSomething(IAttachment *att)
{
if (something very bad happens)// this could be really at the few
levels
att->detach();// deeper at whole call stack
}
Well, we can decrement refcount in detach() on success only, but it sounds
as (not necessary) complication of reference counting policy.
> I mean, what is the point in requiring an extra release() call if the
> user has explicitly stated that he no longer needs an object. Note that
> we don't require an explicit addRef() call when the object gets created.
I understand your concerns. BTW, changing way to create objects from
IAttachment *attach(...)
by
void attach(..., **IAttachment)
allows us to use smart pointers and make no additional call of release() :
{
smart_ptr att = NULL;
provider->attach(..., att.refPtr());// refcnt == 1 here !
doSomething(att);
att->detach();// refcnt == 1
}// ~smart_ptr will decrement refcnt
Regards,
Vlad
--
Xperia(TM) PLAY
It's a major breakthrough. An authentic gaming
smartphone on the nation's most reliable network.
And it wants your games.
http://p.sf.net/sfu/verizon-sfdev
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Faster restore
> There is a simple change which makes it much faster, tested with Linux > 2.5 embedded restore: Well done ;) ... > My change creates this BLR: > blr_loop > blr_receive > blr_store > ... > > Calls isc_start_and_send first time then N-1 isc_send. Very good example of very old feature almost not used for a long time. Do you have idea how it could be propagated at DSQL level ? I.e. we could introduce some special syntax to create input stream of messages (like SUSPEND but waiting for next input message). > I suppose this works in all GBAK versions. I see no reason for it to not work ;) > PS: There is other restore improvement capable of take down time from > around 8.5s to 0.5s over TCP, but it still requires some analisys. Very interesting. Let me guess... batching of records ? Regards, Vlad PS Another not used feature is ability of statement to work with more then just one message. We can have few resultsets produced by the one stored procedure, for example. If drivers developers will not kill us ;) -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> And, I've just learned that the new > Delphi 64-bit compiler(hopefully released later this year) will only > use fastcall for DLL interfaces(even Microsoft has that same > restriction for 64-bit). Where can i read about it ? Because i doubt fastcall is *only* supported calling convention in Win64. Moreover, in COM. According to MSDN __fastcall is MS-specific while COM have more wide scope. So, there is no possibility to use anything else then stdcall in IUnknown despite of 32\64 bitness. Delphi used "fastcall" as default calling convention for class methods for a very long time and this is not new and not related to the x64. It is named "register" in Delphi Help. Probably new marketing department decided to rename good old "register" by new cool "fastcall" in still not released Delphi x64 compiler :) Am i wrong ? Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> At April-08-11, 3:56 AM, Vlad Khorsun wrote: > >>> And, I've just learned that the new >>> Delphi 64-bit compiler(hopefully released later this year) will only >>> use fastcall for DLL interfaces(even Microsoft has that same >>> restriction for 64-bit). > >> Where can i read about it ? Because i doubt fastcall is *only* supported >> calling convention in Win64. Moreover, in COM. According to MSDN __fastcall >> is MS-specific while COM have more wide scope. So, there is no possibility to >> use anything else then stdcall in IUnknown despite of 32\64 bitness. > > https://forums.codegear.com/thread.jspa?threadID=52117 > http://msdn.microsoft.com/en-us/library/ms235286.aspx > http://www.embarcadero.com/products/delphi/64-bit (Here it is > mentioned in the video presentation that only one calling convention > will be supported in 64-bit, but unfortunately doesn't specifically > mention which one. Which is where the speculation of fastcall/register > originates, as mentioned in the Delphi forum thread, most likely > because of the MSDN documentation.) > > And, here's another document that I found: > http://www.agner.org/optimize/calling_conventions.pdf > Look on page 16, in the paragraph starting with "Calling convention", > the last line refers to 64-bit mode. Thanks. But, if this is true : "In 64 bit mode, there is only one calling convention for each operating system" then we have no problem with interfaces despite of compiler\language on x64 systems. I have no doubt it is true on Windows and modern Linuxes but have no idea about HPUX, AIX, Solaris, etc... Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> 08.04.2011 15:42, Vlad Khorsun wrote: >> Thanks. But, if this is true : "In 64 bit mode, there is only one >> calling convention for each >> operating system" then we have no problem with interfaces despite of >> compiler\language >> on x64 systems. > > Yep, if forget that current FB API, implemented in current fbclient.dll use > stdcall and > thus Delphi 64 won't be able to work with it. And if you change this calling > convention to > fastcall, all programs that already work with 64-bits client will be broken. In Win64 stdcall is the same as thiscall, cdecl, register, fastcall, anythingelsecall. Do you know that all Win32 API declared with stdcall ? Are you going to say that Delphi Win64 applications wiil not be able to work with Win32 API at all ? :-D There is no problem with calling conventions on x64 systems between modules by different compilers as log as all compilers vendors follow the same rules about fastcall. And it seems they follow. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Reference counters in API
> 08.04.2011 16:25, Vlad Khorsun wrote: >>> 08.04.2011 15:42, Vlad Khorsun wrote: >>>> Thanks. But, if this is true : "In 64 bit mode, there is only one >>>> calling convention for each >>>> operating system" then we have no problem with interfaces despite of >>>> compiler\language >>>> on x64 systems. >>> >>>Yep, if forget that current FB API, implemented in current fbclient.dll >>> use stdcall and >>> thus Delphi 64 won't be able to work with it. And if you change this >>> calling convention to >>> fastcall, all programs that already work with 64-bits client will be broken. >> >> In Win64 stdcall is the same as thiscall, cdecl, register, fastcall, >> anythingelsecall. > > Translate, please. How is it possible? If caller push arguments into stack > from left to > right but callee expect them from right to left, what magic let them to work > properly? This is compiler, who put argumets on stack, isn't it ? :) > Or you mean that our 64bits fbclient.dll already uses fastcall only? And > any other > third-party DLLs as well?.. Yes. At least this is how i read papers above. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
> On 08-04-2011 13:43, alexpeshk...@users.sourceforge.net wrote: >> Revision: 52693 >> http://firebird.svn.sourceforge.net/firebird/?rev=52693&view=rev >> Author: alexpeshkoff >> Date: 2011-04-08 16:43:05 + (Fri, 08 Apr 2011) >> >> Log Message: >> --- >> Use interface IBlrMessage instead passing C++ class pointer to API >> interface's functions >> > I'm *very* against this *undiscussed* change. You also against endless discucssions of every tiny code change, isn't is ? :) > Pointer to plain struct is know to be portable and usable by the clients > we expect to use our API. Plain struct ? It was declared as "class", not as "struct", sorry. If you want understanding from other developers, please, make you ideas clear. Else there will be such "incidents". Something obvious for you is not obvious for me and far not obvious for end users. This is life. Btw, i see no problem curently. Alex made change but it is not means it can't be undone if necessary. Also, please, when you introduce new class or struct or constant - any bit of code in API - please, make comments about its intended usage. Also, please, write explicitly who is responsible for implementation of every interface (firebird itself, plugin creator or application developer) and how to obtain instance of this interface. This is request for all developers, not for Adriano only :) API headers should be as much self documented as possible and new readers should understand it without huge efforts. > This change puts a very number of extra virtual call for each (from each > layer) method called. Considering the methods involved are ones that are > heavily called, this is going to make a vary large overhead. Could you point me - where do you see the large overhead ? Probably i missed some loop with hundreds calls of this methods ? Or it is just a few calls per whole API call ? I see no problem to revert this change in favor of "struct", but i'm against "class" in public API. Public API should use "struct" (as it have no methods) and pure virtual classes (as it have no data and no methods implemented). Else at some day someone will add methods into your POD "class Message" and ruin our public API. And last point. Please, replace name Message with something more exact. The structure you named "Message" is not a message itself - it is sooner message description or message metadata. Names should be clear and not generic when possible. Name "Message" is misleading, sorry. Lets make new entities clear and easy to understand. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
>> Btw, i see no problem curently. Alex made change but it is not means it >> can't be undone if >> necessary. >> > Ah, that's it. Every change I do, specially from you come this discurse > of where the thing was discussed. > > But for others, no. It's just a commit that can be undone. This is just not true. And i'm not going to waste my time to argue about this. > BTW what about if I just came to whole plugin thing and just removed the > reference counters because it was obviously wrong? Is it just a piece of > code that was write to be undone? (at least it appears to be till now, > cause the implementation is currently broken) If you'll remove something then there are two choices always : a) it will be accepted by team b) it will not be accepted by team in the case (b) someone will undo this removal. > And "MessageBuffer" was discussed here in this list and you had all your > time to comment on it. I look at (not mine) code when i have time and need to look at it. If you think that i looking with microscope at every your change just to find a way to undo it - you are terrible wrong. I have more important things to do, believe me ;) >>> This change puts a very number of extra virtual call for each (from each >>> layer) method called. Considering the methods involved are ones that are >>> heavily called, this is going to make a vary large overhead. >> >> Could you point me - where do you see the large overhead ? Probably i >> missed some loop with >> hundreds calls of this methods ? Or it is just a few calls per whole API >> call ? >> > To retrieve records, to insert records. Records * 4 *unnecessary* > virtual calls at least. a) all methods have no parameters so there is no stack allocation b) 4 virtual calls per few thousands another instructions not looks as killer overhead >> I see no problem to revert this change in favor of "struct", but i'm >> against "class" in public API. >> Public API should use "struct" (as it have no methods) and pure virtual >> classes (as it have no data >> and no methods implemented). Else at some day someone will add methods into >> your POD "class >> Message" and ruin our public API. >> > If you understand C++ well you would understand that for C++ usage (this > is a C++ header, after all) inline non-virtual methods and constructors > would not make any damage to our interfaces. When you start to port this C++ header into another language, you will understand what i wrote. Regards, Vlad PS Alex will explain you why "struct" or "class with data members" is bad for *portable* code. -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
> 11.04.2011 14:00, Alex Peshkoff wrote: >> On 04/11/11 15:21, Dimitry Sibiryakov wrote: >>> You are talking about using structures (actually, pointers to them) in >>> new API, right? Passing pointers to structure to interfaces doesn't mean >>> that user has do >>> work with these structures' members directly, AFAIU. >> >> Certainly, one can safely pass any pointer to interface if it is only >> passed, no access to it's internals is attempted. >> The problem is that with mentioned class Message there was need to work >> with members directly. > > Specifically for this task structures can have methods. We talk there about API *declarations*. API *declarations* ideally shouldn't have code else it will be hard to port it into another language. As for struct\interface choice : - struct should be as simple as possible to avoid problems with different alignment rules of different compilers\languages - struct used as array member should be even more simple to avoid problems with different rules of array member alignment - struct is very hard to extend in the feature and it have no versioning (many structures in Win32 API have fixed first field used for versioning, user *must* fill it with sizeof(struct) value, btw) - interface (pure virtual class) are free from issues above but direct access to the interface "members" is impossible and costs virtual function call So, we can use struct, yes. But in very limited places. Hope it is more clear now, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Memory use during restore (Firebird2.5.1.26253-0_x64)
> Does Firebird use the page cache settings in the database (backup file) > during restore? Yes. You can override it using appropriate gbak switch. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
> But reading the discussions during last couple of weeks I see no reason why > this API > declaration would ever be ported into another language. It is your fault then ;) > It just don't have enough advantages over old ISC API. With this new API many simple programs (like yours, for example) will be easy to write and require NO additional access layers. New features (do you need longer SQL identifiers ?) also will be implemented in new API. Call overhead also will be less than with ISC API. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
> 11.04.2011 14:59, Vlad Khorsun wrote: >>> It just don't have enough advantages over old ISC API. >> >> With this new API many simple programs (like yours, for example) will >> be easy to write and require >> NO additional access layers. > > Not quite so. "Easy to write" (for me) means "use well documented API which > has enough > examples of usage". Nothig from this is applied to the new API. Do you have completed API somewhere ? Where do you read that API is already completed and documented ? This mailing list is for developers of Firebird, and we talk about work in progress here. All you can look at - is the source code, so far. It will be documented at appropriate time of course. You can ask even now about its intended usage, if it is not obvious to you, but don't expect wide examples. > And, BTW, what "additional access layers" you have on mind? Y-valve? I speak about "drivers", such as IBPP, IBX\FIB+, ODBC, etc >> New features (do you need longer SQL identifiers ?) also will be implemented >> in new API. > > They can be implemented in an old API as well. So far i know no plans to extend XSQLDA. >> Call overhead also will be less than with ISC API. > > How big performance gain do you expect from it? I don't know. When it will be ready, you are welcome to test it. > AFAIK, CPU time wasted in old API > serialization is microscopic in comparison with network or disk interaction > time. Its not about serialization. Its about converting of data between formats and calls thru many levels of internal functions. Regards, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
> 11.04.2011 15:44, Vlad Khorsun wrote: >>>Not quite so. "Easy to write" (for me) means "use well documented API >>> which has enough >>> examples of usage". Nothig from this is applied to the new API. >> >> Do you have completed API somewhere ? Where do you read that API is >> already completed and >> documented ? This mailing list is for developers of Firebird, and we talk >> about work in progress here. > > Usually development of such things as _public_ API comes in backward > direction: > specification first, implementation later. Probably you missed that we are not creating *new* API calls (or new ideology) there. We are creating object-based interface for the existing API and looking forward how to create it in efficient and backward compatible way. > Breaking of this rule used to lead to a mess. > Look at _already implemented_ API extension - fb_ping() and point me to > it's > description in doc/ folder, please. None. Ok, let's take much older extension > - > fb_interpret(). Also only one mention in RN, no parameters description, no > usage in > examples. Did you noted descriptions of much more important new API's, such as fb_shutdown ? Where are your ticket at the tracker, btw ? > Any reason to think that new API will be more fortunate? I don't have it. So, what do you do here ? Just argue about the things you even not going to use ? Wasting our time with useless questions ? (if one ask a question and not going to read answer - this is useless question, isn't is ?). >>>And, BTW, what "additional access layers" you have on mind? Y-valve? >> >> I speak about "drivers", such as IBPP, IBX\FIB+, ODBC, etc > > You know that I don't use them. Yes, i know. And what ? >>>> Call overhead also will be less than with ISC API. >>> >>>How big performance gain do you expect from it? >> >> I don't know. When it will be ready, you are welcome to test it. > > So, "call overhead also _will_ be less" is just a hope and new API may be > slower than > old one, right? When i said "Call overhead also will be less" i meant exactly "Call overhead also will be less". Is it clear ? >>> AFAIK, CPU time wasted in old API >>> serialization is microscopic in comparison with network or disk interaction >>> time. >> >> Its not about serialization. Its about converting of data between >> formats and calls thru many >> levels of internal functions. > > AFAIU, new API is based on the Message API (at least in this thread is > discussed > exactly wrapper around BlrMessage). There is only one transformation between > XSQLDA API > and Message API, and it includes no data conversion, just copying it. Exactly > as the new > API does. Where profit will come from? You wiil be able to use message-based DSQL API in a more easy way then old XSQLDA-based DSQL API. Oh, no, sorry. You are not going to use new API :-D Regards, Vlad PS When i started to answer your questions i thought that you really not understand something obvious for us. I was wrong. Now i see you just mocks of us. -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] SF.net SVN: firebird:[52693] firebird/trunk/src
>>> They can be implemented in an old API as well. >> >> So far i know no plans to extend XSQLDA. > > I mentioned that a few times here and I still support extending XSQLDA > (to the extent it's still maintainable). Then i leave to you a "pleasure" to answer on DS questions. It is enough for me and i'm shut up here. Good luck, Vlad -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request: database registration via aliases.conf, databases system table support
> In this way, you lose dynamic registration, and have the administrator > manage the list of databases shown in RDB$DATABASES. In what database ? Regards, Vlad -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request: database registration via aliases.conf, databases system table support
> How is this feature related to upcoming support for schemas? This feature have nothing common with schema support. Regards, Vlad PS Access to the aliases list via new service seems enough as for me. -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Aliases
> Just curious, has there been any discussion about expanding aliases to allow > defining just a path to a directory > where multiple database reside? > > The reason I'm asking is that in some situations, my apps are designed to > allow the users to archive fiscal data. > In general, they have a main database for active information and an archive > database for current archive data. > Each fiscal year, the information in the current archive database is copied > to a new fiscal archive database and > then cleared for the new fiscal year. That means that there could be years of > data in individual databases but they > all reside in the same directory. > > Currently, to use aliases, I would have to add an alias each time a new DB is > created. You can create aliases for the next 100 years > If there were a way to specify > an alias for a particular directory, I could use that to build a complete > database name something like: > > servername:aliasname:databasename > > That would allow more dynamic building of connection strings. At the moment, > I usually store the database path in > each users settings file and just tack on the database name as I need it. > However, if the location changes, each users > settings would have to be changed. It would be nice if I could declare an > alias in the alias.conf file to use as a base > directory. What are you going to do, when databases will overflow this single directory ? Buy new HDD ? Create new directory and teach application to look for databases at two "superaliases" ? Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] is there query cache in firebird 1.5
> i want to ask is there query cache in firebird 1.5 ? Use stored procedures. Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird internals - NULL mask
> If I understand correctly, there is E9 (23) bytes with zeros in NULL > mask? NULL nask should occupy 4 or 8 or 12 etc bytes. What am I missing? You missing that record are RLE compressed at whole, including NULL mask. Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Aliases
> From: "Vlad Khorsun" >> >>What are you going to do, when databases will overflow this single >> directory ? Buy new HDD ? Create new >> directory and teach application to look for databases at two >> "superaliases" ? >> > > As you pointed out, I can create a database for each year for the next > hundred years. No. I talk about *aliases* not databases. > Doubtful, but theoretically possible. Even that won't fill up > the directory because the data doesn't grow nearly as fast as hard drive > space does. > > However, it would be useful, IMO, to have an alias for directories so that > conceptually, you could separate entire functionalities. For instance, the > main databases reside in an alias directory called MainDataDir. Backups > could be performed to the BackupDataDir. aliases for backup files ? Oh... > User importing/exporting of data > could be in a shared directory known as UserDataDir. Seems you want that Firebird implement solution for your applied issue. > Changing the location > of these would become much easier just like it is now with database aliases. Let me guess how you going to use this super-aliases : a) create entry at aliases.conf for common folder for historical databases : databases = c:\archive\databases b) Application have built-in (or configurable) knowledge about this super-alias c) Application want to work with database of year NNN. It should open it as databases:yearNNN d) in Year 2015 space on disk C: is over and you want to add second super-alias, say databases2 = d:\archive\databases e) Aplication should be changed to take it into account !!! If you pre-create *aliases* for databases : year2010 = c:\archive\databases\year2010.fdb year2011 = c:\archive\databases\year2011.fdb year2012 = c:\archive\databases\year2012.fdb year2013 = c:\archive\databases\year2013.fdb year2014 = c:\archive\databases\year2014.fdb year2015 = c:\archive\databases\year2015.fdb year2016 = c:\archive\databases\year2016.fdb and at some point of time you are out of disk space on drive C, all you need to do is to change few aliases and don't touch application : year2015 = D:\archive\databases\year2015.fdb year2016 = D:\archive\databases\year2016.fdb Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU. VS2010 project files
> I spend some time for revision and adaption of your copy of ICU library for > VS2010 Thanks > 1. Reduce number of compiler warnings > I defined the _CRT_SECURE_NO_WARNINGS=1 in firebird2intl.props Good change, imho > 2. Creation of PDB files for all binaries (debug/release) Not necessary. I doubt we are going to distribute pdb files for ICU package. There was ZERO issues with ICU which could require to install .pdb at use machine, AFAIR. > 3. Unified names of release/debug binaries Good change too. I have nothing againt applaying this patch to the our copy of ICU but first i want to raise more important question re. ICU : are we going to update our copy of ICU package in FB3 ? We used very outdated copy and it have some still not fixed (mostly minor) issues. On Linuxes there is possibility to use system ICU libraries but on Windows there is no ICU distributed with OS itself. So, are we going to use more recent ICU libraries in FB3 ? Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] The boot build again
> Hello, I will continue making noise until I'm explained how the new build > system is supposed to work. I'm tired of checking for possible differences > between the VC10 and VC8 projects and my VC9 project. I found no differences between VC8 and VC9 (except of usage of manifest files and file versions) - .vcproj files are easy to compare by WinMerge. > Obviously the > developers that have attempted the boot build had something left in a > temporary directory. Nope. Before commit new build system i run make_boot and make_all for both Win32 and Win64 arch and manually deleted gen and temp dirs. Right now i checked it again - no problem. I have VC8 and VC10 and have no VC9, btw. > I started clean, deleting the temporary and compilation > dirs, etc. The boot build should be self sustained. Sure > I mean, it should be > able to complete itself without my manual intervention to copy some files in > addition to run make_boot. What files are you need to copy ? > I still get this with the latest sources: > > 1>-- Build started: Project: isql, Configuration: Debug Win32 -- Hmm.. never run make_boot DEBUG :) Tried it right now (VC10, Win32) - all is OK. > 1>Compiling... > > 1>isql - 0 error(s), 1 warning(s) > == Build: 1 succeeded, 0 failed, 3 up-to-date, 0 skipped == > > Statement failed, SQLSTATE = 39000 > Entrypoint of plugin > F:\fb3dev\fbbuild\firebird30\temp\Win32\debug\firebird\plugins/fbtrace does > not > exist > Statement failed, SQLSTATE = 39000 Do you have file \builds\win32\defs\plugin.def ? ... > I hope the people that develop FB from Windows will share their trick on how > to bypass this problem. Believe me or not, but there is not tricks known to me. It just works :) Regarsd, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Index is corrupt (missing entries) in table
> However, the important thing to note is, that backing up and restoring > does not rectify the problem! Backing up and restoring finish their > respective jobs without problems, but the validation problem is > *immediately* there (i.e. without any usage of the DB between restoring > and validating). Interesting. Could you provide DDL of table and index ? Also tell us what is charset of database. > The index build during restore creates this problem with these two large > DBs absolutely always, i.e. corruption is in this case apparently always > repeatable. (We experienced such a thing never before, as either backup or > restore fail, or they do not fail but there is no more validation problem > immediately afterwards. Well, not so in this story.) Is it reproducible ? I.e. if you restore same backup few times it have missed entries every time ? What if you restore same backup at the another machine ? >> I think Vlad is going to need a copy of your database you get to the >> bottom of the problem, so if you're customer is not going to allow that, >> it's going to be a problem :-( > > It is unfortunately out of the question that we make these DBs available. > I know that this would be great help, but am unable to change it. On the > other hand, we would be more than willing to use any beta, gamma :) or > possibly special test builds for testing this, in order to help Vlad. It is too early to discuss special builds or to ask you to provide copy of database. Regards, Vlad -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] discussion about real fix for CORE-2348
> I see that there this problem was analized "completly" More-or-less > Is this really plan to fix this in FB3.0 None, AFAIK > Vlad Korshun say that only transaction id will be unsigned int in FB3.0 - is > this true > or is in plan to fix this with two record header versions or other solution? Transaction ID's (and page numbers) will be unsigned 4-byte int in ODS12. Regards, Vlad -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Index is corrupt (missing entries) in table
Sorry for long silense, i'm a buzy with other things... > Thank you for your time Vlad. You wrote: > >> Interesting. Could you provide DDL of table and index ? > > CREATE TABLE MATBEW > ( > MSEG_MATNR Varchar(18) NOT NULL, > MSEG_WERKS Varchar(4) NOT NULL, > MSEG_LGORT Varchar(4) NOT NULL, > MSEG_MBLNR Varchar(10) NOT NULL, > MSEG_MJAHR Varchar(4) NOT NULL, > MSEG_ZEILE Varchar(4) NOT NULL, > MSEG_CHARG Varchar(10), > MSEG_BWART Varchar(3), > MSEG_SHKZG Varchar(1), > MSEG_SOBKZ Varchar(1), > MSEG_XAUTO Varchar(1), > MSEG_KZZUG Varchar(1), > MSEG_KZVBR Varchar(1), > MSEG_MEINS Varchar(3), > MSEG_MENGE Double precision, > MSEG_ERFME Varchar(3), > MSEG_ERFMG Double precision, > MSEG_BSTME Varchar(3), > MSEG_BSTMG Double precision, > MKPF_BUDAT Date, > MKPF_CPUDT Date, > MSEG_LIFNR Varchar(10), > MSEG_EBELN Varchar(10), > MSEG_EBELP Varchar(5), > MSEG_WEMPF Varchar(12), > MSEG_AUFNR Varchar(12), > MSEG_KUNNR Varchar(10), > MSEG_KDAUF Varchar(10), > MSEG_KDPOS Varchar(6), > MSEG_KDEIN Varchar(4), > MSEG_WAERS Varchar(5), > MSEG_DMBTR Double precision, > MKPF_BLDAT Date, > CONSTRAINT MATBEW_PK PRIMARY KEY > (MSEG_MATNR,MSEG_WERKS,MSEG_LGORT,MSEG_MBLNR,MSEG_MJAHR,MSEG_ZEILE) > ); > > The additional index (from the other DB): > > CREATE INDEX IDX_MATBEW1 ON MATBEW (MSEG_WERKS); > >> Also tell us what is charset of database. > > It is not defined. So, all key fileds is VARCHAR with charset NONE, correct ? Is it possible to have zero bytes in data values of any of the key fields ? What about leading or trailing zero's ? > I know that this is not recommended. I was already > thinking of defining it to be UTF-8 and than making backup and restore. > > Also, I was today thinking of defining 6 indices, each based on only one > field of the current primary key and see what happens. > > Are these tests worth trying? Sure. >> Is it reproducible ? I.e. if you restore same backup few times it have >> missed entries every time ? > > I believe this was done, but shall try to do it once more until weekend > and report here. (My colleague and I have done numerous independent tests > on various machines last week.) Any news ? >> What if you restore same backup at the another machine ? > > This was definitely done. No improvement. > > Please tell what would be your preferred FB version for these backup and > restore tests. We have different versions on different machines, so I > could see what can be done. 2.1.4 or 2.5.0, i.e. latest official release of 2.1 or 2.5 Regards, Vlad -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU. VS2010 project files
>> we must provide binaries inside our Windows package :(
>>
>
> Yes, that is right.
>
> I don't object to agreeing on a specific version of ICU for each release. We
> would just change the build script as required. (Presumably we will need .lib
> files.) Our main problem is likely to be finding icu binaries built with
> MSVC10.
ICU 4.6 (more exactly icu4c 4.6 and latest 4.6.1) are built using MSVC10, so
this is not an issue.
BTW, we still have no MSVC10 at our build farm, but this is another
question...
> Not much would change as far as packaging is concerned. We would just include
> the icu dlls as now, except their location at package time will / may be
> different.
>
> Of courses, we may get lucky and ibm will have upgraded their ridiculous icu
> build system and we could just incorporate it again into the windows build.
> However, if it hasn't been changed it would require a ton of work for each
> icu
> version, which is one of the reasons why we haven't upgraded icu.
One note. Binary packages at IBM site
http://download.icu-project.org/files/icu4c/4.6.1/icu4c-4_6_1-Win32-msvc10.zip
http://download.icu-project.org/files/icu4c/4.6.1/icu4c-4_6_1-Win64-msvc10.zip
contains relatively "big" ICU libraries. Size of 3 dll's needed for Firebird is
more than
16MB, the biggest is icudt46.dll (14.4MB) which contains a lot of unused by FB
data.
Should i said that whole FB folder have similar size ?
Here http://apps.icu-project.org/datacustom/ we can see what could be
removed
without any harm.
We should decide (at least for Windows)
a) will we provide ICU libraries
- within our packages (as with FB2.x), or
- ask users to download ICU from IBM site, or
- provide separate package with just 3 DLL's we need
b) will we provide default ("fat") ICU libraries or our customized ("light")
version ?
Regards,
Vlad
--
WhatsUp Gold - Download Free Network Management Software
The most intuitive, comprehensive, and cost-effective network
management toolset available today. Delivers lowest initial
acquisition cost and overall TCO of any competing solution.
http://p.sf.net/sfu/whatsupgold-sd
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Shared page cache
Hi all. After more than a year of development, testing, switching on another tasks and returning back i'm ready to commit shared page cache implementation into trunk. I consider it stable enough to be committed into SVN. It runs some stability tests more than 10 hours for longest run and many shorter runs at different configurations. Here i want to do overview of changes, what was tested, what was not, and what is work in progress. Page cache was re-implemented looking at both Vulcan code (thanks to Jim) and our code base. It is not a blind copy of Vulcan code, every decision was thinking on and adjusted when needed with my understanding and expirience with existing code. The cache is syncronized at different levels. There is a few BufferControl (BCB) level sync's used to syncronize access to the common structures such as LRU queue, Precedence graph, Dirty page list and so one. These common sync's should be locked at as short as possible period of time as this is kind of global locks. Also there is a two syncs for every BufferDesc (BDB) - one for guard contents of page buffer itself and second one is to prevent partial writes of page wich edition is not completed and to prevent simultaneous write of the same page by different threads. I don't explain cache algoritms here, if someone have a question i'll be happy to answer it. And i not going to said that page cache implementation is frozen - if necessary we will change it of course (there is still space for impovements and bugs fixing). You know i already committed SyncObject class (and related classes) ported from Vulcan. I removed some unused code and restored fairness of locking : initially it was fair in Vulcan too (i.e. all lock requests was put in waiting queue and granted in first-in-first-out, or fair, order), but later Jim found some (unknown to me) issue with this and made preference for SHARED locks. I found no performance issues with fair locking so i decided to revert that and restore original behavior. Of course it could be changed when necessary. Shared page cache is not an isolated change. It affects many parts of engine and our syncronization model changed significantly again. There was agreement that we will not implement shared metadata cache in v3 as this is risky and we could not deliver v3 in reasonable time frame. In shared cache mode we have single Database instance and many Attachment instances linked to it (this is not new). All metadata objects moved into Attachment. Metadata syncronization is guarded by attachment's mutex now. Database::SyncGuard and company are replaced by corresponding Attachment::XXX classes. To make AST's work we need to release attachment mutex sometimes. This is very important change after v2.5 : in v2.5 attachment mutex is locked during whole duration of API call and no other API call (except of asyncronous fb_cancel_operation) could work with "busy" attacment. In v3 this rule is not worked anymore. So, now we could run more that one API call on the same attachment (of course not really simultaneously). I'm not sure it is safe but not disabled it so far. To make asyncronous detach safe i introduced att_use_count counter which is incremented each time when API call is entered and decremented on exit. detach now marks attachment as shutdown and waits for att_use_count == 0 before processing. Parallel access to the attachment could be easy disabled making every API call wait for att_use_count == 0 on enter or even introducing one more mutex to avoid spin wait. Also it seems this counter make obsolete att_in_use member as detach call should wait for att_use_count == 0 and drop call should return "object is in use" if att_use_count != 0. All AST's related to attachment-level objects should take attachment mutex before access attachment internals. This is implemented but not tested ! Transaction inventory pages cache (TIP cache) was reworked and is shared by all attachments. To avoid contention on common dbb_pool its usage was replaced by att_pool when possible. To make this task slightly easy there was introduced jrd_rel::rel_pool which is points currenlty to the attachment's pool. All relation's "sub-objects" (such as formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). When we'll return metadata objects back to the Database it will be easy to redirect rel_pool to dbb_pool at one place in code instead of makeing tens of small changes again. About stability testing of different parts of the engine : - page cache - tested and worked - nbackup - tested and worked - monitoring - tested (less) and seems worked - server stop\database shutdown - somewhat tested, no crash observed, client reaction is not perfect (mostly network write erros) - shadow - not tested - metadata changes in concurrent environment - not tested - garbage colle
Re: [Firebird-devel] Index is corrupt (missing entries) in table
> We have in the meantime made the following test: > > - copied DB to another machine (always under VB 2.1.4) and > on that machine: > > - validate => Error > > - dropped a number of fields, but did not change primary > index definition in any way. The original table was > described in one of my previous messages. The resulting > table is: > > CREATE TABLE MATBEW( > MSEG_MATNR Varchar(18) NOT NULL, > MSEG_WERKS Varchar(4) NOT NULL, > MSEG_LGORT Varchar(4) NOT NULL, > MSEG_MBLNR Varchar(10) NOT NULL, > MSEG_MJAHR Varchar(4) NOT NULL, > MSEG_ZEILE Varchar(4) NOT NULL, > MSEG_BWART Varchar(3), > MSEG_SHKZG Varchar(1), > MSEG_SOBKZ Varchar(1), > MSEG_MENGE Double precision, > MKPF_BUDAT Date, > MSEG_LIFNR Varchar(10), > MSEG_KUNNR Varchar(10), > CONSTRAINT MATBEW_PK PRIMARY KEY > (MSEG_MATNR,MSEG_WERKS,MSEG_LGORT,MSEG_MBLNR,MSEG_MJAHR,MSEG_ZEILE)); > > - backup > - restore > - validate => OK!!! > > It seems as if the fields that do not have anything to do > with index are influencing the index build up. Hard to believe ;) > We would like to make a definitive check if in any of our > string fields zero byte values are occurring (h00), since > Vlad has asked about this and I couldn't completely dismiss > that possibility. > > Please help: How can I check if a string field value > includes at least one zero byte - I would like to write a > SELECT statement using these checks, but have no clue how. Try this: field CONTAINING ASCII_CHAR(0) > (We have still not made tests with defining UTF-8 as the DB > character set.) The most quick way to research the issue is to provide me with database or backup. You can drop all not related objects and obfuscate all important data. Regards, Vlad -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Shared page cache
>> All metadata objects moved into Attachment. Metadata syncronization is >> guarded >> by attachment's mutex now. Database::SyncGuard and company are replaced by >> corresponding Attachment::XXX classes. >> >> To make AST's work we need to release attachment mutex sometimes. This is >> very >> important change after v2.5 : in v2.5 attachment mutex is locked during >> whole duration >> of API call and no other API call (except of asynchronous >> fb_cancel_operation) could work >> with "busy" attachment. In v3 this rule is not worked anymore. So, now we >> could run more >> that one API call on the same attachment (of course not really >> simultaneously). I'm not >> sure it is safe but not disabled it so far. > > Are there other reasons (besides AST delivery and possible > per-attachment parallelism) to have checkouts? Yes. Database-level objects with internal syncronization also need to release attachment mutex while waiting for own syncronization object. Look at GlobalRWLock, for example. >> To make asynchronous detach safe i introduced att_use_count counter which is >> incremented each time when API call is entered and decremented on exit. >> detach >> now marks attachment as shutdown and waits for att_use_count == 0 before >> processing. > > How is it done? Sleeping/timeouts inside the spin loop? Currently - yes. See JAttachment::freeEngineData (former jrd8_detach) >> Also it seems this counter make obsolete att_in_use member as detach call >> should wait >> for att_use_count == 0 and drop call should return "object is in use" if >> att_use_count != 0. > > Agreed. > >> All AST's related to attachment-level objects should take attachment mutex >> before >> access attachment internals. This is implemented but not tested ! > > Are there any database-level ASTs known to implicitly access the > attachments? When should they lock the appropriate mutex? Database-level ASTs not need to access attachment internals usually. Therefore NULL attachment can be passed into both Attachment::SyncGuard and Attachment::Checkout classes. Not perfect i know. Better ideas appreciated. >> Transaction inventory pages cache (TIP cache) was reworked and is shared by >> all >> attachments. > > What other layers (besides TPC, CCH and supposedly PIO) are kept in the > shared Database class? SharedCounter, PageSpace manager, backup manager, shadow, monitoring, blob filters, generator pages, loaded modules, External Engines manager, probably something else... Some of them requires additional protection, so this is work in progress. >> To avoid contention on common dbb_pool its usage was replaced by att_pool >> when >> possible. To make this task slightly easy there was introduced >> jrd_rel::rel_pool which >> is points currently to the attachment's pool. All relation's "sub-objects" >> (such as >> formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). >> When we'll >> return metadata objects back to the Database it will be easy to redirect >> rel_pool to >> dbb_pool at one place in code instead of makeing tens of small changes again. > > Good idea. What about going even further and renaming it to meta_pool > and then switching procedures/functions/triggers to this pool as well? Should think about it ;) >> In configuration file there are two new settings : >> a) SharedCache - boolean value which rules the page cache mode >> b) SharedDatabase - boolean value which which rules the database file open >> mode. > > Do we really need the SharedCache option in production? It was implicitly introduced by Alex and i just add it to the config file. I see no problem if we remove it - less options is less headache for us and for users :) > I thought it would be the only mode available. See below. > >> - SharedCache = true, SharedDatabase = false (default mode) >> this is traditional SuperServer mode when all attachments share page >> cache and >> database file is opens in exclusive mode (only one server process could >> work with >> database) > > OK. > >> - SharedCache = false, SharedDatabase = true >> this is Classic mode when each attachment have its own page cache and >> many >> server processes could work with the same database. > > The Classic mode could be handled by the listener itself. I mean both CS and SC here and explained it below :) > As long as it > keeps launching one process per connection, there's no difference > between SharedCache being true or false, as there will always be only > one Database/Attachment pair per process. Sure >> To run SuperClassic you should use switch -m in command line of firebird.exe >> (on Windows) or run fb_smp_server (on Posix, here i'm not sure and Alex will >> correct me). Else ClassicServer will run. > > This is the only case when we seem needing SharedCache = false. But what > are the benefits of SuperClassic in v3.0? For end users - i don't know
Re: [Firebird-devel] Shared page cache
>>> As long as it >>> keeps launching one process per connection, there's no difference >>> between SharedCache being true or false, as there will always be only >>> one Database/Attachment pair per process. >> >> Sure > > Isn't the cache shared between processes? No. There was never such promises. Regards, Vlad PS Seems i kill your day ;) -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Shared page cache
> In this case may I suggest to use separate memory pool for the cache's > elements?.. It is already separate > Someday someone could implement shared memory pool... It is already shared ;) Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Shared page cache
>>> Are there any database-level ASTs known to implicitly access the >>> attachments? When should they lock the appropriate mutex? >> >> Database-level ASTs not need to access attachment internals usually. > > "Usually" differs from "always" :-) I seem to remember cases when the > AST saves lck_attachment into tdbb_attachment and then it could be > accessed by lower levels even for database-level locks (lck.cpp with > att_long_locks comes to mind first). IIRC, i removed tdbb->setAttachment() from database-level ASTs, at least it was my intention. Anyway, AST's routines requires careful re-thinking. So far i can be more or less sure about BDB and NBackup ones only. >>> Do we really need the SharedCache option in production? >> >> It was implicitly introduced by Alex and i just add it to the config file. I >> see no problem >> if we remove it - less options is less headache for us and for users :) > > This was exactly my point :-) Good :) >>> But what are the benefits of SuperClassic in v3.0? >> >> For end users - i don't know. For us to easily debug Classic mode - >> definitely. > > Debugging is surely must be supported, but I'm believe we could make it > possible without exposing the SharedCache option to the public. No problem for me. >> I always said that Affinity should work even for Classic. > > And on POSIX? ;-) Yes, of course. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Shared page cache
>> -Original Message- >> From: Vlad Khorsun [mailto:hv...@users.sourceforge.net] >> Sent: Lunes, 09 de Mayo de 2011 6:07 > >> To avoid contention on common dbb_pool its usage was >> replaced by att_pool when >> possible. To make this task slightly easy there was >> introduced jrd_rel::rel_pool which >> is points currenlty to the attachment's pool. All relation's >> "sub-objects" (such as >> formats, fields, etc) is allocated using rel_pool (it was >> dbb_pool before). When we'll >> return metadata objects back to the Database it will be easy >> to redirect rel_pool to >> dbb_pool at one place in code instead of makeing tens of >> small changes again. > > And the statistics about memory usage in the engine, do they return the same > values as before? It shouldn't be changed because of this. To be true - i don't know, and even not sure we have correct memory statistics currently. It should be checked separately. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> Could you explain, please, semantics of addRef/release on public > provider interfaces? > > The issue was discussed on this list with two possibilities: > - when refCount = 1, release does things like detach, rollback, free, > etc. It's what was implemented so far. > - sometime mentioned also that detach/rollback/etc would not release, > and user would need to explicit call it. > > But now Vlad says we can't call release when refCount = 1, and it's just > there to decrement addRefs. > > This broke code I wrote based on discussed ideas, and I don't feel it's > correct. > > (Also note Alex code in jrd.cpp and is clear he wrote code not using > this new assumption.) I assume there must be a kind of symmetry : attach\detach, startTx\commit, startTx\rollback, allocateStmt\freeStmt, addRef\release. From semantic POV attach\detach is like constructor\destructor pair while addRef\release should be used for additional references only (i.e. not instead of destructor). So, if code calls attach\release - this is bug, as for me, and i see no reasons to write code in this way intentionally. I consider at as very bad practice as it could lead to the inexpected side effects (such as implicit rollback). Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 05/16/11 20:33, Vlad Khorsun wrote: >>> Could you explain, please, semantics of addRef/release on public >>> provider interfaces? >>> >>> The issue was discussed on this list with two possibilities: >>> - when refCount = 1, release does things like detach, rollback, free, >>> etc. It's what was implemented so far. >>> - sometime mentioned also that detach/rollback/etc would not release, >>> and user would need to explicit call it. >>> >>> But now Vlad says we can't call release when refCount = 1, and it's just >>> there to decrement addRefs. >>> >>> This broke code I wrote based on discussed ideas, and I don't feel it's >>> correct. >>> >>> (Also note Alex code in jrd.cpp and is clear he wrote code not using >>> this new assumption.) >> I assume there must be a kind of symmetry : attach\detach, >> startTx\commit, >> startTx\rollback, allocateStmt\freeStmt, addRef\release. From semantic POV >> attach\detach is like constructor\destructor pair while addRef\release >> should be >> used for additional references only (i.e. not instead of destructor). >> >> So, if code calls attach\release - this is bug, as for me, and i see no >> reasons >> to write code in this way intentionally. I consider at as very bad practice >> as it >> could lead to the inexpected side effects (such as implicit rollback). > > Unfortunately release() is also a kind of dtor, and we can't report > about an error that happened inside release(). Therefore the question is > - what to do if one called release() for an object with refCount == 1? I > assume that for us this is a kind of broken network connection, i.e. > attachment is closed, transactions rolled back, etc. Bad style - but we > must do something here... Sure, we must do something and current code seems more-or-less OK in this regards. > Certainly, nothing prevents us from adding error reporting to our > release. Should we do it? We can introduce some special mode to report such errors using exceptions, or write message into log, or even provide callback for debugging purposes. But Adriano's question was about *semantics*. So, do you agree with me that explicit detach() must be called by good code and not just release() ? And that call of release() when refCount == 1 is a bug in caller's code ? Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 16/05/2011 13:33, Vlad Khorsun wrote: >> So, if code calls attach\release - this is bug, as for me, and i see >> no reasons >> to write code in this way intentionally. I consider at as very bad practice >> as it >> could lead to the inexpected side effects (such as implicit rollback). >> > Now every provider class has a way to "release" with another methods, Again, "release" it not how object should be finally destroyed. It is for additional references *only*. > but take into account that release is base method. Not sure i understand what do you want to say here. addRef\release used for reference counting and not for creating\destructing objects. > But what about different release-able classes? Are you going to add > additional methods to release them, cause release is only for addRef? Again not sure i understand you correctly. Basically, if something have *explicit* constructor (attach) it must have explicit destructor (detach). If something is not constructed *explicitly* (fb_get_master_interface) it should be just release()'ed without additional methods. Yes, in a very simple cases we can omit creation of explicit "destructor" but i prefer to have it even in simplest case. Your never know if tomorrow simplest thing will not became more complex ;) Show me example of what do you mean by "different release-able classes" and i'll try to be more precisely. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 3, Engine12 exists, but can not be loaded
> I downloaded the Firebird-3.0.0.29349 snapshot of firebird, for a test. > After unzipping the archive, I ran Firebird from the command prompt by: > > firebird.exe -a > > No errors were written to firebird.log at that moment, but as soon as I want > to > connect to a database using ISQL I have the following error in firebird.log: > > C:\Program Files\Firebird-3.0.0.29349-0_Win32\plugins/Engine12 exists, but > can not be loaded. Looks like CRT8 issue. Window Event log could have some info. > Do I have to alter a config file, or can't I test the snapshots yet? > OS: Windows XP SP 3, 32 bit. Try current snapshot (build 3.0.0.29364). It was recently changed to use CRT10 which is not used fusion technology (finally) Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 16/05/2011 14:12, Vlad Khorsun wrote: >>> On 16/05/2011 13:33, Vlad Khorsun wrote: >>>> So, if code calls attach\release - this is bug, as for me, and i see >>>> no reasons >>>> to write code in this way intentionally. I consider at as very bad >>>> practice as it >>>> could lead to the inexpected side effects (such as implicit rollback). >>>> >>> Now every provider class has a way to "release" with another methods, >> Again, "release" it not how object should be finally destroyed. It is >> for >> additional references *only*. >> > This is something new you invented. You asked about semantics and i explained how i understand it. If you have something technical against it - show it here and we'll decide what to do. >>> but take into account that release is base method. >> Not sure i understand what do you want to say here. addRef\release used >> for reference counting and not for creating\destructing objects. >> > See above. I see nothing, are you ? >>> But what about different release-able classes? Are you going to add >>> additional methods to release them, cause release is only for addRef? >> Again not sure i understand you correctly. Basically, if something have >> *explicit* constructor (attach) it must have explicit destructor (detach). If >> something is not constructed *explicitly* (fb_get_master_interface) it should >> be just release()'ed without additional methods. >> > Everything is constructed explicitly. I'm not agree here. See below. >> Yes, in a very simple cases we can omit creation of explicit >> "destructor" but >> i prefer to have it even in simplest case. Your never know if tomorrow >> simplest >> thing will not became more complex ;) >> >> Show me example of what do you mean by "different release-able classes" >> and i'll try to be more precisely. >> > IFirebirdConf and IPluginConf, for example. They're released with release. IPluginConf is not constructed explicitly from the user POV - it is created by the Firebird and passed into user code, see : IPluginFactory::createPlugin(IPluginConfig* factoryParameter), where user is implemented IPluginFactory interface. IFirebirdConf also not constructed by user request as it is obtained via method of IPluginConfig instance passed above : IFirebirdConf* IPluginConfig::getFirebirdConf() If you ask me why attach() and getFirebirdConf() is different - the answer is : by semantics. And by presence explicit "destructor" function. > The situation with new classes trying to be smart using reference > counting is still almost unexplained from the POV of why that was done. > > Sure you may create artificial and specific examples of where they make > things to not crash. > > But nothing except an atomic counter is currently being cared regard > MT-safeness. Y-valve has internal state not synchronized, for example. > Y-valve delegates call to another providers, and they do nothing re. > died objects to return information. This is still work in progress, i think. Probably Alex have better answer. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 05/16/11 21:00, Vlad Khorsun wrote: >> >> We can introduce some special mode to report such errors using >> exceptions, >> or write message into log, or even provide callback for debugging purposes. >> >> But Adriano's question was about *semantics*. So, do you agree with me >> that >> explicit detach() must be called by good code and not just release() ? And >> that >> call of release() when refCount == 1 is a bug in caller's code ? > > My initial thinking was - for objects, that have explicit dtor functions > (detach/commit/etc.), explicit dtor() must be called by good code and > not just release(). Here we are agree, ok. > But calling release() with refCount == 1 is not always bug. It's > (logically) like handling errors in dtors - sometimes it's better to > ignore error in dtor cause it can be already called due to another > error. And if some thread needs to tell to the object - I do not need > you any more, I anyway can't handle an error if it happens, why not let > it call release()? As I've already said, logically this does not differ > from lost network connection. It is almost true. Almost, because in the case of unrecoverable error in detach() we must offer *correct* way to release attachment and all underlying objects. For remote interface we have fb_cancel_operation(...fb_cancel_abort...) which forcebly closed connection and (should) allow to free all other handles in usual way. I can't imagine unrecoverable error in, say, statement->drop(), which is not related to the same unrecoverable error in detach(), so forced close of attachment is enough for us. If we have explicit destructor - we *must* use it. Else remove it and use release() always. Is it wrong ? Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 05/16/11 23:47, Vlad Khorsun wrote: >>> On 16/05/2011 14:12, Vlad Khorsun wrote: >>>>> On 16/05/2011 13:33, Vlad Khorsun wrote: >>>>>> So, if code calls attach\release - this is bug, as for me, and i see >>>>>> no reasons >>>>>> to write code in this way intentionally. I consider at as very bad >>>>>> practice as it >>>>>> could lead to the inexpected side effects (such as implicit rollback). >>>>>> >>>>> Now every provider class has a way to "release" with another methods, >>>> Again, "release" it not how object should be finally destroyed. It is >>>> for >>>> additional references *only*. >>>> >>> This is something new you invented. >> You asked about semantics and i explained how i understand it. If you >> have >> something technical against it - show it here and we'll decide what to do. >> > > Vlad, let's start with the following - why is calling release() to > remove last reference is bad? Because it opens a way to make bugs - rolled back work, for example. > Currently I see just one reason - error can't be reported. What if we > find a way to report an error in such case? What error do you mean ? At previous message you said that release() could be called if detach() failed. Do you see a problem if release() also will fail ? Or you talk about something else ? >>> IFirebirdConf and IPluginConf, for example. They're released with release. >> IPluginConf is not constructed explicitly from the user POV - it is >> created >> by the Firebird and passed into user code, see : >> >> IPluginFactory::createPlugin(IPluginConfig* factoryParameter), where >> >> user is implemented IPluginFactory interface. >> >> IFirebirdConf also not constructed by user request as it is obtained via >> method of IPluginConfig instance passed above : >> >> IFirebirdConf* IPluginConfig::getFirebirdConf() >> >> If you ask me why attach() and getFirebirdConf() is different - the >> answer >> is : by semantics. And by presence explicit "destructor" function. > > Vlad, I'm afraid that this difference is not enough precise... attach() constructed object for *my private use* and i'm responcible for *destroying* it after use. getFirebirdConf(), in general, could return some *common* object instance and i can't destroy it - i just must *release* it after usage. Difference between "destoy" and "release" is semantics. I don't know how to explain it else. >>> The situation with new classes trying to be smart using reference >>> counting is still almost unexplained from the POV of why that was done. >>> >>> Sure you may create artificial and specific examples of where they make >>> things to not crash. >>> >>> But nothing except an atomic counter is currently being cared regard >>> MT-safeness. Y-valve has internal state not synchronized, for example. >>> Y-valve delegates call to another providers, and they do nothing re. >>> died objects to return information. >> This is still work in progress, i think. Probably Alex have better >> answer. > > In FB2.5 yValve did not need any more MT-safeness except provided by > atomic counters and some helper locks like hanlers' map RwLock. > Initially I've planned to keep same approach for FB3. But I did not > review latest Adriano's changes from this POV. I think we still have what to do to make code and interfaces more clear for end users. This is why i said that it is work in progress. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> What about the following - we always pass status vector to release(). I'm sorry but i against it. addRef\release is and well known pair and its usage also well known. I see no good reason to change it. More, if we need to change it, then we use it in a wrong way. > It's not a problem technically - IStatus is not refCounted. We use it in Call detach() is also not a problem technically, why don't use it when necessary ? Why we want to replace detach by release ??? > the following way - if refCounter > 1, no error happened (later it > always means that nothing like system call failed took place). If > refCounter == 1 and object has no special dtor, again no error happened. If object have no explicit destructor then release must be called, no problem. > But if we release something like transaction and due to it rollback some > job - transaction is released (what can we do?), but error (or may be > better warning? not sure...) is reported - job was implicitly rolled back. You can kill me, but i don't understand - why do we want to call release, if good special explicit destructor is present ??? ... >> Difference between "destoy" and "release" is semantics. I don't know how >> to explain it else. >> > > _I_ understand you. Good. > But not sure it's OK for documentation POV. Speaking about documentation i'd said we must document usage of every public interface and specify explicitly how instance is constructed and destroyed. > Notice - if we add status vector to release, the problem is almost gone. No, please. > We may just say: people, always check error status please. And add - use detach when needed and release in other cases :) Regards, Vlad PS You make me remember MS way to report errors in their DAC's. They have global Errors collection and it must be checked after every call of any object. Or remember GetLastError. Do you want something similar ? We can remove IStatus parameter from every method and add IStatus* IMaster::getStatus() to the master interface. Not sure i like it... -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
>>> the following way - if refCounter> 1, no error happened (later it >>> always means that nothing like system call failed took place). If >>> refCounter == 1 and object has no special dtor, again no error happened. >> If object have no explicit destructor then release must be called, no >> problem. >> > Your words are very incongruent. "Release" releases, and shall be valid > for any objects. Where i said it is not valid ? Current code not disabled it in any way. You can call release event for last reference if you wish. But this is wrong practice and should be avoided. > You broke the IAttachment from external engines, with are only > release-able. They should not be dropped or detached. If you would be more technical it will help to understand characters you wrote here. So far i don't understand what i broke, why IAttachment from external engines are only release-able while IAttachment from y-valve is detach-able (maybe something wrong in external engines ?) and why they should not be dropped or detached. I can guess that *internal attachment* should not be detach'ed but this is perfectly fits "my model" as it was not attach'ed by external engine code. But i can't even guess why *externla attachment* can't be detach'ed... > Making the release an error is like create private destructors. You > could do it, but difficultly you may justify it. If you'll play by proposed rules all will be ok. So far you gave zero technical reasons agains this rules, only emotions and offence. >>> But if we release something like transaction and due to it rollback some >>> job - transaction is released (what can we do?), but error (or may be >>> better warning? not sure...) is reported - job was implicitly rolled back. >> You can kill me, but i don't understand - why do we want to call >> release, >> if good special explicit destructor is present ??? >> >> ... > Why are we adding addRef/release after all? > > To support wrong code and solve nothing. With the cost of all this mess, > all this otherwise unneeded discussion, all the global/local pool mess. ...only emotions and offence... Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> Hey Vlad, you already made your excellent job of misrepresent a > technical discussion into personal one. Congratulations. Thanks. Do you have something else to say ? From technical side, please. ... > See, providers was committed a lot of time ago. The implementation of > (only) *reference counting* is still confusing and working in progress, It seems you are the only confused person (oh, i again "made my excelent job") > and the design changed two days ago by private decision, and it even > didn't considered already wrote and committed code. There was no changes in design. There was fix for huge memory leak after (your, btw) change. Regards, Vlad -- Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> PS: There is a paper > (http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf) from Scott > Meyers and Andrei Alexandrescu showing even or volatile usage in base > classes are wrong. Unfortunately it's down. With MSVC we are safe as we use "volatile" and compiler used read\write memory barriers (or acquire\release semantics) for access to volatile variables : http://msdn.microsoft.com/en-us/library/ms686355.aspx Can't said for other compilers (GCC at the first place). Regards, Vlad -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Semantics of release
> On 17-05-2011 19:58, Vlad Khorsun wrote: >>> PS: There is a paper >>> (http://www.aristeia.com/Papers/DDJ_Jul_Aug_2004_revised.pdf) from Scott >>> Meyers and Andrei Alexandrescu showing even or volatile usage in base >>> classes are wrong. Unfortunately it's down. >> >> With MSVC we are safe as we use "volatile" and compiler used read\write >> memory >> barriers (or acquire\release semantics) for access to volatile variables : >> >> http://msdn.microsoft.com/en-us/library/ms686355.aspx >> >> Can't said for other compilers (GCC at the first place). >> > Most usage of volatile to implement double-check locking pattern is in > platform neutral code, hence wrong. We can or use compiler-\arch- specific read\write barriers, or use interlocked access for InitMutex::flag and InitInstance::flag. At first look i'm not found very hot singletons at our code, so cost of full memory barriers (because of interlocked access) could be acceptable for us. Regards, Vlad -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Is correct the use of RAM in classic server
> The problem is that FB 2.5.1 requiere 60% more RAM than fb 2.1.4 for the > same database, same metadata and same process, and i think that 60% is so > much. If it is normal, is a problem for me to use classic server and fb 2.5. Could you provide minimal reproducible test case ? Regards, Vlad -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Is correct the use of RAM in classic server
>> > The problem is that FB 2.5.1 requiere 60% more RAM than fb 2.1.4 for the >> > same database, same metadata and same process, and i think that 60% is so >> > much. If it is normal, is a problem for me to use classic server and fb >> 2.5. >> >> Could you provide minimal reproducible test case ? ... >> > Yes of course, haw can i send to you both databases. The compressed size of > each one is around 8 MB. Ideally would be to upload it somewhere and send url to me (privately, if necessary) Regards, Vlad -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Is correct the use of RAM in classic server
> I have sent you the links to hv...@users.sourceforge.net Thanks. I reproduced it and comfirm that 2.5 used ~30% more memory then 2.1 on your database. The memory is used by big number of metadata objects. Why 2.5 needs more memory is investigated currently. Regartds, Vlad -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] File system cache vs. Firebird page cache?
Den 2011-06-10 11:11 skrev Pavel Cisar såhär: >> MaxFileSystemCache is intended for use with SuperServer, where really >> big FB cache on top of file system cache could be counter productive. >> Classic and SuperClassic work with very small FB cache (to conserve >> memory and lower IPC traffic to synchronize them between FB >> processes/threads). Pavel told about MaxFileSystemCache setting which controls if Firebird will use file system cache or not. This setting was introduced in v2.1 as FileSystemCacheThreshold and renamed in v2.5 into MaxFileSystemCache. > On the other hand, I experienced undesirable file system cache growth > for a web app with essentially a single connection with a Classic server > set to default page buffers (75?). > > I hade to make OS API calls to limit the file system cache. > > This is with a ~50 Gbyte database with two tables containing 150 million > records each and 50-100 other (smaller) tables. Especially on unindexed > queries on those large tables, the file system cache tended to grow too > large, causing performance degradation. > > So, what I'm saying is that MaxFileSystemCache may be useful for classic > as well, in some situations. I haven't tried it though. Still on 2.1. And you speak about FileSystemCacheSize setting which was introduced in v2.5 and allows to limit size of file system cache. This is system-wide setting and it is fully independent of Firebird's usage of file system cache. Regards, Vlad -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] File system cache vs. Firebird page cache?
>>> So, what I'm saying is that MaxFileSystemCache may be useful for classic >>> as well, in some situations. I haven't tried it though. Still on 2.1. >> >> And you speak about FileSystemCacheSize setting which was introduced >> in v2.5 and allows to limit size of file system cache. This is system-wide >> setting and it is fully independent of Firebird's usage of file system cache. > > I'm confused now. ;-) I don't find FileSystemCacheSize in firebird.conf, > thus I guess it isn't a V2.5 setting? It was introduced in v2.5 :) Probably you have copied firebird.conf from older version when installed your v2.5. > So, in respect to memory usage: Is the worst case that pages are cached > in both, Firebird page cache and OS file system cache? Is this possible? If Firebird is using file system cache (and by default it used) then double caching (by both FB and OS) is in place. If RAM is not enough for both caches (set of active pages in DB can't fit into each of the caches) there will be competition for memory and sooner of all OS will intensively use its own page file (AKA swap file) creating additional IO load. Therefore, if you plan to make huge page cache in Firebird it have sence to try to disable file system caching. By default, file system cache will be disabled for particular database if Firebird's cache for that database is more than 65536 pages. Note, Firebird have no such features as pre-fetch (or read-ahead) and multy-block IO, therefore it will be slower at physical IO than file system. Also, disabled file system cache means enabled Forced Writes, despite of database setting (at least on Windows). Regards, Vlad -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - Detect new statement in raw output
> Hello, > > while we have a strategy in the parser in FB TraceManager to detect a > new statement in the raw output, I wonder if there is an ultimate > solution for that? ;-) > > With several event types producing more or less an undefined number of > rows at the end of the raw output, I currently don't see some kind of > end of statement terminator character thingy in the raw output. > > Currently, we use the first line of a new statement to detect that it is > a new statement, but obviously, this won't get me some kind of end of > statement thingy for the last received statement in the raw output, thus > the parsed output somehow one statement "behind" the raw output. > > Any ideas/comments? Could you put example of such raw output and point me where you have a troubles with parsing ? Regards, Vlad -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API and PSQL
> Hello, > > I have the following two execute block calls: > > execute block > as > declare i Integer; > begin > i = 1; > while (i <= 10) do > begin > execute procedure p_donothing; > i = i + 1; > end > end > > > and: > > execute block > as > declare i Integer; > declare v Integer; > begin > i = 1; > while (i <= 10) do > begin > select 1 from rdb$database into :v; > i = i + 1; > end > end > > > I do see EXECUTE BLOCK and the various sub P_DONOTHING calls in the > trace output, but I don't see the individual SELECT 1 FROM ... calls in > case of the second EXECUTE BLOCK statement. > > Is this as designed? Yes. The whole PSQL block (except of calls of procedures, triggers and dynamic statements) is the single statement from the engine POV. Regards, Vlad -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - Detect new statement in raw output
> Hello Vlad, > >>> while we have a strategy in the parser in FB TraceManager to detect a >>> new statement in the raw output, I wonder if there is an ultimate >>> solution for that? ;-) >>> >>> With several event types producing more or less an undefined number of >>> rows at the end of the raw output, I currently don't see some kind of >>> end of statement terminator character thingy in the raw output. >>> >>> Currently, we use the first line of a new statement to detect that it is >>> a new statement, but obviously, this won't get me some kind of end of >>> statement thingy for the last received statement in the raw output, thus >>> the parsed output somehow one statement "behind" the raw output. >>> >>> Any ideas/comments? >> >> Could you put example of such raw output and point me where you >> have a troubles with parsing ? > > Imagine the following output: > > 2011-06-14T10:05:50.3790 (2032:00F3DC88) CLOSE_CURSOR > softnet_art.fdb (ATT_70, SYSDBA:NONE, NONE, TCPv4:127.0.0.1) > C:\Program Files (x86)\Upscene Productions\Database Workbench 4 > Pro\DBW4.exe:6064 > > Statement 111: > --- > select rf.rdb$field_name, f.rdb$field_type, f.rdb$field_sub_type, > rf.rdb$description, >rf.rdb$null_flag, rf.rdb$default_source as field_default, > f.rdb$default_source as domain_default, >f.rdb$field_scale, f.rdb$computed_source, rf.rdb$field_position, >f.rdb$field_length, f > ^^^ > PLAN SORT (JOIN (JOIN (F NATURAL, FD INDEX (RDB$INDEX_36)), RF INDEX > (RDB$INDEX_3, RDB$INDEX_4))) > > 2011-06-14T10:05:50.3790 (2032:00F3DC88) FREE_STATEMENT > softnet_art.fdb (ATT_70, SYSDBA:NONE, NONE, TCPv4:127.0.0.1) > C:\Program Files (x86)\Upscene Productions\Database Workbench 4 > Pro\DBW4.exe:6064 > > Statement 111: > --- > select rf.rdb$field_name, f.rdb$field_type, f.rdb$field_sub_type, > rf.rdb$description, >rf.rdb$null_flag, rf.rdb$default_source as field_default, > f.rdb$default_source as domain_default, >f.rdb$field_scale, f.rdb$computed_source, rf.rdb$field_position, >f.rdb$field_length, f > ^^^ > PLAN SORT (JOIN (JOIN (F NATURAL, FD INDEX (RDB$INDEX_36)), RF INDEX > (RDB$INDEX_3, RDB$INDEX_4))) > > > > There are two event types here: CLOSE_CURSOR, FREE_STATEMENT > > Currently we detect the end of statement thingy of CLOSE_CURSOR, by > parsing the first line of FREE_STATEMENT, because due to possibly > multi-line plan output etc., IMHO there is no reliable way to detect the > end of CLOSE_CURSOR (just an example here). > > So, currently, our parser knows, that CLOSE_CURSOR is finished, by > receiving the first line of the next event (FREE_STATEMENT). The end of > FREE_STATEMENT is detected when the next event is received. If > FREE_STATEMENT is the last event, the parser somehow thinks, that > FREE_STATEMENT isn't finished yet. > > I possibly could add some kind of timer, which e.g. notifies the parser > of the finished last event type, in case no other trace raw output is > received. > > A simplified example: > > A > B > C > D > E > ... > > where A, B ... are output for a particular event type. I need to receive > the first line of B to detect that the raw output of A is finished etc > ... Thus, my grid of parsed output is always one trace event behind the > raw output. Hmm... not sure if text below could satisfy you, but anyway : trace always puts whole event record into log, i.e. it is imposible to read half of record's text from a trace log. So, if next call of isc_service_query is blocked (or returns nothing, depending of how you call it) you can consider previous text as complete event record. Another possibility for you is to create own trace plugin and put into trace log (or whatever you choose) records in you own format. If speak about FB3 - you are free to propose new format for trace log and we'll consider it. Regards, Vlad -- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Interesting article about scalability of MySQL in Amazon (?) RDS
> On 06/22/11 11:13, Roman Rokytskyy wrote: >> Would be interesting to check how does Firebird behaves. > > Roman, we have some implementation of tpc-c, which works with firebird. > I often use it to check firebird stability under load. But telling true > it seems that it measures using some other tpmC. I think so because with > one of the latest tests with trunk I've got > TPC-C Throughput: 199378.00 tpmC > on >2 years old box (5 warehouse / 5 client, embedded connection). And > when the result differs >2 orders of magnitude (compared with 7000 in > that article), it looks like something different is measured. This is because you tested CPU-bound load. Make FW=ON and you'll see the difference :) Regards, Vlad -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Data protection magic? Nope - It's vRanger. Get your free trial download today. http://p.sf.net/sfu/quest-sfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] nbackup: CORE-3521 and CORE-2648
> Hello, > > currently investigating nbackup reliability/bugs. CORE-2648 is marked as > fixed in 2.1.4 and CORE-3521, which is currently open, seems to like > similar (forced writes vs. flushing to disc). > > How do they differ? CORE-2648 set FW mode for delta file in the same value as of main database file. CORE-3521 flushed delta file buffers when main database files buffers are flushed. This is different things. FW mode defines how OS file cache will handle writes. When FW is OFF then file system free to accumulate writes in file cache and do actual writes when OS think it is necessary\feasible. When FW is ON then file system still cached data pages but not delayed physical writes to disk. Flushing is different thing. It instructs OS to write all dirty cached buffers of given file immediately. Firebird calls flush every time some transaction is finished and on disconnect (look also at MaxUnflushedWrites and MaxUnflushedWriteTime settings at firebird.conf). Note, when FW is ON then OS shouldn't have dirty buffers and Firebird do not flushed its files. So, before CORE-2648 was fixed, delta file was always opened in FW=OFF mode despite of database settings. Before CORE-3521 was fixed delta file contents was not flushed to disk when transactions finished. Hope it is clear, Vlad PS CORE-3521 is not marked as fixed because it is not backported into 2.1.5 still. -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on "Lean Startup Secrets Revealed." This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] nbackup: CORE-3521 and CORE-2648
>> So, before CORE-2648 was fixed, delta file was always opened in FW=OFF >> mode >> despite of database settings. > > So, if the server crashes, we have a problem here? If database setting is FW = ON but (before the fix) delta file is opens with FW = OFF then Firebird crash in stalled (or merge) mode could corrupt delta file contents. Note, it *could be* corrupted but it is not necessary since OS still have dirty pages in its file cache and will write it on disk sooner or later. But if OS is crashed then corruption is very possible. > Or does writing the > delta file also take both config options mentioned above into account? Of course. >> Before CORE-3521 was fixed delta file contents was not >> flushed to disk when transactions finished. > > So, without a fix for CORE-3521, cached dirty pages aren't written to > disk while nbackup is running at all? If FW = ON, then this bug should have no impact. If FW=OFF then *on disk* contents of delta file fully depends on OS and its policy of delayed writes. I.e. it is OS decides when to write dirty pages on disk. Since bug is fixed we have equal safety in any nbackup mode. > Do both issues relate to corrupting the database in any way? Probably. Regards, Vlad -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on "Lean Startup Secrets Revealed." This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] nbackup: CORE-3521 and CORE-2648
Before CORE-3521 was fixed delta file contents was not flushed to disk when transactions finished. >>> >>> So, without a fix for CORE-3521, cached dirty pages aren't written to >>> disk while nbackup is running at all? >> >> If FW = ON, then this bug should have no impact. > > FW = ON for the database or the delta file? If it is the delta file, > then I have to use an engine version with CORE-2648 fixed as well. For database of course. We have no way to set FW for delta separately :) Regards, Vlad -- AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries, the creator of the Lean Startup Methodology on "Lean Startup Secrets Revealed." This video shows you how to validate your ideas, optimize your ideas and identify your business strategy. http://p.sf.net/sfu/appsumosfdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] CORE-2670: exclusive mode
> On the FB 2.5 change log I found: > > 2009-12-04 11:44 hvlad >M src/remote/server.cpp > Fixed bug CORE-2670 : FB 2.5 freezes during data pump operations It was the typo in check-in's comment, sorry. That fix was for CORE-2760. Regards, Vlad -- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos & much more. Register early & save! http://p.sf.net/sfu/rim-blackberry-1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Failure to read database file in embeddeddeployment
> 2. Constrained by the engine i must use RootDirectory so i can set the
> file location. Changing the variable in the conf file had no affect.
> The only way i could make the RootDirectory change is by setting
> Environment variable, but unfortunately thats not a good for solution
> for my application deployment.
All you need is to call SetEnvironmentVariable("FIREBIRD", )
*before* loading fbembed.dll
Regards,
Vlad
--
BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
The must-attend event for mobile developers. Connect with experts.
Get tools for creating Super Apps. See the latest technologies.
Sessions, hands-on labs, demos & much more. Register early & save!
http://p.sf.net/sfu/rim-blackberry-1
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Failure to read database file in embedded deployment
> Thanks Vlad, i tried that as well before posting here, but it didn't work. > If i do get this to work, will i be able to set a relative path to the DB > file? Sorry, i'm not sure i understand what does you did, what you got and what you really need. More details could help us to help you ;) Usually you need to change default RootDirectory to specify place of security2.fdb, firebird.msg and firebird.log. Before v2.5 RootDirectory for embedded was defined as one folder upper than application's folder. In v2.5 RootDirectory is defined as fbembed.dll folder. Why do you want to pass relative path to the DB file into engine ? It is considered as bad practice. Much better is to resolve relative path into absolute before attaching to the database. Regards, Vlad -- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos & much more. Register early & save! http://p.sf.net/sfu/rim-blackberry-1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Failure to read database file in embeddeddeployment
>I will start by describing the environment i have now. > My application code is running inside a game engine Unity3D. The code > is C# using mono which supports .NET 2.0. > The DB i am interested in is the embedded deployment, so when i ship > the game users don't need to install anything else. > > The problem starts with the engine game packaging. I needed to place > the DLL in one place adn the DB file in a different place. I followed > the instructions here: > http://www.firebirdsql.org/manual/ufb-cs-embedded.html > and so my folder tree looked like this: > ... > GameDirectory-> > Assets > Plugins > FirebirdSql.Data.FirebirdClient.dll > fbembed.dll > cu*.dll > ib_util.dll > firebird.conf -> RootDirectory = d:\GameDirectory\Assets\DB > DB > firebird.msg > GAMEDB.FDB > When running the game i with this connection: > FbConnectionStringBuilder cs = new FbConnectionStringBuilder(); > cs.ServerType = FbServerType.Embedded; > cs.Database = "GAMEDB.FDB"; > cs.UserID = "SYSDBA"; > cs.Password = "masterkey"; RootDirectory have nothing common with database path. Don't change it and use full path to the database file at your connection string. Regards, Vlad -- uberSVN's rich system and user administration capabilities and model configuration take the hassle out of deploying and managing Subversion and the tools developers use with it. Learn more about uberSVN and get a free download at: http://p.sf.net/sfu/wandisco-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Failure to read database file in embedded deployment
> Vlad, > > At 08:19 AM 9/08/2011, Vlad Khorsun wrote: > >>Usually you need to change default RootDirectory to specify place of >>security2.fdb, firebird.msg and firebird.log. Before v2.5 RootDirectory for >>embedded was defined as one folder upper than application's folder. In >>v2.5 RootDirectory is defined as fbembed.dll folder. > > I think that the root directory for embedded on Windows was always the same > folder as the one where *both* fbembed.dll (renamed to fbclient.dll for > application > code that required a client named fbclient.dll) and the user application > executable > file were located. Other dirs (as needed, e.g., \UDF, \doc, \intl) are hung > beneath > this "application directory", as before. What looks different to me is that > the icu* > libraries in the 2.5 distribution package are in this root directory, not in > a \bin subdir > as previously. Root directory for windows embedded was changed according to this ticket : http://tracker.firebirdsql.org/browse/CORE-1814 See also http://tracker.firebirdsql.org/browse/CORE-2719 > From your comment on this thread, it seems at least *unclear* now, how the > deployment > structure should be in v.2.5. As I am building what *I hope* will be the > final build of release > notes for v.2.5.1 today, an update to the installation section (at least) > seems called for Deployment rules on Windows is almost the same. We just allow embedded library (with all additional binaries, etc) to reside anywhere (not necessary at the same folder as an application's .exe file). Regards, Vlad -- uberSVN's rich system and user administration capabilities and model configuration take the hassle out of deploying and managing Subversion and the tools developers use with it. Learn more about uberSVN and get a free download at: http://p.sf.net/sfu/wandisco-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] CORE-2670: exclusive mode
> And what about CORE-2670?. Is there any way or switch to avoid sharing while > a database sesion over the file is open?. It was explained at ticket. Database file opened in the exclusive mode by the: a) SuperServer engine in all versions and platforms b) Windows embedded engine before v 2.5 Regards, Vlad PS In Firebird 3 will be per-database setting for shared\exclusive mode. -- uberSVN's rich system and user administration capabilities and model configuration take the hassle out of deploying and managing Subversion and the tools developers use with it. Learn more about uberSVN and get a free download at: http://p.sf.net/sfu/wandisco-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
>I am currently going over the XA implementation in Jaybird, and this > prompted some questions regarding transactions, connections and > two-phase commit in Firebird. > > First of all: are transactions associated with the connection that > created it, or is it possible to 'share' the same transaction handle > with different connections? In other words: Can I use a single > transaction handle for multiple connections that are enlisted in the > same distributed transaction? When you start distributed transaction, you pass handles of all participated attachments and got single common handle of distributed transaction. This handle is valid within every attachments above. In FB3 we are going to implement additional API call to join two or more pre-existing transactions. It will return new handle of new distributed transaction, iirc. > Second: if I read the IB6 ApiGuide.pdf correctly, then a single > connection can have multiple transactions open in parallel, is that a > correct interpretation? Yes. And this is unique feature of IB\FB. > Third: Can Firebird handle enlistment of multiple transaction handles in > a single two-phase commit, or would this require using a single > transaction handle for a single transaction branch? Not sure i understand you correctly... Attachment could participate in few separate distributed transactions at the same time (if this was your question). Hope this helps, Vlad -- EMC VNX: the world's simplest storage, starting under $10K The only unified storage solution that offers unified management Up to 160% more powerful than alternatives and 25% more efficient. Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
> On 27-8-2011 14:01, Vlad Khorsun wrote: >>> I am currently going over the XA implementation in Jaybird, and this >>> prompted some questions regarding transactions, connections and >>> two-phase commit in Firebird. >>> >>> First of all: are transactions associated with the connection that >>> created it, or is it possible to 'share' the same transaction handle >>> with different connections? In other words: Can I use a single >>> transaction handle for multiple connections that are enlisted in the >>> same distributed transaction? >> >> When you start distributed transaction, you pass handles of all >> participated attachments and got single common handle of distributed >> transaction. This handle is valid within every attachments above. > > What if I don't know all attachments that will be involved in the same > distributed transaction when the first attachment is enlisted? It looks > like it would be impossible to enlist an attachment later on. And we are going to address this issue in FB3, as i told below :) >> In FB3 we are going to implement additional API call to join two or >> more pre-existing transactions. It will return new handle of new distributed >> transaction, iirc. > > Wouldn't it be better to join a new attachment to an existing > distributed transaction? No. Attachment *itself* can't participate in distributed transaction. At least not directly. Transaction of this attachment - can. Just understand: attachment *is not a* transaction. When i tell "attachment participate in distributed transaction" i mean "some transaction of given attachment is a part (or sub-transaction) of given distributed transaction". > As far as I can see in the XA specification[1], > having a pre-existing transaction join a new distributed transactions > would be incorrect. Could you point me at exact place in 94-page document ? ;) I suspect XA spec implicitly mixed terms "transaction" and "attachment" as most DBMS allows just one active transaction per attachment... >>> Second: if I read the IB6 ApiGuide.pdf correctly, then a single >>> connection can have multiple transactions open in parallel, is that a >>> correct interpretation? >> >> Yes. And this is unique feature of IB\FB. > > Ok, so this would essentially mean that a single attachment could be > involved in multiple distributed transactions at the same time. Are > there any gotchas with doing that? None that i know. >>> Third: Can Firebird handle enlistment of multiple transaction handles in >>> a single two-phase commit, or would this require using a single >>> transaction handle for a single transaction branch? >> >> Not sure i understand you correctly... Attachment could participate in >> few separate distributed transactions at the same time (if this was your >> question). > > This is essentially restating that I would like to know if it is > possible to have multiple attachment work as a single transaction when > performing a two-phase commit. Two phase commit commits exactly that (distributed) transaction which you ask to commit. If attachment participate in more than one distributed transactions every of them should be committed separately. Example: tx1 = startTran(att1, att2); tx2 = startTran(att1, att3); // at this point att1 participate in two separate distributed transactions // while each att2 and att3 participate in one distributed transaction tx1.commit // tx2 still active // att1 and att3 still participate in distributed transaction tx2 txc2.commit Regards, Vlad PS If i understand you wrong, just keep asking. If necessary ;) -- EMC VNX: the world's simplest storage, starting under $10K The only unified storage solution that offers unified management Up to 160% more powerful than alternatives and 25% more efficient. Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
>> > Wouldn't it be better to join a new attachment to an existing >> > distributed transaction? >> >> No. Attachment *itself* can't participate in distributed transaction. At >> least >> not directly. Transaction of this attachment - can. Just understand: >> attachment >> *is not a* transaction. >> >> When i tell "attachment participate in distributed transaction" i mean >> "some >> transaction of given attachment is a part (or sub-transaction) of given >> distributed >> transaction". > > Certainly, we can make API look like we _do_ join a new attachment to an > existing distributed transaction. To do so we just start new transaction > in API call internally (need to know TPB...) and join it with existing > distributed transaction. But I see no '+' in this approach compared with > joining 2 transactions into new distributed transaction. I also see no reason to make such unclear hack. Regards, Vlad -- EMC VNX: the world's simplest storage, starting under $10K The only unified storage solution that offers unified management Up to 160% more powerful than alternatives and 25% more efficient. Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
> 28.08.2011 15:52, Alex Peshkoff wrote: >> Certainly, we can make API look like we_do_ join a new attachment to an >> existing distributed transaction. To do so we just start new transaction >> in API call internally (need to know TPB...) and join it with existing >> distributed transaction. > > What would be the purpose of such action? > If I started distributed transaction at 00:00 and joined an attachment to > it at 00:15, > I will see data in this database as they were at 00:15 instead of 00:00 as I > would expect. Your expectation is wrong as it is *you* who joined new attachment too late. I mean - you wrote the code and you have control when to start transactions and what to expect. 2PC protocol is about *commit*, not about "distributed consistency". Regards, Vlad -- EMC VNX: the world's simplest storage, starting under $10K The only unified storage solution that offers unified management Up to 160% more powerful than alternatives and 25% more efficient. Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
> The XA semantics are a bit complicated, but essentially the database is a > Resource Manager (RM), and a distributed transaction is assigned an Xid by > the Transaction Manager (eg in the case of JavaEE it could be the > application server). If I read the spec correctly, it is expected that a > connection to the RM can be part of the Xid (start), be requested to end it > is work (not prepare / commit!), then any connection (different or same) to > the RM can again part of that same Xid (start), etc. Up to this point i see no need in special support from the Firebird. XID is generated by TM, attachment details and interaction with TM is handled by Jaybird (?). > And finally the RM > specific work of the distributed transaction should be preparable and > commitable by any connection to the RM. This is where Firebird support is necessary. Below i'll show that it is already present in the engine. > In that system the Xid is the starting point, so joining two transaction > together into a new distributed transaction makes no sense, as the Resource > Manager (Firebird) is not the owner of the Xid, but just one of the > entities involved in the distributed transaction. Yes. From the TM point of view there is no need in ability to join local transactions at Firebird level. So, lets forget about it :) > Maybe it is an idea to provide an XA compliant facade for Firebird? I see no problem to implement such facade for any existing Firebird version. All necessary bits is already present in Firebird, AFAIU. Look at isc_prepare_transaction2() API. It could attach *any* application-specific message to the local transaction before actual commit. TM could pass XID + "something to identify this part of distributed transaction" into isc_prepare_transaction and later read this info back from RDB$TRANSACTIONS at recovery phase, if necessary. And\or TM could store above "something to identify this part of distributed transaction" + local TxID of every part of distributed transaction into its own storage before start distributed commit (to identify record at RDB$TRANSACTIONS later at recovery phase). Look also at isc_reconnect_transaction() API which allows to re-connect to the in-limbo transaction at recovery phase and finally commit or rollback it. Regards, Vlad PS I'm not sure it was clear explanation, so let's discuss it... -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
>> Look at isc_prepare_transaction2() API. It could attach *any* >> application-specific >> message to the local transaction before actual commit. TM could pass XID + >> "something >> to identify this part of distributed transaction" into >> isc_prepare_transaction and later read >> this info back from RDB$TRANSACTIONS at recovery phase, if necessary. And\or >> TM >> could store above "something to identify this part of distributed >> transaction" + local TxID >> of every part of distributed transaction into its own storage before start >> distributed commit >> (to identify record at RDB$TRANSACTIONS later at recovery phase). >> >> Look also at isc_reconnect_transaction() API which allows to re-connect >> to the in-limbo >> transaction at recovery phase and finally commit or rollback it. > > This code should be already working in Jaybird since day 0 (David Jencks > implemented it in the first version of the driver). The recovery process > uses message to query the in-limbo transactions, is able to join them > and is able to commit or rollback them. Very good :) > However, if I remember correctly, the requirement to support transaction > independent from the attachment came from the fact that TM may be asked > to activate a transaction that "belongs" to the attachment that is being > currently used and is associated with a different transaction. Here we > talk about normal processing, not the recovery process. I guess the main > issue here is the synchronization of the wire communication, so that two > transactions are properly interleaved and server is able to handle the > communication. Before v2.1 it should be able to handle it properly. Since v2.1 it must be able to handle it properly. I mean - it should not hung and handle "asyncronous" TM call after processing of current "syncronous" call on the same attachment. > Another issue here is that when a non-recoverable error > happens (e.g. request synchronisation error), Hmm... is it unrecoverable error ? BTW, request synchronisation error could happen only at fetching records, iirc... > we have to kill all > transactions that belong to that attachment, even if only one is in trouble. Even if we have to kill all transactions - what is the problem for TM ? > But this requirement is so old, that I do not remember exact details. :) > The only thing I remember is that it becomes obvious when somebody > starts refactoring the XA part (which is one of the core layers there). > That happened to me when I refactored that code few years ago, that's > happening to Mark right now. :) Let's solve this puzzle together ;) Regards, Vlad -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Association of transactions, connections, and two-phase commit
>> Look also at isc_reconnect_transaction() API which allows to >> re-connect to the in-limbo >> transaction at recovery phase and finally commit or rollback it. > > As far as I have seen, this is already done in Jaybird. Very good. > So the commit can > be done from any connection, but what about the prepare? My impression from > this discussion is that the prepare would need to happen from the same > connection which started the transaction on Firebird, is that right? Prepare (and regular commit\rollback) must be done by the attachment which was started transaction, of course. Commit\rollback *after reconnect* could be done by attachment which reconnects to the in-limbo (prepared) transaction after some kind of failure where original attachment was lost. >> PS I'm not sure it was clear explanation, so let's discuss it... > > I will go over the whole discussion again later this week and come up with > more questions :) Ok Regards, Vlad -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] fb 2.1.4 pdb windows 2008
> I've got a bugcheck in 2.1.4 final (pdb) that I'd like to report. It's under > windows server 2008, which doesn't have the normal drwtsn32, so I used ADPlus > instead (see instructions at http://blogs.technet.com/b/askperf/ > archive/2007/06/15/capturing-application-crash-dumps.aspx). Can you guys use > those dumps, or do you need me to use one of the other methods listed? > > Which files do you need (dmp, log, etc.), and where should I send them? There > was > an old post from Vlad telling someone to zip (and remove the extension) and > email > to him, but what's the current process? JIRA? Send it to me (yes, zipped and without extension) and, if it is something new, i'll ask you to create new ticket at tracker. Thanks, Vlad -- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA Learn about the latest advances in developing for the BlackBerry® mobile platform with sessions, labs & more. See new tools and technologies. Register for BlackBerry® DevCon today! http://p.sf.net/sfu/rim-devcon-copy1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Support conditional trigger firing (WHEN clause ala Oracle)
> At the first glance it's possible to avoid loading trigger's body with > WHEN clause. Should we check permissions of trigger's code ? If yes - we will load it anyway at query (insert\update\delete) prepare time. Regards, Vlad -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Support conditional trigger firing (WHEN clause ala Oracle)
> On 10/10/11 11:53, Vlad Khorsun wrote: >>> At the first glance it's possible to avoid loading trigger's body with >>> WHEN clause. >> Should we check permissions of trigger's code ? If yes - we will load >> it anyway at query (insert\update\delete) prepare time. > > We should not check permissions of trigger in that case. Imagine clause: > > WHEN CURRENT_USER != 'SYSDBA' If such clause is inside trigger code then it not affects security checks and *whole* trigger code is checked. SQL standard says something about security checks for conditional triggers ? Regards, Vlad -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Support conditional trigger firing (WHEN clause ala Oracle)
> 10.10.2011 12:26, Vlad Khorsun wrote: > >> If such clause is inside trigger code then it not affects security checks and >> *whole* trigger code is checked. SQL standard says something about security >> checks for conditional triggers ? > > I believe the rules are the same. In general, it's impossible to > evaluate the trigger condition at the prepare time (it can refer to the > fields of the current record), so I strongly believe we shouldn't even > try doing that. > > So IMO this feature is not going to affect the body parsing and > permission checking. Exactly. And we still must load all active PSQL triggers despite of any declared condition. Seems conditional triggers have sence only for external (non-PSQL) triggers. Regards, Vlad -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 2.5.1 Winx64 Trace problem - crash dumpavailable
> Hello, > > A crash dump is available here: > http://www.iblogmanager.com/download/misc/fb251_x64_crashdump.zip I see why it is crashed (during error reporting about failed read from config file) and i can fix it. But i don't see why read was failed. And this is the real issue. Could you point me how to reproduce crash ? Thanks, Vlad -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 2.5.1 Winx64 Trace problem - crash dumpavailable
> Thanks to Vlad and Alex the 2.5.2 snapshot from today (Oct. 17th) works > fine so far from a Trace API POV. ;-) Thanks for confirmation :) > One question though, which isn't specific to the 2.5.2 snapshot. I've > seen that the various fb_trace* files in e.g. c:\programdata\firebird > aren't cleaned up. While they do as long as no trace data has been > produced, they won't be cleaned up once there was trace data for the > particular session, even if the trace session is stopped properly. Trace files could be not deleted only if Firebird was not shutted down properly. This is specific Windows issue and we plan to adress it in the future. Regards, Vlad -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2d-oct Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] news from kernel 3.1
> 25.10.2011 14:03, Adriano dos Santos Fernandes wrote: > >> 2) FW=OFF, and use fsync on COMMIT - pages will not be reordered, and >> when COMMIT happens they will be written to disk in order > > I believe this is wrong assumption. Nobody guarantees that OS will be > flushing the dirty pages from the file-system cache to disk in the > original order. Same opinion. Regards, Vlad -- The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Lock error in CCH
> Running CORE-3650 test-case in different isql embedded sessions with > SharedCache and SharedDatabase, I had (one time) a hang and a crash. The > crash happened after the first command. I ran it in session1, session2, > session1, etc (or some variant). This is more or less expected. When shared page cache was introduced i said that lock manager is not ready to work in this mode. Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Sweep Start/Finish via Trace API?
> I've checked the various servivce filters available, but haven't found > anything in respect to tracing a sweep operation. > > Is it technically possible to add some kind of sweep start/finish events > in the Trace API for both, manual and automatical (sweep interval) > triggered? Sure. Feel free to fill new ticket at tracker :) Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Still on the ext4 slowness...
> Let's remember the problem I initially observed. By default ext4 uses > barriers and that makes it much slower than ext3 (when ext3 wasn't using > barriers by default). > > But it slower only whens FW=ON. With FW=OFF there is almost no difference. > > Barriers are supposed to act on file's metadata only, but the speedy > problem we see don't make that appear to be true. > > Firebird don't use O_SYNC but O_DSYNC instead. O_DSYNC is supposed to > sync the data and only the metadata necessary to retrieve the data > correctly. > > O_DSYNC and O_SYNC is not always different on the implementation, but at > least in Kernel 2.6.38 the implementation are surely different and > O_DSYNC is faster. > > So considering a database which is already preallocated on the > filesystem and Firebird don't create any new file pages, barriers would > not influence the performance. But it's slower even on this condition. > > So I think not everything is well know on this problem, but I did some > tests. A thing I saw is that opening a file with O_DSYNC and opening a > file without it but calling fdatasync after each write has almost equal > performance. Did you tried single connection or concurrent writes of multiply connections also ? > I really don't know well how Firebird does the page precedence graph, > but if we could write all "independent" (say, the ones with the order > they're write didn't matter) then call fdatasync and continues, would be > much better than O_DSYNC mode. > > Is it possible? How generally is that precedence graph, does it have to > many "independent" pages on it? There is 2 general places when Firebird writes pages. All of them, of course, used precedence graph. See below: a) flush on commit\rollback\detach\background_gc : Pages (bdb's) to be flushed are collected into array, array is sorted by page numbers and then pages which have no dependent pages are written (in page numbers order). Of course, after page write, its dependency is cleared, so after first write pass we have another set of "independent" pages. This process continues until all pages from array is written. b) "single" page write when old dirty page is eliminated from the page cache Page cache choosed oldest dirty page and attempt to write it. But if there is another dirty pages which is dependent on "victim" page - they will be written before "victim" page. This is recursive process as currently to be written page could have another dependent pages which must be written first. Same "single" write occured also when : - page lock is downgraded and page is dirty (classic only) - new precedence relationship will make a circle in precedence graph - dirty page buffer is marked as system (or must_write) and page is released - cache_writer thread (SS only) writes an oldest dirty page You see - in case (a) we have more or less group writes while in case (b) we have mostly single write (but in some unfortunate cases it could affect many pages). Case (a) is optimized and we can call fdatasync() after each pass of writes of independent pages. In case (b) we have no such possibility, at least in current code. Hope this helps, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Still on the ext4 slowness...
>>> I really don't know well how Firebird does the page precedence graph, >>> but if we could write all "independent" (say, the ones with the order >>> they're write didn't matter) then call fdatasync and continues, would be >>> much better than O_DSYNC mode. >>> >>> Is it possible? How generally is that precedence graph, does it have to >>> many "independent" pages on it? >> >> There is 2 general places when Firebird writes pages. All of them, of >> course, >> used precedence graph. See below: >> >> a) flush on commit\rollback\detach\background_gc : >> >> Pages (bdb's) to be flushed are collected into array, array is sorted by >> page numbers >> and then pages which have no dependent pages are written (in page numbers >> order). >> Of course, after page write, its dependency is cleared, so after first write >> pass we have >> another set of "independent" pages. This process continues until all pages >> from array is >> written. >> >> b) "single" page write when old dirty page is eliminated from the page cache >> >> Page cache choosed oldest dirty page and attempt to write it. But if >> there is another >> dirty pages which is dependent on "victim" page - they will be written >> before "victim" page. >> >> This is recursive process as currently to be written page could have >> another dependent >> pages which must be written first. >> >> Same "single" write occured also when : >> - page lock is downgraded and page is dirty (classic only) >> - new precedence relationship will make a circle in precedence graph >> - dirty page buffer is marked as system (or must_write) and page is released >> - cache_writer thread (SS only) writes an oldest dirty page >> >> You see - in case (a) we have more or less group writes while in case >> (b) we have mostly >> single write (but in some unfortunate cases it could affect many pages). >> Case (a) is optimized >> and we can call fdatasync() after each pass of writes of independent pages. >> In case (b) we >> have no such possibility, at least in current code. >> > > So I suppose if fdatasync does not interfere when used in multiple > threads, it may not make case (b) noticeable slower, but case (a) much > faster. If your guess is correct (that O_DSYNC makes barrier after every single write) - then yes, it should be better to remove O_DSYNC and add fdatasync() after every pass of flush of group of independent pages > Does (b) happens only when there is no free buffer for a another page? I enumerated 5 cases above. > May it be changed to eliminate not only one independent dirty pages, but > a number of it in one pass? I thought about to change cache_writer in this direction - make it to do "mini-flush" instead of writting one page at time. But, note, we have no cache_writer thread in CS\SC and we can do a little (almost nothing) when page lock is downgraded (often case in CS, i think). I don't think that writting more than one dirty page when free buffer is needed is good idea as it could delay user process significantly. I could be wrong... Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Still on the ext4 slowness...
> On 11/14/11 02:17, Vlad Khorsun wrote: >> I thought about to change cache_writer in this direction - make it to do >> "mini-flush" >> instead of writting one page at time. But, note, we have no cache_writer >> thread in CS\SC >> and we can do a little (almost nothing) when page lock is downgraded (often >> case in CS, i >> think). >> > > What prevents us from adding cache writer to classic? Cache writer is useful for large cache. It adds almost none or even could drop performance for small cache (which is typical for CS). Small cache often overfilled during one operation and dirty pages written as result of preemption by new pages. Technically we can enable cache writer for CS but i doubt it is good idea and i'm afraid to add more threads per process on Linux. >> I don't think that writting more than one dirty page when free buffer is >> needed is good >> idea as it could delay user process significantly. I could be wrong... >> > > Worth checking... > Remember how changing FW affects tpc/c performance on small database? It was expected :) My doubt is that we will write more pages than necessary. Imagine small cache (CS) and set of actively modifying pages. Currently we will write them one-by-one when new buffer is required. You offer to write, say, 2 pages at time and replace one of them. Second written page could be modified again and we will write it again, so first write was not needed. Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - What's the unit for number of (reads |fetches ...)
> as the subject says. What's the unit for the resulting values for number > of reads, writes, fetches, marks? Just want to be sure if a particular > result makes sense here. ;-) Same as for isql's statistics - number of operations. Note, we have no operations on group of pages, so, for ex. N writes means N times write and every write is length of one page. Hope this helps, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - What's the unit for number of (reads |fetches ...)
> Hi Vlad, > >>> as the subject says. What's the unit for the resulting values for number >>> of reads, writes, fetches, marks? Just want to be sure if a particular >>> result makes sense here. ;-) >> >> Same as for isql's statistics - number of operations. Note, we have no >> operations on group of pages, so, for ex. N writes means N times write >> and every write is length of one page. > > So, the numbers I see is number of pages? Yes > I have the following trace output for a select count(*) on a table with > ~100.000 records. > > --- > select count(*) from accommodation > ^^^ > PLAN (ACCOMMODATION NATURAL) > 1 records fetched > 81 ms, 7149 read(s), 214192 fetch(es) > > > If it is pages, then this would mean ~ 800MB (4K page size) fetched > from the cache? Does this make sense? Fetch is not a physical read. This is more like reference. When Firebird need access to page buffer (to get record, for ex.) it asks page cache to "fetch" page buffer. If this buffer is not present at page cache, it will be read from disk first. In your example we see 7149 physical reads and 214192 times this pages was referenced by the engine. To read a record engine needs to access pointer page and (at least one) data page. You have ~100K records so we can explain ~200K fetches. Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace - Character set in output
> Hello, > > I have the following character set (ISO88591) information in the trace > output: > > 2011-10-27T20:58:16.5210 (920:00D4DA30) COMMIT_TRANSACTION > fbtm_unittest.fdb (ATT_2, FBTM:NONE, ISO88591, TCPv4:127.0.0.1) > G:\Development\FBTraceManager\svn\trunk\qa\unittest\isql.exe:4520 > (TRA_5, READ_COMMITTED | NO_REC_VERSION | WAIT | READ_WRITE) > 10 ms, 15 read(s), 43 write(s), 394 fetch(es), 29 mark(s) > > > While it should be ISO8859_1. Not a big deal, but can this be made > consistent with character sets available in RDB$CHARACTER_SETS? Hmm, interesting... These names come from /intl/cs_narrow.cpp and it seems they was not changed since IB6 codebase. Probably Adriano knows if we can change them to be the same as in RDB$CHARACTER_SETS and other places (such as fbintl.conf). Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - What's the unit for number of (reads |fetches ...)
- Original Message - From: "Thomas Steinmaurer" To: "For discussion among Firebird Developers" Sent: Tuesday, November 15, 2011 12:11 AM Subject: Re: [Firebird-devel] Trace API - What's the unit for number of (reads |fetches ...) > Hi Vlad, > > as the subject says. What's the unit for the resulting values for number > of reads, writes, fetches, marks? Just want to be sure if a particular > result makes sense here. ;-) Same as for isql's statistics - number of operations. Note, we have no operations on group of pages, so, for ex. N writes means N times write and every write is length of one page. >>> >>> So, the numbers I see is number of pages? >> >> Yes >> >>> I have the following trace output for a select count(*) on a table with >>> ~100.000 records. >>> >>> --- >>> select count(*) from accommodation >>> ^^^ >>> PLAN (ACCOMMODATION NATURAL) >>> 1 records fetched >>> 81 ms, 7149 read(s), 214192 fetch(es) >>> >>> >>> If it is pages, then this would mean ~ 800MB (4K page size) fetched >>> from the cache? Does this make sense? >> >> Fetch is not a physical read. > > I know. I said fetched from the cache. > > This is more like reference. When Firebird need >> access to page buffer (to get record, for ex.) it asks page cache to "fetch" >> page >> buffer. If this buffer is not present at page cache, it will be read from >> disk first. >> >> In your example we see 7149 physical reads and 214192 times this pages >> was >> referenced by the engine. To read a record engine needs to access pointer >> page and >> (at least one) data page. You have ~100K records so we can explain ~200K >> fetches. > > Ok, but is there a way then to tell how many pages have been fetched > from the cache as the number above for fetched is more likely > "referenced" and not real number of pages fetched from memory? What do you understand under "fetched from memory" ? There is no operations like memmove or so on. Fetch is just : - find buffer by given page number - lock it - return it address Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - Service names in include filter?
> the fbtrace.conf sample file has examples on what services names one > can/should use when using e.g. the include filter. I'm not sure if the > following is ok or as designed, thus I'm checking here ... > > I want to trace Database Stats and I tried the following include filter > samples: > > 1) include_filter %Database Stats% > 2) include_filter Database Stats > 3) include_filter %Stats% > 4) include_filter "Database Stats" > > While 3) and 4) gave me a trace for e.g. when querying the header page, > 1) and 2) didn't, which looks a bit weird, especially as: > > Database Stats > > is mentioned in the fbtrace.conf file. > > > Is there an inconsistency here somewhere? The config file parser came from Vulcan and it is more like generic XML-style parser. It allows to have more than one value for given setting, i.e. include_filter %Database Stats% is interpreted as setting name : include_filter value1 : %Database value2 : Stats% Caller code (trace plugin config reader) don't look for 2nd and all other values. So, i recommend to enclose values with embedded spaces into quotes to avoid confusion. Regards, Vlad -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Trace API - What's the unit for number of (reads |fetches ...)
> Hi Vlad, > > >>In your example we see 7149 physical reads and 214192 times this pages >> was >> referenced by the engine. To read a record engine needs to access pointer >> page and >> (at least one) data page. You have ~100K records so we can explain ~200K >> fetches. >> >> > I think that when doing an exhaustive retrieval like this select count (*), > Firebird doesn't go back to the pointer page for each record but reads > every record on the page before it goes back to the pointer page, starting > the page/line index entry 0. For an indexed read, it would reference the > pointer page and the page/line index. Unfortunately, it does. When record is read, data page is released and should be fetched again. It means access of pointer page first as after DP was released world could be changed and that DP could be released and even new DP could be reallocated at the same slot. Regards, Vlad PS Snapshot (concurrency) transaction guarantees that once read record could be read again and will be the same. So, in theory, we can just re-read same data page when looking for next record. But read-committed transactions don't have such guarantee and we must fetch PP first. -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
