Re: [Firebird-devel] isc_dpb_dbkey_scope
https://en.wikipedia.org/wiki/Multiversion_concurrency_control#History -Eredeti üzenet- Feladó: Pól Ua L. via Firebird-devel [mailto:firebird-devel@lists.sourceforge.net] Küldve: 2022. május 26., csütörtök 9:11 Címzett: For discussion among Firebird Developers Másolatot kap: Pól Ua L. Tárgy: Re: [Firebird-devel] isc_dpb_dbkey_scope Hi Jim, great to see that you're still keeping your finger in the Firebird pie! > There is a very good reason that multi-version concurrency control won > out big time (Wikipedia lists 80+ database systems with MVCC). > Interbase was the second. Just as a matter of historical interest, which was the first database to release with MVCC? Oracle? Best regards, Pól... > Jim Starkey, AmorphousDB, LLC Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Tablespaces proposal
"Tablespaces has meaning for large databases only that don't fit into single storage (terrabytes)." That is not true. It has meaning whatever the programmers meant to use it. It might not be about read performance, but e.g. logical data serparation, backup speedup, etc... Also you should not just thinking in a single huge database, but hundreds or thousands mid-to-large databases as well. That could also occupy huge amount of disk space. (cloud service) -Eredeti üzenet- Feladó: Dimitry Sibiryakov [mailto:s...@ibphoenix.com] Küldve: 2021. október 7., csütörtök 10:47 Címzett: For discussion among Firebird Developers Tárgy: Re: [Firebird-devel] Tablespaces proposal Roman Simakov wrote 06.10.2021 21:49: >> How are data moved? >> >> Under transaction control? >> >> In background (and interruptible) like db crypt? > it's a dfw operation with EX database lock. I hope to relax this > limitation later. So there are no concurrent changes. > 1) copy all data pages > 2) switch RDB$POINTER_PAGE and RDB$ROOT_PAGE transactionally > 3) Rebuild RDB$PAGES > 4) clear old data pages (as post-dfw operation) > > It can be interrupted but not resumed. I'm afraid it starts from the > beginning. > > I guess it's better to discuss in a context of PR. Tablespaces has meaning for large databases only that don't fit into single storage (terrabytes). For such volumes this algorithm is no-go. Better not to have move option in the initial implementation at all. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Dialect 3 inconsistent round/trunc - configurable calculation method needed
Ok, I understand it's not "wrong", works as designed/intended. Just put into documentation. Also, how do you expect to people to leave dialect 1 when dialect 3 gives no benefit, in both dialect extra effort needed to get calculations right, plain operation and funcation usage is not enought to get correct results: dialect 1 requires rounding everywhere because the everything is a float dialect 3 requires double precition cast and rounding in case of division because of low precision/scale during calculation A new dialect or a new option should do the job. -Eredeti üzenet- Feladó: Dimitry Sibiryakov [mailto:s...@ibphoenix.com] Küldve: 2021. szeptember 1., szerda 11:27 Címzett: For discussion among Firebird Developers Tárgy: Re: [Firebird-devel] Dialect 3 inconsistent round/trunc - configurable calculation method needed Molnár Attila wrote 01.09.2021 8:29: > The problem: why the storage format has affect on calculation. It just feels > wrong. It is ordinary thing in all programming languages when data types decide calculation rules. In the Delphi product of integer division used to be floating point, but that's Pascal exclusive rules. SQL (including Firebird) is based rather on C where division of two integers produce integer. > What is the point of dialect 3 when it does not spare work, it just replace > one boilerplate code writing to an other? If it doesn't work the way you expect it doesn't mean that it works wrong. Just accept that rules of arithmetic are different in different programming languages and perform calculations in the language which comply with your goals. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Dialect 3 inconsistent round/trunc - configurable calculation method needed
"I suggested you start a discussion about your wish that dialect 1 would not be dropped" Oh, sorry for the misunederstanding. After your comment on the GitHub issue I understand how the calcuation works. The problem: why the storage format has affect on calculation. It just feels wrong. I did not suggesed to throw away current method, I understand the concept of backward compability. I asked for a new configuration. In dialect 1 you have to round always because of the float format storage. In dialect 3 you have to cast to double always for not to loose precision. So both dialect you have to write unnecessary boilerplace code. If I want divide, it just not enough to write a divide. I want a mode, where calcuation do not require boilerplate round or cast: - numeric storage format is like dialect 3: exact value stored - calcuations are made in double precision float like dialect 1, OR made as in dialect 3, but with higher precision, and the rounding happens automatically when you assign/store a double value to a numeric. What is the point of dialect 3 when it does not spare work, it just replace one boilerplate code writing to an other? (The project is ~20 years old started with Delphi 5, which did not supported dialect 3, so dialect 1 was not a deliberate choice, but the only option.) Attila -Eredeti üzenet- Feladó: Mark Rotteveel [mailto:m...@lawinegevaar.nl] Küldve: 2021. augusztus 31., kedd 19:02 Címzett: firebird-devel@lists.sourceforge.net Tárgy: Re: [Firebird-devel] Dialect 3 inconsistent round/trunc - configurable calculation method needed On 31-08-2021 16:19, Molnár Attila wrote: > 127.13 / 3.4618 = 36,72366976717315 It isn't, taken literally in dialect 3, the result is 36,723669 ;). > EXECUTE BLOCK > RETURNS ( > c0 NUMERIC(15,10), > c1 NUMERIC(15,10), > c2 NUMERIC(15,6), > c3 NUMERIC(15,5), > c4 NUMERIC(15,4)) > AS > DECLARE VARIABLE a NUMERIC(15,2); > DECLARE VARIABLE b NUMERIC(15,4); > BEGIN > a = 127.13; > b = 3.4618; > c0 = a / CAST(b AS DOUBLE PRECISION);--36,7236697672, this is OK > c1 = a / b;--36,723669, this shoud be 36,723670 > c2 = a / b;--36,723669, this shoud be 36,723670 > c3 = a / b;--36,72367, this is OK > c4 = a / b;--36,7237, this is OK > SUSPEND; > END > > If return variable scale is bigger or equal than operation result scale, > then computed value is truncated, but when return variable scale is > smaller than operation result scale then computed value is rounded. > I think in all case value should be rounded, this behaviour is > inconsistent, and also truncation is unexpected behaviour. > > I opened a GitHub issue , but was closed : > https://github.com/FirebirdSQL/firebird/issues/6928 > <https://github.com/FirebirdSQL/firebird/issues/6928> > Mark suggested to start a conversation here. I didn't suggest you start a discussion about this behaviour, I suggested you start a discussion about your wish that dialect 1 would not be dropped ("If you want to discuss retention of dialect 1, then please take that to firebird-devel."). This has been the behaviour of NUMERIC/DECIMAL division in dialect 3 since InterBase 6.0. Changing that behaviour now would break or invalidate all applications that have been built with dialect 3 semantics of NUMERIC/DECIMAL division in mind. It complies with the SQL standard. Though to be honest, that is a pretty low bar, as the specification doesn't specify much for NUMERIC/DECIMAL division other than that precision and rounding is implementation specific ("iv) The precision and scale of the result of division are implementation-defined." - SQL:2016-2, 6.29 ) > #1 : This behaviour should be documented, I found nothing regarding > this, and this result is unexpected. Calculation do not happens in > double precision, and calculation stops when result scale is reached. The result is unexpected *to you* and does not match what you've been relying on in dialect 1. Yes, it is not explicitly documented other than the resulting precision, but neither is the behaviour of division in dialect 1 other than that it results in a DOUBLE PRECISION. You can think of NUMERIC/DECIMAL division in dialect 3 as a form of integer division like in most programming language. That also stops when the final digit has been calculated. For example, 14/3 = 4, not 5, 14.0/3 = 4.6, not 4.7, 14.0/3.0 = 4.66, not 4.67, etc, in dialect 1, all those calculations produce 4.667. For dialect 3, you can think of 14.0/3.0 as doing 14000/30 and then shifting the decimal by two positions (14000/30 = 466, shift decimal by 2 = 4.66). You haven't brought forth any real arguments why 21+ years of dialect 3 behaviour should be thrown out the window, other than that you don't like it because it isn't the same as in dialect 1. I understa
[Firebird-devel] Dialect 3 inconsistent round/trunc - configurable calculation method needed
Hi *! 127.13 / 3.4618 = 36,72366976717315 EXECUTE BLOCK RETURNS ( c0 NUMERIC(15,10), c1 NUMERIC(15,10), c2 NUMERIC(15,6), c3 NUMERIC(15,5), c4 NUMERIC(15,4)) AS DECLARE VARIABLE a NUMERIC(15,2); DECLARE VARIABLE b NUMERIC(15,4); BEGIN a = 127.13; b = 3.4618; c0 = a / CAST(b AS DOUBLE PRECISION);--36,7236697672, this is OK c1 = a / b;--36,723669, this shoud be 36,723670 c2 = a / b;--36,723669, this shoud be 36,723670 c3 = a / b;--36,72367, this is OK c4 = a / b;--36,7237, this is OK SUSPEND; END If return variable scale is bigger or equal than operation result scale, then computed value is truncated, but when return variable scale is smaller than operation result scale then computed value is rounded. I think in all case value should be rounded, this behaviour is inconsistent, and also truncation is unexpected behaviour. I opened a GitHub issue , but was closed : https://github.com/FirebirdSQL/firebird/issues/6928 Mark suggested to start a conversation here. #1 : This behaviour should be documented, I found nothing regarding this, and this result is unexpected. Calculation do not happens in double precision, and calculation stops when result scale is reached. #2 : Calculation method should be cofigurable #2/A: possible configuration modes - default: current mode - double precision: operands converted to double, calculations happens in double, then result rounded and converted back to result type - operand max: current calcuation method, but calcuation stops at max(op1.precision, op2.precision) instead of res.scale, then result rounded to res.scale - : current calcuation method, but calcuation stops at instead of res.scale, then result rounded res.scale #2/B: possible configuration places - some kind of SQL command (connection level), e.g.: SET CALCULATION PRECISION DEFAULT | DOUBLE PRECISION | OP_MAX | . Then in a database on connect trigger can set this, when needed - new dialect (don't prefer, this just affects calculation not storage) - firebird.conf (don't prefer, this should be linked to the database file) - connection string (don't prefer, this should be linked to the database file) [cid:image002.png@01D79E83.EB3C4920]<http://www.libra.hu/> CÉGÜNK A LIBRA CSOPORT TAGJA Molnár Attila fejlesztő LIBRA Szoftver Zrt. 1113 Budapest, Karolina út 65. +36 (1) 255-3939 i...@libra.hu<mailto:i...@libra.hu> | www.libra.hu<http://www.libra.hu> __ Information from ESET Mail Security, version of virus signature database 23884 (20210831) __ The message was checked by ESET Mail Security. http://www.eset.com Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ODP: Modern C++: constexpr
In my point of view (not a native english speaker) owner : ownership is a legal term basicly. It gives the owner rights over the owned property. master : master has actual power over the slave to command/control the slave. -Eredeti üzenet- Feladó: Dimitry Sibiryakov [mailto:s...@ibphoenix.com] Küldve: 2020. június 18., csütörtök 11:37 Címzett: For discussion among Firebird Developers Tárgy: Re: [Firebird-devel] ODP: Modern C++: constexpr 18.06.2020 10:05, Mark Rotteveel wrote: >> BTW, when you start your search-and-replace work, don't forget to >> find substitutions for "owner", "invalid", "dependency" and "creator". >> First and third offend ex-slaves, second - disabled people, last - >> religious ones. > > I'm sorry, but that is just plain silly and trying to trivialize the problem. Sorry, it is out of my English again. What is the difference between "master" and "owner"? -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel __ Information from ESET Mail Security, version of virus signature database 21513 (20200618) __ The message was checked by ESET Mail Security. http://www.eset.com Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] ICU File Deployment Guidance
You can posrt the difference between new ICU versoins in firebirdsql.org to help developers/users/admins to decide wheter an update nedded at all. -Eredeti üzenet- Feladó: Paul Reeves [mailto:pree...@mail.ibphoenix.com] Küldve: 2020. április 29., szerda 13:38 Címzett: firebird-devel@lists.sourceforge.net Tárgy: Re: [Firebird-devel] ICU File Deployment Guidance On Wed, 29 Apr 2020 11:59:43 +0100 Tony Whyman wrote: > Hopefully, an installer package will eventually be made available to > automate the process. However, the above manual procedure is all that is > currently available. I'm wondering how people think we could automate this process. Thoughts that immediately come to mind are: - We have no control over the release cycle of tzdata. The update to https://github.com/FirebirdSQL/firebird/tree/master/extern/icu/tzdata seems to be manual at the moment. - Looking at the update history over at https://github.com/unicode-org/icu-data/tree/master/tzdata/icunew there can be up to 10 changes per year. - The changes will, in all probability, more than 99% of the time, have no relevance for over 99% of our users. But can one take the risk of ignoring them? - Even if we can provide timely, automated updates how will a server installation know to look for them? A task scheduler of some sort would need to check. We don't have one built in to Firebird so it would mean adding something platform specific. Paul -- Paul Reeves http://www.ibphoenix.com Supporting users of Firebird Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel __ Information from ESET Mail Security, version of virus signature database 21244 (20200429) __ The message was checked by ESET Mail Security. http://www.eset.com __ Information from ESET Mail Security, version of virus signature database 21244 (20200429) __ The message was checked by ESET Mail Security. http://www.eset.com Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] interesting test fails with Firebird 3.0
Firebird UPDATE supports ORDER BY clause. Feladó: marius adrian popa [mailto:map...@gmail.com] Küldve: 2019. november 19., kedd 10:48 Címzett: For discussion among Firebird Developers Tárgy: [Firebird-devel] interesting test fails with Firebird 3.0 https://twitter.com/FranckPachot/status/1196415263031414786 update demo set n=n+1; violation of PRIMARY or UNIQUE KEY constraint "INTEG_2" on table "DEMO" Problematic key value is ("N" = 2) Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Inserts and FKs
BTW I also agree that this shoud throw FK voilation exception on detail insert. -Eredeti üzenet- Feladó: Carlos H. Cantu [mailto:lis...@warmboot.com.br] Küldve: 2019. szeptember 6., péntek 14:13 Címzett: For discussion among Firebird Developers Tárgy: Re: [Firebird-devel] Inserts and FKs MA> If you do master and detail in separate transaction then tx2 should MA> not be started before tx1, tx2 start MUST wait for tx1 SUCCESSFUL MA> commit, otherwise your business logis FLAWED. Tx2 wasn't start before tx1. Read again my first email. []s Carlos http://www.firebirdnews.org FireBase - http://www.FireBase.com.br Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel __ Information from ESET Mail Security, version of virus signature database 19976 (20190906) __ The message was checked by ESET Mail Security. http://www.eset.com __ Information from ESET Mail Security, version of virus signature database 19976 (20190906) __ The message was checked by ESET Mail Security. http://www.eset.com Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Inserts and FKs
Why do you insert master and detail record in separate transaction? If you do master and detail in separate transaction then tx2 should not be started before tx1, tx2 start MUST wait for tx1 SUCCESSFUL commit, otherwise your business logis FLAWED. This scenario is looks like "Dirty read" : https://en.wikipedia.org/wiki/Isolation_(database_systems)#Dirty_reads. You leak uncommied (for tx2) information. -Eredeti üzenet- Feladó: Carlos H. Cantu [mailto:lis...@warmboot.com.br] Küldve: 2019. szeptember 6., péntek 1:47 Címzett: firebird-devel@lists.sourceforge.net Tárgy: [Firebird-devel] Inserts and FKs Recently I had a real situation that drove me crazy for several days while trying to find what actually happened. After lots of thinking, checking the auditing logs and chatting with Vlad, the only scenario that could explain what happened is: TableA (master) TableB (detail) There is FK between TableB and TableA tx1 starts tx1 inserts master record in TableA tx2 starts (snapshot isolation) tx1 commits tx2 inserts a detail record in TableB Trigger on TableB tries to update master record, since the record is still not visible for this snapshot, update has null effect tx2 commits (no error) The detail insert does not fail with broken FK because FK checking is out of transaction control, but the update ran by the trigger had no effect because the master record was not visible for the update. >From the consistency point of view, in this example, if the trigger could not >see the record and modify it, the FK validation should fail too. IMHO, if the >transaction isolation doesn't allow the record to be seen (at trigger level) >it should not be seen at the FK validation level too. I understand that there are other scenarios where the currently FK behavior is correct and makes sense, for example, in the case of avoiding deleting a master record with "commited but not visible childs", but for the reported example, the currently behavior looks incorrect, and for people with business logic implemented in triggers, it may/will lead to incorrect results. Does anyone knows if this behavior is following the Standard? Any comments? []s Carlos http://www.firebirdnews.org Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel __ Information from ESET Mail Security, version of virus signature database 19974 (20190905) __ The message was checked by ESET Mail Security. http://www.eset.com __ Information from ESET Mail Security, version of virus signature database 19976 (20190906) __ The message was checked by ESET Mail Security. http://www.eset.com Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Database opened in another session
This might be a GC/sweep issue http://tracker.firebirdsql.org/browse/CORE-4751 http://tracker.firebirdsql.org/browse/CORE-4745 On 2017.12.07 16:35, Vlad Khorsun via Firebird-devel wrote: 07.12.2017 17:16, Jiří Činčura wrote: Nope. Still happening. Even with `-i`. Hmm... i was not attentive enough, error is about database not about listener. It means that one engine instance is already open database and another instance going to open it too and both instances runs in different windows sessions. For example, one instance run in user session (for ex. embedded connection) and second instance run as a service. Regards, Vlad -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] [SPAM] Re: Start transaction from base transaction
Awesome! :) On 2017.04.21. 1:30, Adriano dos Santos Fernandes wrote: > Em 20/04/2017 17:53, Vlad Khorsun escreveu: >> Also it is necessary >> to teach engine to use that metadata (instead of current one) within >> attachment >> working "in the past". >> > I'm doing a prototype implementation of this for active transactions, > i.e., the things I mentioned in this thread start. > > I've it starting working, but implementation is very simple, weak and > almost non-tested at the moment. > > > Adriano > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Start transaction from base transaction
+1 for this feature. I would be very happy for this. Also it would be awesmone if this consistent view were accessible later in time (this woudl mean garbage collection blocking). On 2017.04.18. 20:56, Leyne, Sean wrote: > >> If you need simultaneous queries - make them possible, >> what the point of transaction hacking? > You want a single "view" of the database from multiple _connections_. > > There is nothing that provides this, today -- There is no way to ensure that > all connections have that same view. > > > Sean > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] [FB-Tracker] Created: (CORE-5507) Wrong value of the new field at the old records, created before that new field was added.
Hi! I expect form the FB engine to return the default value at INSERT/UPDATE TIME, not the current default value. So default values should be kept in the rdb$format table, because versioning is needed. On 2017.03.24. 8:18, Dmitry Yemanov wrote: > 24.03.2017 09:33, Vlad Khorsun wrote: >>> Firebird is known to upgrade the record format while reading. "Upgrade" >>> here means using the latest (aka current) format. The current format is >>> the one that can be seen in RDB$RELATION_FIELDS. So one may expect that >>> the default value to be used is also the latest one, that's stored in >>> RDB$ tables. IIRC, this is how FB 2.5 works. >> Exactly. When i read the table and know that some field should have default >> value, i expect to see this value at system catalog. > It depends on the definition. In fact, adding a NOT NULL DEFAULT X > column means two things: (1) DEFAULT X will be used implicitly for > inserts or explicitly for the DEFAULT keyword and (2) X will be used to > replace the missing values. For (1), everything works as expected. For > (2), your expectation relies on the fact that X is substituted while > reading. But this is just an implementation detail. The contract is > "replacing missing values", whatever it could mean. Someone else does > not know such internals and expects this "replacement" happening by some > voodoo magic during ALTER ;-) > > Also, lets consider this: > > SQL> create table t (col1 int); > SQL> insert into t values (1); > SQL> commit; > SQL> alter table t add col2 int default 123 not null; > SQL> select * from t; > > COL1 COL2 > > 1 123 > > SQL> alter table t alter col2 drop default; > SQL> select * from t; > > COL1 COL2 > > 1 123 > > There's no default at the end, but COL2 is still returning 123. This > matches the "update" expectation, not the "convert missing to the > current default" one. > > Yesterday I considered this issue being a bug, but now I'm not that > sure. It may look obvious for some users but counter-intuitive for the > others. > > Anyone outside the development team cares to comment? :-) > > > Dmitry > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Virtual metadata
Don't forget about SAVEPOINT handling! It can make things very messy in this case (BUT expected to work as part of the transaction logic from my point of view). Lock should be applied in case of altering, parallel altering should be not allowed. (same as update a record, instant lock conflict error at command execution time, not commit time (even when different values are updated)) On 2017.01.18. 11:09, Dmitry Yemanov wrote: > 18.01.2017 12:38, Alex Peshkoff wrote: >> Currently with dfw we do have a lot of DDL errors raised at commit time >> i.e. it's not a regression. > True, but only because the actual work is performed during commit. If we > claim that DDL changes are applied immediately, but error is thrown at > commit, this looks weird. Especially if we find a way to allow mixed DDL > and DML - imagine ALTER TABLE and subsequent UPDATE both executing OK > but failing at commit because of the metadata conflict. > > > But don't forget that under > > normal circumstances such conflicts will be very rare. > > I would seriously question the need to allow concurrent DDL against the > same objects. This is simply not the way people work with the relational > databases. I'd rather lock the metadata being changed at the DDL time > and until commit. > > > Dmitry > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] FB2.5.6 - '' = ' ' is evaluated as true
Yes, this is correct. SQL specification defines this behaviour. On 2016.12.07. 14:24, Slavomir Skopalik wrote: > Hi, > > is it correct that empty string '' in comparison with one space string ' > ' is evaluated as true? > > SELECT * FROM rdb$database WHERE ''=' ' > > FB 2.5.6, database dialect 1 > > Slavek > > -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today.http://sdm.link/xeonphi Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] RFC: Timeouts
This is a GREAT idea! +1 And you might define the timeout in the CREATE/ALTER command (no need for config). On 2016.08.18. 12:04, liviuslivius wrote: > Kiling statement or transaction is not good as a general solution > It must be customized for situations. > > I suppose better feature will be "timeout messaging" - something like > TRIGGER ON STATEMENT_TIMEOUT > TRIGGER ON TRANSACTION_TIMEOUT > > and inside it we have access to MON$ tables and we can cancel statement, > transaction if we need. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] RFC: Timeouts
Hi! I think timeout should depend on these independent factors : - transaction parameters : RORC = false else true - first fetch : not possible at the timeout moment = true else false - average fetch time (start to measure after the first fetch) : very high (config) = true else fales and optionally (config) - plan : has NATURAL on some table (config) = true else false If at least one of the factor is true then cancel the statement, else let the statement live. On 2016.08.18. 9:37, liviuslivius wrote: > > W dniu 2016-08-18 09:26:22 użytkownik Vlad Khorsun> napisał: >> 18.08.2016 10:08, liviuslivius пишет: >>> Hi Vlad, >>> > I.e interactive Delphi application that fetch only really shown records > will get error > when user press "Down" key, If user fetch one record per hour - yes, such application should be better rewritten >>> Is this query in different state that can be distinguished from "running" >>> queries? I see that yes. >>> And should be possibility to exclude it from this feature or make 2 >>> different settings for that. >> Could you show good reasons to do it ? Real use case also welcome. >> >> Regards, >> Vlad >> >> -- >> Firebird-Devel mailing list, web interface at >> https://lists.sourceforge.net/lists/listinfo/firebird-devel >> > Query timeout is good for queries that consume CPU resources and not finished > in limited time. > But queries that feth e.g. 100 records for user and wait to fetch rest do not > consume much resources > and user can fetch (rest or next portion) e.g. after 10 minutes. > And that queries are not problem for DBA. Yes, such queries consume some > resources but not extensivly. > But feature like timeout is for queries that can utilize all server resources. > > regards, > Karol Bieniaszewski > > > > -- > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request & discussion for V4 (same as for V3)
On 2016.04.09. 20:25, Ann Harrison wrote: On Fri, Apr 8, 2016 at 5:54 AM, Molnár Attila <amol...@mve.hu> wrote: Optimizations - IS NOT NULL should use index. It is equivalent with >= min_value or <= max_value based on index direction Histograms and clustered indexes (if they're being considered) could help here to detect cases where IS NOT NULL returns a small subset of the records in a table. In general, searches that touch more than half the records in a table are more efficient when made in storage (natural) order rather than through an index. Remember that Firebird stored data and indexes separately, so setting up an indexed retrieval that will touch every page in a table is just overhead compared with straight-forwardly reading every page. I ment min_value and max_value not base on table data, but on column data type. - condition pre-evaluation and reduction. e.g.: WHERE 1 = 2 AND field = :param is always FALSE. Evaluation does not needed for all records, can decide at prepare time whether the result is an empty result set or an unfiltered result set. When InterBase was created, there was a lot of academic work on optimizing corner cases, with the result that academic databases tended to spend more time optimizing than retrieving. We made the deliberate choice not to spend optimizer time saving idiots from themselves. Thirty years later, maybe we'd choose differently. However, lots of programs depend on tricks like +0 and concatenating with an empty string to coerce unnatural but effective plans. I'd worry about the damage done to those cases. This optimization can only be done if constant conditions are in the SQL, so +0 like tricks would not be affected. - use index in "NATURAL" mode when column in a conditional appears in a multi column index, but not in the first place. You may reduce number of database page visits in this way : index page can hold more effective record data because it's narrower than the table data page record (also in worst case it could be worse than NATURAL because ot the mixed index and table data page read, but I think overall it could worth it, especially in big tables. measurements needed) I not sure what you mean by "NATURAL" index mode - "natural" usually means reading the data pages in storage order without any index. If you mean reading across the leaf level of the index to find matches in the second and subsequent keys in an index, you have no idea how hard that would be. Firebird index keys are mashed up values created so they compare bytewise in the desired order. When using an index, Firebird hasn't a clue where the boundaries fall between columns in multi-column index. It's just bytes. The format makes indexes dense and comparisons quick. Changing the key format to support partial matches on second and third columns seems like a bad idea, given that there's very little difference between having an index on each column and a multi-column index. Remember that Firebird uses multiple indexes on a single table. I ment the second case (reading across the leaf level). It's hard and costly (or even possible) to deconstruct the index format to get the original column values? The question is reading all leaf deconstruct and filter is faster than reading all records natural an filter. - SELECT DISTINCT FROM table is slow (natural scan on all records) and SELECT FROM table GROUP BY is
Re: [Firebird-devel] Feature request & discussion for V4 (same as for V3)
Hi Dmitry! Hope never dies. I prefer optimizations over the other new features. ;) Thanks for the reply. On 2016.04.08. 12:30, Dmitry Yemanov wrote: > 08.04.2016 12:54, Molnár Attila wrote: >> Here is my list. > List of v4 features is already composed. We may add some more > improvements here and there, but no promises about them. > > > Dmitry > > > -- > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Feature request & discussion for V4 (same as for V3)
Hi *! Here is my list. DOMAIN OF COLUMN . - should work like TYPE OF COLUMN, but CHECK and NOT NULL constraints are checked when value assigned to this variable - gain : performance (allows early check, error raised before the actual DML statement run) SIZE OF riable name>, SCALE OF - SIZE OF : returns max CHAR/VARCHAR length or NUMERIC precision, SCALE OF : return scale of NUMERIC - gain : by defining a variable with DOMAIN or TYPE OF (or DOMAIN OF) the declaration is dynamic. But the in the PSQL body there is no dynamic access to this information, you still need hard code these values. With this you can avoid hard coded constants. ARRAY TYPE (only in PSQL) - static and dynamic, one and multi dimensional - gain : performance (no need for table write storing temporary data), shorter and easy to understand PSQL code (UDF parameters with array types?) RECORD TYPE (only in PSQL) - gain : shorter and easy to understand PSQL code (UDF parameters with record types?) TRY-FINALLY - gain : code reduction (Now with AUTONOMUS TRANSACTION it make sense to have try-finally structure. Currently I have the same code in the successfull run path and also in the WHEN ANY path) TRIGGER : NEW/OLD values accessible by column name, and column number (PSQL) - e.g.: NEW['id'], NEW[0] - gain : code reduction, dynamic code (don't have to alter the trigger ICO the table structure altered) Optimizations - CORE-2589 - CORE-2589 - CORE-4266 (please rename the ticket : "NULLS FIRST/LAST should not cause index loss") - IS NOT NULL should use index. It is equivalent with >= min_value or <= max_value based on index direction - condition pre-evaluation and reduction. e.g.: WHERE 1 = 2 AND field = :param is always FALSE. Evaluation does not needed for all records, can decide at prepare time whether the result is an empty resultset or an unfiltered resultset. - use index in "NATURAL" mode when column in a conditional appears in a multi column index, but not in the first place. You may reduce number of database page visits in this way : index page can hold more effective record data beacouse it's narrower than the table data page record (also in worst case it could be worse than NATURAL beacose ot the mixed index and table data page read, but I think overall it could worth it, especially in big tables. measurements needed) - SELECT DISTINCT FROM table is slow (natural scan on all records) and SELECT FROM table GROUP BY is also slow (worse! : index scan on all records). I think in this case it's not neccessary to read all the records in the table, it should be enough to read #of distinct values from table. (currently you have to keep a separate table with this information because you can't access to this information fast) Thank You! -- Molnár Attila szoftverfejlesztő Libra Szoftver Zrt. 1113 Budapest, Karolina út 65. Tel.: +36 1 255 3939 Fax: +36 1 209 1477 http://www.libraszoftver.hu Tisztelt Ügyfelünk! Tájékoztatjuk, hogy 2015. október 14-től megújult a Libra Groupware, a Libra Szoftver Zrt. bejelentő rendszere. A rendszerhez, korábban már regisztrált ügyfeleink új jelszót a https://libragroupware.mve.hu/lgw/ link használatával kérhetnek. Új regisztráció a regisztra...@mve.hu címre írt levélben igényelhető. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Proposal of new feature: Event triggers
The goal is to send notification from one connection to other connection(s) without roundrip to client using current event mechanism. On 2015.02.26. 15:56, James Starkey wrote: Could you describe what you are actually trying to do rather than how a possible solution might work? It's a lot easier to work from an actual problem than to reverse engineer from a murky proposal. On Thursday, February 26, 2015, Carlos H. Cantu lis...@warmboot.com.br wrote: I would like to start a discussion to propose the implementation of "Event triggers" (for the lack of a better name for now). Those special triggers would be associated to an EVENT, not to a TABLE. When the associated event is fired in any of the active attachments, all the active attachments "interested" in that event would run the trigger. My personal example of use for such feature, would be to force the update of global context variables in all active attachments, when the EVENT is fired. Of course there are questions to be discussed, for example: the trigger would ran in what transaction context? Should it be forced to be ReadOnly (making things simpler, but much more limited)? This would be for FB 3.x or 4. If you guys thinks this is not the right moment to talk about it, I can just open a ticket and leave it in the tracker waiting for the right time. []s Carlos http://www.firebirdnews.org FireBase - http://www.FireBase.com.br -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Jim Starkey -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 3 memory manager
Hello all! Have a look at ScaleMM. It has great multithreaded performance, AND low fragmentation rate. https://code.google.com/p/scalemm/ You should consider rewrite it to C++. On 2014.09.20. 23:57, Jim Starkey wrote: Memory management is perhaps the single most performance part of a database system (at least once you figure out how never touch the disk). A significant part of the effort on each of my umpteen database systems has been spent squeezing cycles out of the memory manager, AVL trees for hunk management, minimizing fragmentation, and eliminating contention with thread-specific sub-pools. That said, the NuoDB guys look a fresh look at the problem and evaluated various other open source memory managers. One, jealloc, was only slightly slower than my best but did a significantly better job reducing fragmentation for a clear net gain. Nikolay's critique is sufficiently well taken that I saw no need to step into the fray. Still, if there's a better mouse trap with an acceptable license (jealloc is BSD), why no go for it? Without doubt, the Firebird memory allocator can be incrementally improved. But unless memory management is your life's work, if an acceptable open source memory manager can be shown to be significantly better than what's in place, adapt it and go on to the next problem. On 9/19/2014 6:33 PM, Nikolay Samofatov wrote: Hello, All! I implemented intermediate versions GC algorithm and tried to run some stress tests on Firebird 3 to check if I have broken something. The problem is that tests that I created were spending most of their time in memory manager. This is not healthy. For example, in my tests I saw up to 30 iterations in this loop during memory allocation: for (hunk = smallHunks; hunk; hunk = hunk-nextHunk) { if (length = hunk-spaceRemaining) { } } Dear Firebird engineers, why did you replace an algorithm which has O(log(n)) performance with an algorithm that has O(n) performance in such a performance-critical part of the engine? O(n) performance in certain scenarios was the reason why original memory manager of Interbase was replaced in Firebird 1.5. I created a small query to demonstrate the effect of O(n) memory manager performance. === create table test7 (id integer); execute block as declare variable i integer = 1; begin while(i = 5000) do begin insert into test7(id) values(:i); i = i + 1; end end; commit; savepoint X; update test7 set id = id; delete from test7; === Last statement uses 3.5 Gb of memory in small blocks. I forward-ported memory manager from Firebird 2.5 to Firebird 3 and compared performance of the last statement. With Firebird 3 memory manager the query runs for 634 s on my machine. With Firebird 2.5 memory manger it completes in 427 seconds. I used very small number of rows, so you can run it on a desktop class machine. It is easy to imagine a query that is a couple of orders of magnitude larger in ETL applications, and see that such queries now become impossible to run (due to at least an order of magnitude estimated slowdown). So the performance of the server is now O(N^2) for queries using large amounts of memory, and the server hits the wall rather quickly. From quick review of new memory manager code I can see the following problems: 1) O(n) performance in small hunk allocation 2) O(n) performance in large hunk deallocation 3) Worst case allocation cost in large hunk allocation algorithm is bounded, but much worse than for older memory manager 4) Lots of memory waste in various scenarios, which is a security issue with mild risk. 5) Not all debug facilities have been preserved. Problem 1 and 2 can be relatively inexpensively fixed with existing code base. Are there volunteers? Problems 3 and 4 require going back to global best-fit allocation strategy, and thus re-design. Alternatively, I may fix Firebird 2.5 memory manager to have better performance in simple cases without compromising worst-case performance. This is rather easy. B-Tree can be replaced with specially designed array of pointers (see FastMM code, for example). Dmitry, do you want me to do this? My understanding is that Firebird project probably does not care about large installs and security, so it will continue using new memory manager as it is marginally faster in simple cases. For Red Database version based on Firebird 3 we shall certainly back out this memory manager as it has unpredictable performance and introduces new security risks. Good thing is that changing or replacing memory manager is very simple task for existing code base. Best Regards, Nikolay Samofatov -- Slashdot TV. Video for Nerds. Stuff
[Firebird-devel] New Interface
Hi! Some note from an outsider view. #1 : both client plugin API MUST support C-tyle API (lowest common dominator) - standard data types, zero terminated strings, pointers/handles, records, stdcall/cdecl functions, NO objects/interfaces - most of the API developers will and can only use this. (if will be not supported most of the user base will be lost) #2 : old client API should not be changed at all - why the new API if old one is also developed? - backward compatibility #3 : client plugin OO API - only C++ developers can use is (small part of the whole user base pie) - not just the different Object memory layout is the problem, but the different memory managers also. Object can be shared via API just in case of same compiler with same memory optimization settings AND shared memory manager. - why maintain a second one API? Can gain siginficant performance over C style API? What are the cons? #4 : new API should be designed from scratch, in mind of : - eliminate/expand current limits - better multithread scalability - make place for future FB developements - make place for supporting missing SQL standard features -- Molnár Attila szoftverfejlesztő Tel : 372- E-mail: amol...@mve.hu LIBRA Szoftver zrt. 1113 Bp. Karolina út 65. Tel: 372- Fax: 209-1477 Web: www.mve.hu E-mail: i...@mve.hu Olvasson ügyfeleinkkel elért közös sikereinkről: http://www.mve.hu/hu/referenciaink -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
Lets see the next scenatios #1 : API call creates object and returned to host - in this case object has to be readonly (at least buffer like fields)1 - host has to support reference counting2 #2 : API call parameterised by a host created object - see #1 1 : can't write because in that case data contained by the object could be allocated by different memory managers 2 : Must have all mayor language/compilers support ref.count and has to be the same With this restrictions what gain do you have with OO/Interface API over C style API? On 2014.08.11. 10:02, Dimitry Sibiryakov wrote: It is not true as well. Memory manager is not a problem as long as object are freed in the same module where were created. Reference counting solves this well. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
BTW API and host has different (can't have) class information (class pointer, VMT, etc..) for the same object, so by definition they not communicate with the SAME class. The classes could have same name, and can act the same, but the equality operation would give you FALSE result. Also inheritance handling definiatily would be problem (different VMT, and different memory mapping at the two sides, etc...). So basicly pure Object based API is off the table. Interface based might be good, and we ended up with COM. On 2014.08.11. 10:49, Molnár Attila wrote: Lets see the next scenatios #1 : API call creates object and returned to host - in this case object has to be readonly (at least buffer like fields)1 - host has to support reference counting2 #2 : API call parameterised by a host created object - see #1 1 : can't write because in that case data contained by the object could be allocated by different memory managers 2 : Must have all mayor language/compilers support ref.count and has to be the same With this restrictions what gain do you have with OO/Interface API over C style API? On 2014.08.11. 10:02, Dimitry Sibiryakov wrote: It is not true as well. Memory manager is not a problem as long as object are freed in the same module where were created. Reference counting solves this well. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] New Interface
So this Interface is not designed and made for clinet component developers and communicatin with server but just for Firebird engine inside use? In that case just do what you want. ;) On 2014.08.11. 11:05, Dimitry Sibiryakov wrote: Remember, that new API wasn't designed as a public API. It was supposed to be a new interface between Y-valve and providers, nothing more. ISC API in this place would add too much overhead or require significant changes in engine code. -- Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] DDL Triggers, how to retrieve type?
E.g. our application still in dialect 1. It would be a huge job to switch dialect 3. On 2014.05.23. 8:41, Martijn Tonies (Upscene Productions) wrote: - Any other suggestion? Drop dialect 1 support. Allow dialect 1 to have access to BIGINT fields. I don't have to work for this to happen, so I don't really have a say... but the question that arises is why? What are the reasons to continue to use dialect 1? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Download Database Workbench for Oracle, MS SQL Server, Sybase SQL Anywhere, MySQL, InterBase, NexusDB and Firebird! -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free. http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free. http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
e.g. - Select value from two dynamic source, add them and write back. - a generic or a very dynamic code (EXECUTE STATEMENT) currenly have to store values in VARCHAR variables, because in other case (variable defined for every data type) the code would be VERY ugly. And because of this VARCHAR conversations must be used heavly. With VARIANT the code become cleaner and could avoid conversations. new context variable access method in PSQL with variant support would be great also On 2014.05.10. 16:43, Adriano dos Santos Fernandes wrote: On 10-05-2014 04:03, Molnár Attila wrote: *VARIANT data type in PSQL* This would be very easy to implement (variables only). What would be the use cases? Adriano -- Is your legacy SCM system holding you back? Join Perforce May 7 to find out: #149; 3 signs your SCM is hindering your productivity #149; Requirements for releasing software faster #149; Expert tips and advice for migrating your SCM now http://p.sf.net/sfu/perforce Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free. http://p.sf.net/sfu/SauceLabs Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
On 2014.05.10. 9:52, Dimitry Sibiryakov wrote: 10.05.2014 9:03, Molnár Attila wrote: *VARIANT data type in PSQL* - gain : a little memory and/or CPU overhead but much cleaner code. Also rdb$get/set_context value colud be variant. Welcome to the hell of unpredictable type conversion. *Optimization II. *- temporal indexing of materialization : e.g. when ORDER/GROUP BY has no index then currently the whole resultset is materialized, and the sorting moves the whole row each time. Instead of this it should create a temporal index on the order/group columns then fetching on the temporal index. In this way much less writes needed. This shold be applied after a treshold : common sense sais after index size/row size rate is smaller than 0.5. Uncommon sense says that random IO can be much more than two times slower. OK, this just means smaller treshold rate. But with this some case you can gain HUGE performance boost (especially when the resultset can't fit into memory and have to write to disk) -- Is your legacy SCM system holding you back? Join Perforce May 7 to find out: #149; 3 signs your SCM is hindering your productivity #149; Requirements for releasing software faster #149; Expert tips and advice for migrating your SCM now http://p.sf.net/sfu/perforce Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
Derived table usage for this problem is GREAT idea! Why the ... I didn't tought that! Thank you! On 2014.05.10. 10:26, Dimitry Sibiryakov wrote: 10.05.2014 10:12, Molnár Attila wrote: OK, this just means smaller treshold rate. But with this some case you can gain HUGE performance boost (especially when the resultset can't fit into memory and have to write to disk) Big result set is a sign of wrong application design. And in any case you can use derived tables to gain the same effect right now without hacking the engine. -- Is your legacy SCM system holding you back? Join Perforce May 7 to find out: #149; 3 signs your SCM is hindering your productivity #149; Requirements for releasing software faster #149; Expert tips and advice for migrating your SCM now http://p.sf.net/sfu/perforce Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
Hello Sean! On 2014.05.01. 21:24, Leyne, Sean wrote: TRIGGER : NEW/OLD values accessible by column name, and column number (PSQL) - e.g.: NEW['id'], NEW[0] - gain : code reduction, dynamic code (don't have to alter the trigger ICO the table structure altered) 1- What advantage does NEW['id'] have over the current syntax new.id??? I could iterate trough columns/values, currently I can't. 2- column numbers reference is extremely dangerous, the column position can be changed using simple DDL. So, all Trigger references would be invalidated! I do not want hardcode index numbers it's about iteration. But you're right, support only access by name is quite enough. Optimization I. - VOLATILE (default) / DETERMINISTIC flag for UDF and FUNCTION (maybe lifecycle for deterministic : statement, transaction, connection) - gain : preformance (can cache DETERMINISTIC UDF/FUNCTION results) Actually, I think that with the exception of a random number generator, the default for UDF should be DETERMINISTIC (Over 99% of UDFs are not volatile). User shold define deterministic flag. Only slow UDF/FUNCTION shold be flagged this or else the cache size and lookup time would increase too big. - SELECT DISTINCT index fields FROM table is slow (natural scan on all records) and SELECT index fields FROM table GROUP BY index fields is also slow (worse! : index scan on all records). - currently you have to keep a separate table with this information because you can't access to this information fast - solution 1 : use index in this case; Using an index may not help: 1- an index is stored without regard to physical disk location, so using it will create a huge amount of random disk IO. Whereas a NATURAL scan follows the table. 2- In an version based database like Firebird each row will need to be read to confirm the current value of the target field. Index usage optimization II. IS NOT NULL should use index. It is equivalent with = min_value or = max_value based on index direction I don't think this is possible. With an MVCC it is possible for all rows to have both a NULL and NOT NULL values stored in the field index, so reading each rows is required. But as noted in #1 above, reading by index can lead to significant disk IO/degradation I don't understand this. An equivalent statement can use index and produce the same result and this why can't? What I'm missing? Index usage optimization III. - condition pre-evaluation and reduction. e.g.: WHERE 1 = 2 AND field = :param is always FALSE. Evaluation does not needed for all records, can decide at prepare time whether the result is an empty resultset or an unfiltered resultset. I agree that the engine needs to add support for 'execution time' optimization. To allow for conditions to be bypassed (as your example) as well as eliminate unneeded JOIN in SELECT and Views. PSQL exception handling : new variables aside GDSCODE and SQLCODE - ERRORMESSAGE : form fbclient - GDSCODE_ORI, SQLCODE_ORI, ERRORMESSAGE_ORI : execute statement errors on external database hides the original error code, you should access them with these - gain : better logging and error tracking and user feedback User feedback? Based on error code we have our extended error messages and suggestions to deal with it, but in this case we can't properly do this. rename objects it's a nightmare to rename soething with many-many dependencies Save blob as file to disk at server side - gain : easy UDF deploy and update (and many other thing) - it's also a seurity risk, so some options needed in firebird.conf This would violate data integrity requirements. If I save data to a Blob, I expect the data to be available -- saving externally would allow for external data to be deleted without the database/me knowing. I believe that there are UDFs available for this already, no? A very old and unsuppoted udf. In my case I do not expect to read back from file to DB just one way. select from execute block - gain : avoid stored procedure creation. With select from select it's easy to generate code, but can't do that with execute block. (select from procedure is allowed, but execute block can't replace the procedure in this case) EXECUTE BLOCK already supports returning data, what are you looking for? Once I was needed the following logic (via code generation result). I bypassed it using stored proc. SELECT FROM (EXECUTE BLOCK AS BEGIN SELECT FROM ( EXECUTE BLOCK AS BEGIN END) ) -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
Hello Dmitry! On 2014.05.02. 8:34, Dmitry Yemanov wrote: 30.04.2014 13:50, Molnár Attila wrote: *SIZE OF CHAR/VARCHAR domain or variable name, SCALE OF NUMERIC domain or variable name* - SIZE OF : returns max CHAR/VARCHAR length or NUMERIC precision, SCALE OF : return scale of NUMERIC - gain : by defining a variable with DOMAIN or TYPE OF (or DOMAIN OF) the declaration is dynamic. But the in the PSQL body there is no dynamic access to this information, you still need hard code these values. With this you can avoid hard coded constants. Why wouldn't you want to create domains and use TYPE OF / DOMAIN OF? I'm createing domains, and using TYPE OF. But as I write it's not enough. This is just for variable declaration but I need a pair in PSQL body. Maybe this example would help to understand. EXECUTE BLOCK AS DECLARE VARIABLE tmp TYPE OF COLUMN rdb$database.rdb$character_set_name; DECLARE VARIABLE i INTEGER; BEGIN i = 1; WHILE (i = 31)--I want avoid hard coded 31 with SIZE OF tmp OR SIZE OF COLUMN rdb$database.rdb$character_set_name DO BEGIN --some logic here i = i + 1; END END *increase or remove completly the context limit* (not the context variable but the table/view/sp stream, current limit is 255) - gain : performance (bigger EXECUTE BLOCK with more DML, e.g.: a lot of INSERT in a single SQL command) Multi-value (multi-row) INSERT would serve the same goal without affecting the context limit. Batch DML API would also allow remove the need for EXECUTE BLOCK to execute batch INSERTs. Why else would you need a bigger EXECUTE BLOCK? I won't if these two are available. I wasn't aware of them. *TRIGGER : NEW/OLD values accessible by column name, and column number (PSQL)* - e.g.: NEW['id'], NEW[0] - gain : code reduction, dynamic code (don't have to alter the trigger ICO the table structure altered) Dynamic code is not going to work reliably, unless you just iterate fields in the loop. In this case, I see it just as a feature misuse, some other solution may be required instead. It's about iteration. Define misuse. *Optimization**I.* - VOLATILE (default) / DETERMINISTIC flag for UDF and FUNCTION (maybe lifecycle for deterministic : statement, transaction, connection) - gain : preformance (can cache DETERMINISTIC UDF/FUNCTION results) Partically done in FB3, to be improved later. *Index usage optimization III.* - condition pre-evaluation and reduction. e.g.: WHERE 1 = 2 AND field = :param is always FALSE. Evaluation does not needed for all records, can decide at prepare time whether the result is an empty resultset or an unfiltered resultset. http://tracker.firebirdsql.org/browse/CORE-1287 *PSQL exception handling : new variables aside GDSCODE and SQLCODE* - ERRORMESSAGE : form fbclient Also exists in the tracker. - GDSCODE_ORI, SQLCODE_ORI, ERRORMESSAGE_ORI : execute statement errors on external database hides the original error code, you should access them with these Maybe an ability to iterate all status vection arguments would be a better (and more generic) solution? Status vectos is at API level if I'n not mistaken. What about PSQL level? *performance info* - ms is not so accurate and it seems also affecte by the thread time slice size. Need a more accurate measure : maybe gettickcount? What performance info do you mean here? CPU time reported by ISQL? anything is a good candidate which has better scaleing than the curreny one. *Foreign key column or referenced column can be a const value* - e.g. : ALTER TABLE table ADD CONSTRAINT FKNAME FOREIGN KEY (field1, field2) REFERENCES referenced_table ('CONST', field3); - gain : data integrity without useless data storage And if you change master.field1 to become 'NONCONST' in some record your database gets logically corrupted due to constraint violation? Also, the prefix compression nearly completely removes your 'CONST' from the FK index anyway, so there's no useless data storage. *select from execute block* - gain : avoid stored procedure creation. With select from select it's easy to generate code, but can't do that with execute block. (select from procedure is allowed, but execute block can't replace the procedure in this case) The only syntax I could live with (still not being happy at all!) is using CTE, something like: WITH EXECUTE BLOCK ... AS ... SELECT ... IMHO, it's terribly ugly, but other options are even uglier. I could live with that kind of ugly. This is not priority for me. Dmitry -- Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple to use. Nothing to install. Get started now for free. http://p.sf.net
Re: [Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
On 2014.05.05. 9:27, Dmitry Yemanov wrote: 05.05.2014 11:05, Molnár Attila wrote: I'm createing domains, and using TYPE OF. But as I write it's not enough. This is just for variable declaration but I need a pair in PSQL body. Maybe this example would help to understand. EXECUTE BLOCK AS DECLARE VARIABLE tmp TYPE OF COLUMN rdb$database.rdb$character_set_name; DECLARE VARIABLE i INTEGER; BEGIN i = 1; WHILE (i = 31)--I want avoid hard coded 31 with SIZE OF tmp OR SIZE OF COLUMN rdb$database.rdb$character_set_name DO BEGIN --some logic here i = i + 1; END END Now I see your point, thanks. *increase or remove completly the context limit* (not the context variable but the table/view/sp stream, current limit is 255) - gain : performance (bigger EXECUTE BLOCK with more DML, e.g.: a lot of INSERT in a single SQL command) Multi-value (multi-row) INSERT would serve the same goal without affecting the context limit. Batch DML API would also allow remove the need for EXECUTE BLOCK to execute batch INSERTs. Why else would you need a bigger EXECUTE BLOCK? I won't if these two are available. I wasn't aware of them. They are not available, but they're the proper solution to the problem and should be considered instead. *TRIGGER : NEW/OLD values accessible by column name, and column number (PSQL)* - e.g.: NEW['id'], NEW[0] - gain : code reduction, dynamic code (don't have to alter the trigger ICO the table structure altered) Dynamic code is not going to work reliably, unless you just iterate fields in the loop. In this case, I see it just as a feature misuse, some other solution may be required instead. It's about iteration. Define misuse. Iterator and ordinal number are two different things, even if they can be implemented the same way. I don't want ordinal numbers for iterating *only*, as people will start using them for other purposes finally shooting themselves in the foot. Access by name only is good for me. (BTW I just realised that in this case it would great to have a VARIANT data type) - GDSCODE_ORI, SQLCODE_ORI, ERRORMESSAGE_ORI : execute statement errors on external database hides the original error code, you should access them with these Maybe an ability to iterate all status vection arguments would be a better (and more generic) solution? Status vectos is at API level if I'n not mistaken. What about PSQL level? This is exactly my point - PSQL access to the status vector. E.g. besides GDSCODE, make available also GDSCODE[n] or whatever iteration solution we choose. *performance info* - ms is not so accurate and it seems also affecte by the thread time slice size. Need a more accurate measure : maybe gettickcount? What performance info do you mean here? CPU time reported by ISQL? anything is a good candidate which has better scaleing than the curreny one. My question was what bad one / current one do you mean? AFAIR, the only one we have now is what ISQL reports (via perf API). Is this what you're talking about? Very fast stored procedures run time can't be measuerd standalone, have to build an EXECUTE BLOCK and repeate many times then divide the run time to get average run time. But in this case there are also side effects (like HW cache, branch prediction, server side statement cache, data cache, etc...) and overhead which influences the measurement. Currenty on my XP it reports 0ms or 16ms runtime, but nothing in between. (this number suggest me the thread time slice size percision of the current one) Dmitry -- Is your legacy SCM system holding you back? Join Perforce May 7 to find out: #149; 3 signs your SCM is hindering your productivity #149; Requirements for releasing software faster #149; Expert tips and advice for migrating your SCM now http://p.sf.net/sfu/perforce Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- Is your legacy SCM system holding you back? Join Perforce May 7 to find out: #149; 3 signs your SCM is hindering your productivity #149; Requirements for releasing software faster #149; Expert tips and advice for migrating your SCM now http://p.sf.net/sfu/perforce Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Feature request discussion (Reply to Planning the post v3 development)
Hello Dimitry! Here is my wish list. DOMAIN OF COLUMN table or view.column - should work like TYPE OF COLUMN, but CHECK and NOT NULL constraints are checked when value assigned to this variable - gain : performance (allows early check, error raised before the actual DML statement run) SIZE OF CHAR/VARCHAR domain or variable name, SCALE OF NUMERIC domain or variable name - SIZE OF : returns max CHAR/VARCHAR length or NUMERIC precision, SCALE OF : return scale of NUMERIC - gain : by defining a variable with DOMAIN or TYPE OF (or DOMAIN OF) the declaration is dynamic. But the in the PSQL body there is no dynamic access to this information, you still need hard code these values. With this you can avoid hard coded constants. ARRAY TYPE (only in PSQL) - static and dynamic, one and multi dimensional - gain : performance (no need for table write storing temporary data), shorter and easy to understand PSQL code (UDF parameters with array types?) RECORD TYPE (only in PSQL) - gain : shorter and easy to understand PSQL code (UDF parameters with record types?) increase or remove completly the context limit (not the context variable but the table/view/sp stream, current limit is 255) - gain : performance (bigger EXECUTE BLOCK with more DML, e.g.: a lot of INSERT in a single SQL command) TRY-FINALLY - gain : code reduction (Now with AUTONOMUS TRANSACTION it make sense to have try-finally structure. Currently I have the same code in the successfull run path and also in the WHEN ANY path) TRIGGER : NEW/OLD values accessible by column name, and column number (PSQL) - e.g.: NEW['id'], NEW[0] - gain : code reduction, dynamic code (don't have to alter the trigger ICO the table structure altered) Optimization I. - VOLATILE (default) / DETERMINISTIC flag for UDF and FUNCTION (maybe lifecycle for deterministic : statement, transaction, connection) - gain : preformance (can cache DETERMINISTIC UDF/FUNCTION results) Optimization II. - SELECT DISTINCT index fields FROM table is slow (natural scan on all records) and SELECT index fields FROM table GROUP BY index fields is also slow (worse! : index scan on all records). - currently you have to keep a separate table with this information because you can't access to this information fast - solution 1 : use index in this case; solution 2 : a system table with this information; soultion 3 : select from index (not so SQL standard) - gain : redundancy reduction and/or speed Index usage optimization I. CORE-2589 Index usage optimization II. IS NOT NULL should use index. It is equivalent with = min_value or = max_value based on index direction Index usage optimization III. - condition pre-evaluation and reduction. e.g.: WHERE 1 = 2 AND field = :param is always FALSE. Evaluation does not needed for all records, can decide at prepare time whether the result is an empty resultset or an unfiltered resultset. More details for exception 335544321 - Arithmetic exception, numeric overflow, or string truncation : table fieldname (procedure variable name) in exception message PSQL exception handling : new variables aside GDSCODE and SQLCODE - ERRORMESSAGE : form fbclient - GDSCODE_ORI, SQLCODE_ORI, ERRORMESSAGE_ORI : execute statement errors on external database hides the original error code, you should access them with these - gain : better logging and error tracking and user feedback longer object names CORE-749 rename objects it's a nightmare to rename soething with many-many dependencies performance info - ms is not so accurate and it seems also affecte by the thread time slice size. Need a more accurate measure : maybe gettickcount? Foreign key column or referenced column can be a const value - e.g. : ALTER TABLE table ADD CONSTRAINT FKNAME FOREIGN KEY (field1, field2) REFERENCES referenced_table ('CONST', field3); - gain : data integrity without useless data storage Save blob as file to disk at server side - gain : easy UDF deploy and update (and many other thing) - it's also a seurity risk, so some options needed in firebird.conf select from execute block - gain : avoid stored procedure creation. With select from select it's easy to generate code, but can't do that with execute block. (select from procedure is allowed, but execute block can't replace the procedure in this case) Thank You! -- Molnár Attila szoftverfejlesztő Tel : 372- E