Re: [Firebird-devel] Start transaction from base transaction
On 19/02/2019 13:56, Vlad Khorsun wrote: > > As i understand you - it doen't break anything. For me, it is safe to > assign > to the new snapshot (CONCURRENCY) transaction number of already existing > and > still alive snapshot. It is very important to ensure existence of that > snapshot > number until assignment happens (and new snapshot slot allocated). > Thank you. I created draft pull request for review and discussion. https://github.com/FirebirdSQL/firebird/pull/193 Adriano Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6008) isql query fails to return correct result
isql query fails to return correct result - Key: CORE-6008 URL: http://tracker.firebirdsql.org/browse/CORE-6008 Project: Firebird Core Issue Type: Bug Environment: ISQL Version: LI-V2.5.7.27050 Firebird 2.5 Reporter: Ray Holme Priority: Minor This was originally a delete query, but select shows the problem well - I have a copy of both the query and a minimal backed up DB that I can provide. The very first query returns erroneous results, the following queries prove the error; Database: working.db, User: sysdba SQL> select count(*) from addresses CON> where addr_id not in (select addr_id from office_defaults) CON> and addr_id not in (select addr_id from people) CON> and addr_id not in (select addr_id from people_more_addrs) CON> and addr_id not in (select addr_id from patients); 0 SQL> select count(*) from addresses where addr_id in (1596, 5341, 3331, 3534); 4 SQL> select count(*) from office_defaults where addr_id in (1596, 5341, 3331, 3534); 0 SQL> select count(*) from people where addr_id in (1596, 5341, 3331, 3534); 0 SQL> select count(*) from people_more_addrs where addr_id in (1596, 5341, 3331, 3534) 0 SQL> select count(*) from patients where addr_id in (1596, 5341, 3331, 3534); 0 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
Hi, Dne 21. 02. 19 v 11:15 Dimitry Sibiryakov napsal(a): 21.02.2019 10:46, Pavel Cisar wrote: It's tempting, but I see potential for problems. If we would allow multiple sets & filters at master node, there is no need to have them at replica. And if replica would have different definitions than master, then it's not possible to replace master with replica in recovery scenario. Yes, if you limit usage of replica to standby/backup. But imagine kind of sharding when some tables are replicated in one database and others - to different one. It's not about the use case (sharding etc.) but about distribution of filters. We could have them either at master or at replica. If they would be at master, there is IMHO no much point to have them also at replica. Having master sets copied over to replica would allow easy standby solutions that could be harder to have if these would not be the same at both sides. Supporting filters at replica is IMHO only interim solution for sharding while multiple sets and filters are not supported at master. So I'd rather have all such multiple sets and filters at master (copied over to replicas) where they could be more easily managed than having different ones scattered all around. That way we could have both, sharding and standby in one shot and much trouble. regards Pavel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] [FB-Tracker] Created: (CORE-6007) Firebird does not build on Mac with fresh toolchain
Firebird does not build on Mac with fresh toolchain --- Key: CORE-6007 URL: http://tracker.firebirdsql.org/browse/CORE-6007 Project: Firebird Core Issue Type: Bug Affects Versions: 4.0 Beta 1, 3.0.4, 2.5.8 Environment: Mac OS Reporter: Alexander Peshkov Compilation fails when building extern editiline library -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://tracker.firebirdsql.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
21.02.2019 12:46, Pavel Cisar wrote: And one partially related question from another angle: does it make sense to implement also replica-side declarative filtering? I mean the case where changes for all tables are journaled but for some reason only some tables should be applied to replica - e.g. two independent replicas with different filters but replicated from the same master journal (to avoid double journaling). If this feature is desirable, then how should the master-side filter (replication set) co-exist with the replica-side filter? It's tempting, but I see potential for problems. If we would allow multiple sets & filters at master node, there is no need to have them at replica. Not really so. If replica 1 needs changes for table A and replica 2 needs changes for table B, then master should include both A and B into its filter. So all three filters are different. And if replica would have different definitions than master, then it's not possible to replace master with replica in recovery scenario. To deal with that, master-side and replica-side filters must have different types, so that replica-side filters never be applied for a database in the master role. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
21.02.2019 10:46, Pavel Cisar wrote: It's tempting, but I see potential for problems. If we would allow multiple sets & filters at master node, there is no need to have them at replica. And if replica would have different definitions than master, then it's not possible to replace master with replica in recovery scenario. Yes, if you limit usage of replica to standby/backup. But imagine kind of sharding when some tables are replicated in one database and others - to different one. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
21.02.2019 9:14, Dmitry Yemanov wrote: Second, IMHO declaring tables as "publishable" via CREATE|ALTER TABLE is too restrictive. I'd rather manage the replication set using some global commands, be it ALTER DATABASE or something different, allowing to include/exclude all tables at once, or comma-separated list of tables, or maybe tables by mask (regexp?). Of course, both SQL solutions (database level and table level) may co-exist. It is not too hard to do FOR SELECT DO EXECUTE STATEMENT for group operation. And one partially related question from another angle: does it make sense to implement also replica-side declarative filtering? I mean the case where changes for all tables are journaled but for some reason only some tables should be applied to replica - e.g. two independent replicas with different filters but replicated from the same master journal (to avoid double journaling). If this feature is desirable, then how should the master-side filter (replication set) co-exist with the replica-side filter? This is where filters in config are useful, IMHO. CDC plugin is called for what is configured in database and apply process apply then apply only what is configured in its config files. I assume that both are plugins, not internal. -- WBR, SD. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
On 2/21/19 12:46 PM, Pavel Cisar wrote: Second, IMHO declaring tables as "publishable" via CREATE|ALTER TABLE is too restrictive. I'd rather manage the replication set using some global commands, be it ALTER DATABASE or something different, allowing to include/exclude all tables at once, or comma-separated list of tables, or maybe tables by mask (regexp?). Of course, both SQL solutions (database level and table level) may co-exist. Agreed. However, would be database and table level definitions independent, or would we translate database level to batch of table ones? First would allow independent revocation but would complicate processing. That's probably one of the most important problems here. Should replication set given at database level to SAM% tables include SAMPLE7 table created after replication setup? As far as I understand with current configuration-file based approach such table will be included. And if we want to keep curretn behavior we can hardly translate DB level to batch of tables. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Replication: declarative control
Hi, Generally, I like the DDL approach more than using the configuration file. Additional benefit to ones listed is possibility to create various front-ends for replication configuration / integration of such functionality into existing Firebird management tools. Comments to specific question inline... Dne 21. 02. 19 v 9:14 Dmitry Yemanov napsal(a): One thing I'm worried about is whether it's enough to have a single global replication set or maybe it's useful to have many independent replication sets. How they can be used, for example: It would be nice to has such feature in future. 1) Two slightly different global replications sets are defined, only one of them is active at a time, but we can switch between them (e.g. via enable/disable commands) Hmm, can see an use case for it that would solve some problem and not cause trouble at the same time. I sense a great potential for users to shoot themselves to foot. 2) Different tables (separated by some rule) are included into different replication sets which are all active together, their intersection is used by the CDC publisher. This may be useful if these replication sets has some declarative customizations (see below). This would be certainly valuable. 3) Different replication sets are declared as intended for different CDC plugins. This implies that multiple CDC plugins may be configured independently. In this cases the CDC publisher checks the source (table) against the target (plugin) before sending the changes. This is certainly interesting. I can imagine a business case for a CDC plugin that does data stream processing instead persistence to some kind of replica. Second, IMHO declaring tables as "publishable" via CREATE|ALTER TABLE is too restrictive. I'd rather manage the replication set using some global commands, be it ALTER DATABASE or something different, allowing to include/exclude all tables at once, or comma-separated list of tables, or maybe tables by mask (regexp?). Of course, both SQL solutions (database level and table level) may co-exist. Agreed. However, would be database and table level definitions independent, or would we translate database level to batch of table ones? First would allow independent revocation but would complicate processing. Finally, if we consider the replication set being a filter, it may be also useful to limit the published change set to some particular operations (INSERT|UPDATE|DELETE) or even some particular rows (WHERE filter). I doubt this is useful for replication per se, but this may allow something similar to "change views" in InterBase, currently with a CDC plugin acting as a client, but perhaps it could be extended later to interact with the real client application. This would be certainly interesting. It would allow data sharding and all sorts of various interesting "utilizations" of replication plumbing for other things than just replication itself. And one partially related question from another angle: does it make sense to implement also replica-side declarative filtering? I mean the case where changes for all tables are journaled but for some reason only some tables should be applied to replica - e.g. two independent replicas with different filters but replicated from the same master journal (to avoid double journaling). If this feature is desirable, then how should the master-side filter (replication set) co-exist with the replica-side filter? It's tempting, but I see potential for problems. If we would allow multiple sets & filters at master node, there is no need to have them at replica. And if replica would have different definitions than master, then it's not possible to replace master with replica in recovery scenario. It could be an interim solution for absence of multiple sets & filters on master, but it would be hard to deprecate them once we implement such master solution. regards Pavel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] CORE-6005
21.02.2019 10:39, liviuslivius wrote: http://tracker.firebirdsql.org/browse/CORE-6005 I do not talk about problem itself but think about new feature. Is this stable cursor a real cursor on which we can work and we can have some system name for it? No, it's just low-level visibility rules - what record versions can be seen by the statement. Dmitry Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Replication: declarative control
All, In v4 Beta, replication is fully driven by the configuration file. In particular, tables to be replicated are (optionally) defined using two regexp-based options: include_filter and exclude_filter. This was the easiest solution that doesn't require ODS changes and this matches the trace/audit configuration thus being famous to Firebird DBAs. However, I don't think this is flexible enough. IMO, there's much sense in separating "what is replicated" from "how it's replicated". The former is a part of the CDC (custom data capture) interface and defines what changes we need to collect. The latter is a set of implementation details belonging to either the built-in replicator engine or 3rd party CDC plugin - caching rules, transport options, etc. Such a separation would allow to build a really flexible architecture. Moving this idea further, it's worth making the "what is replicated" part controlled declaratively, using SQL. I.e. DBA defines some "replication set" by including or excluding tables, this set is stored inside the database and used by the CDC publisher to filter out unnecessary changes before passing them to the CDC handler. These include/exclude rules are also replicated. I see the following (at least) benefits in this approach: 1) Allow custom CDC solutions without interacting with the built-in replication configuration 2) Automatic setup for cascaded replication - every replica knows what tables are allowed for replication and reuse these rules without any explicit include/exclude settings 3) We may allow modification of non-replicated tables in read-only replicas - this will not cause replication conflicts 4) With some additional efforts, we could allow the replication set to be changed at runtime, without restarting the server (it could be bad practice in general, but perhaps useful to quickly fix some configuration mistake) Dimitry Sibiryakov has kindly provided a pull request implementing this feature using the CREATE|ALTER TABLE extensions. It uses RDB$FLAGS for storage and thus doesn't require ODS changes. I suppose this can be accepted as a straightforward solution for v4. However, it may be somewhat limited in the long term. So I'd like to have it discussed before accepting the PR. One thing I'm worried about is whether it's enough to have a single global replication set or maybe it's useful to have many independent replication sets. How they can be used, for example: 1) Two slightly different global replications sets are defined, only one of them is active at a time, but we can switch between them (e.g. via enable/disable commands) 2) Different tables (separated by some rule) are included into different replication sets which are all active together, their intersection is used by the CDC publisher. This may be useful if these replication sets has some declarative customizations (see below). 3) Different replication sets are declared as intended for different CDC plugins. This implies that multiple CDC plugins may be configured independently. In this cases the CDC publisher checks the source (table) against the target (plugin) before sending the changes. These cases are purely theoretical, but I believe we should consider them and decide whether it's worth to be prepared for them or not. Second, IMHO declaring tables as "publishable" via CREATE|ALTER TABLE is too restrictive. I'd rather manage the replication set using some global commands, be it ALTER DATABASE or something different, allowing to include/exclude all tables at once, or comma-separated list of tables, or maybe tables by mask (regexp?). Of course, both SQL solutions (database level and table level) may co-exist. Finally, if we consider the replication set being a filter, it may be also useful to limit the published change set to some particular operations (INSERT|UPDATE|DELETE) or even some particular rows (WHERE filter). I doubt this is useful for replication per se, but this may allow something similar to "change views" in InterBase, currently with a CDC plugin acting as a client, but perhaps it could be extended later to interact with the real client application. And one partially related question from another angle: does it make sense to implement also replica-side declarative filtering? I mean the case where changes for all tables are journaled but for some reason only some tables should be applied to replica - e.g. two independent replicas with different filters but replicated from the same master journal (to avoid double journaling). If this feature is desirable, then how should the master-side filter (replication set) co-exist with the replica-side filter? Please provide your feedback on these questions. I'm not talking about implementing everything in FB4, I just need to understand how to build the foundation that could be extended later with minimal efforts. Dmitry Firebird-Devel mailing list, web interface at
