Re: {Disarmed} Re: [firebird-support] Forced write, page size and buffer size
Am 19.08.2014 um 18:30 schrieb 'Carlos H. Cantu' lis...@warmboot.com.br [firebird-support]: Re: [firebird-support] Forced write, page size and buffer size The best thing would be if someone could simulate some crashes (like power failure) in both configurations and report back about existence of corruption. Such an simulation does hardly cover all possible problems. In general there was an statement about old interbase database which should be up again shortly after an power fail in an tank-environment caused by firing the gun. There is some chance that firebird can recover after crash. The Problem itself does arise from the order the physical writes are done. If the OS (or even the disk) does reorder writes I would suspect to generate problems. You can simulate power-fail by using QEMU emulation and killing the emulator. Elmar ++ Visit http://www.firebirdsql.org and click the Documentation item on the main (top) menu. Try FAQ and other links from the left-side menu there. Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/ ++ Yahoo Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/firebird-support/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/firebird-support/join (Yahoo! ID required) * To change settings via email: firebird-support-dig...@yahoogroups.com firebird-support-fullfeatu...@yahoogroups.com * To unsubscribe from this group, send an email to: firebird-support-unsubscr...@yahoogroups.com * Your use of Yahoo Groups is subject to: https://info.yahoo.com/legal/us/yahoo/utos/terms/
Re: {Disarmed} Re: [firebird-support] Forced write, page size and buffer size
I never said it would cover all possibilities. I just said that instead of guessing or stay only with the theory, a real test could give more trustful results ;) Carlos Firebird Performance in Detail - http://videos.firebirddevelopersday.com www.firebirdnews.org - www.FireBase.com.br EHehdfs Am 19.08.2014 um 18:30 schrieb 'Carlos H. Cantu' lis...@warmboot.com.br EHehdfs [firebird-support]: Re: [firebird-support] Forced write, page size and buffer size The best thing would be if someone could simulate some crashes (like power failure) in both configurations and report back about existence of corruption. EHehdfs Such an simulation does hardly cover all possible problems. EHehdfs In general there was an statement about old interbase database which EHehdfs should be up again shortly after an power fail in an tank-environment EHehdfs caused by firing the gun. There is some chance that firebird can recover EHehdfs after crash. EHehdfs The Problem itself does arise from the order the physical writes are EHehdfs done. If the OS (or even the disk) does reorder writes I would suspect EHehdfs to generate problems. EHehdfs You can simulate power-fail by using QEMU emulation and killing the EHehdfs emulator. EHehdfs Elmar EHehdfs EHehdfs EHehdfs ++ EHehdfs Visit http://www.firebirdsql.org and click the Documentation item EHehdfs on the main (top) menu. Try FAQ and other links from the left-side menu there. EHehdfs Also search the knowledgebases at EHehdfs http://www.ibphoenix.com/resources/documents/ EHehdfs ++ EHehdfs EHehdfs Yahoo Groups Links
Re: [firebird-support] Forced write, page size and buffer size
The hard truth is that the only _absolute guarantee_ to prevent database corruption is FW = ON. Provided that the file system also has barrier enabled ... Regards, Aldo ++ Visit http://www.firebirdsql.org and click the Documentation item on the main (top) menu. Try FAQ and other links from the left-side menu there. Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/ ++ Yahoo Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/firebird-support/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/firebird-support/join (Yahoo! ID required) * To change settings via email: firebird-support-dig...@yahoogroups.com firebird-support-fullfeatu...@yahoogroups.com * To unsubscribe from this group, send an email to: firebird-support-unsubscr...@yahoogroups.com * Your use of Yahoo Groups is subject to: https://info.yahoo.com/legal/us/yahoo/utos/terms/
Re: [firebird-support] Forced write, page size and buffer size
I've a question regarding this, since the context has been in Linux. I run 2.5.2 on a virtual Server 2008R2 machine. Is FW on by default on this install? If not, is that a setting in the conf file or do I need to use GFix On Wed, Aug 20, 2014 at 9:05 AM, Aldo Caruso aldo.car...@argencasas.com [firebird-support] firebird-support@yahoogroups.com wrote: The hard truth is that the only _absolute guarantee_ to prevent database corruption is FW = ON. Provided that the file system also has barrier enabled ... Regards, Aldo ++ Visit http://www.firebirdsql.org and click the Documentation item on the main (top) menu. Try FAQ and other links from the left-side menu there. Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/ ++ Yahoo Groups Links -- Dixon Epperson
Re: [firebird-support] Forced write, page size and buffer size
You can check whether forced writes status is on or off submitting on a FB console the following command SHOW DATABASE In order to set it on / off, you shoud run gfix -user [user_name] -password [psw] -w [sync | async] [database_name] Regards, Aldo
Re: [firebird-support] Forced write, page size and buffer size
Thanks for the tip. I did take the default install and yes, FW is on. On Wed, Aug 20, 2014 at 10:08 AM, Aldo Caruso aldo.car...@argencasas.com [firebird-support] firebird-support@yahoogroups.com wrote: You can check whether forced writes status is on or off submitting on a FB console the following command SHOW DATABASE In order to set it on / off, you shoud run gfix -user [user_name] -password [psw] -w [sync | async] [database_name] Regards, Aldo -- Dixon Epperson
Re: [firebird-support] Forced write, page size and buffer size
On Aug 20, 2014, at 9:05 AM, Aldo Caruso aldo.car...@argencasas.com [firebird-support] firebird-support@yahoogroups.com wrote: The hard truth is that the only _absolute guarantee_ to prevent database corruption is FW = ON. Provided that the file system also has barrier enabled ... Firebird's forced write should be sufficient if the file system and disk honor forced write (fsync) - and if Firebird implements forced write correctly. If those criteria are met, barriers serve only to protect the disk journal. Firebird long predates file system journalling and operates correctly without journals. Best regards, Ann
RE: [firebird-support] Forced write, page size and buffer size
I run 2.5.2 on a virtual Server 2008R2 machine. Is FW on by default on this install? FW is not a function of the install or server config, it is a database property that is determined by the OS which was used to create the database*. For all version of Windows the default = ON I am not sure about *nux platforms, other will need to confirm. Sean * 1 - It is possible to copy a database between Windows and Linux (with the same CPU architecture). So, a database created on Windows would have FW = ON on Linux, even if the default for that platform is FW=OFF. 2- The property survives backup/restore and can only be changed using gfix. So, create and backup from Windows and restore to Linux would have FW=ON.
Re: [firebird-support] Forced write, page size and buffer size
Ann, thanks for the correction. I erroneously missinterpreted that file system barriers were also necessary. Aldo ++ Visit http://www.firebirdsql.org and click the Documentation item on the main (top) menu. Try FAQ and other links from the left-side menu there. Also search the knowledgebases at http://www.ibphoenix.com/resources/documents/ ++ Yahoo Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/firebird-support/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/firebird-support/join (Yahoo! ID required) * To change settings via email: firebird-support-dig...@yahoogroups.com firebird-support-fullfeatu...@yahoogroups.com * To unsubscribe from this group, send an email to: firebird-support-unsubscr...@yahoogroups.com * Your use of Yahoo Groups is subject to: https://info.yahoo.com/legal/us/yahoo/utos/terms/
Re: [firebird-support] Forced write, page size and buffer size
Sean, thanks for your answer. Aldo El 18/08/14 a las 15:59, 'Leyne, Sean' s...@broadviewsoftware.com [firebird-support] escibió: Given this scenario my questions are the following: 1) Does it makes sense to activate forced writes on Linux ? Different versions of Linux file systems provide better protection than others. Personally, I believe that forced write = ON is necessary for all OS -- but I suspect that I am in the minority (some will swear they have never had a problem/db corruption, but I am too chicken to take the risk -- I like sleeping at night). 2) Is increasing the page size the right approach to improve performance ? Which are the drawbacks of setting page size to 16K ? There is no universally true answer, it depends. 3) The same question stands for cache pages: is it useful ( or has disadvantages ) to have so many cache pages as there are on disk pages ( provided there is enough RAM size ) ? As long as you have the RAM there is no problem. Sean
Re: [firebird-support] Forced write, page size and buffer size
Rich, I got the point: with forced writes off you can't be sure of when the data is sent to disk. But, in that case, what is the point of turning it off ? If you must wait for the server shut down to be 100% sure that the data is written to disk, isn't the risk too high to have forced write off ? I suspect that data is written to disk far before than the server shut down or a connection close, otherwise FB designers shoudn't leave forced write as an option, it should always be activated. Aldo El 18/08/14 a las 16:11, 'Saunders, Rich' greym...@mykolab.com [firebird-support] escibió: On 2014-08-18 14:51, Aldo Caruso aldo.car...@argencasas.com [firebird-support] wrote: After that I noticed that some massive updates ( 100K records ) took nearly 5 minutes, whereas with async writes it used to take 10 seconds. Of course, we don't know how much work actually took place during that 10 second period when forced writes were off. Could be that very little of the update was actually on disk at that point. So do you consider the massive update actually done at that point? Also while forced writes are off, when is all that 5 minutes worth of work actually done? When the database is closed? When the server was shut down? Thats's the problem with forced writes being off - you never know. -- Cheers! Rich Saunders
Re: [firebird-support] Forced write, page size and buffer size
Carlos, I agree with you. The performance degradation was very high on an ext4 file system ( which has barrier enabled by default ). On the other hand, I found no noticiable performance degradation on an ext3 file system ( which has barrier disabled by default ). Having barrier disabled on a server with an ext3 file system, does FW=ON improve reliability or is it useless ? Aldo Caruso El 18/08/14 a las 16:27, 'Carlos H. Cantu' lis...@warmboot.com.br [firebird-support] escibió: Usually, if you turn FW = ON on Linux, and your filesystem has barrier enabled, it will affect performance of batch updates really badly. You would either accept the performance degradation, or disable one of them (FW or barrier). Carlos Firebird Performance in Detail - http://videos.firebirddevelopersday.com www.firebirdnews.org - www.FireBase.com.br ACacacfs Hello, ACacacfs For reliability reasons, I decided to turn on forced writes on a ACacacfs database running on Linux. ACacacfs After that I noticed that some massive updates ( 100K records ) took ACacacfs nearly 5 minutes, whereas with async writes it used to take 10 seconds. ACacacfs One solution is, of course, disabling sync writes when doing massive ACacacfs updates. Unfortunately not always massive updates are under database ACacacfs admin control ( some end users actions can lead to massive updates, ACacacfs indirectly, by means of triggers ). ACacacfs Another aproach I tested was augmenting page size from its default ACacacfs value ( 4 KB ) to its maximum allowed value ( 16 KB ). The speed was ACacacfs notably enhaced ( 1 minute for the update + 10 seconds for the commit, ACacacfs but sometimes 2 seconds for the update and 40 seconds for the commit). ACacacfs It should be pointed out that 4 KB was fine, taking into account ACacacfs record size ( max. 300 bytes ) and index max depth ( always 3 ). ACacacfs Going one step further, I augmented cached pages from its default ( ACacacfs 2048 ) to 8192. Some small performance improvement was observed, but not ACacacfs very significative. It shoud be noted also that, with a 16 KB page size, ACacacfs the database has 5700 pages on disk, so there are enough cache pages to ACacacfs hold the entire database. ACacacfs Given this scenario my questions are the following: ACacacfs 1) Does it makes sense to activate forced writes on Linux ? ACacacfs 2) Is increasing the page size the right approach to improve performance ACacacfs ? Which are the drawbacks of setting page size to 16K ? ACacacfs 3) The same question stands for cache pages: is it useful ( or has ACacacfs disadvantages ) to have so many cache pages as there are on disk pages ( ACacacfs provided there is enough RAM size ) ? ACacacfs Thanks in advance for any clue. ACacacfs Aldo ACacacfs ACacacfs ACacacfs ++ ACacacfs Visit http://www.firebirdsql.org and click the Documentation item ACacacfs on the main (top) menu. Try FAQ and other links from the left-side menu there. ACacacfs Also search the knowledgebases at ACacacfs http://www.ibphoenix.com/resources/documents/ ACacacfs ++ ACacacfs ACacacfs Yahoo Groups Links
Re: [firebird-support] Forced write, page size and buffer size
RE: [firebird-support] Forced write, page size and buffer size
Afair, in previous talks, Ann suggested that if you have to choose one or another, choose FW ON. The best thing would be if someone could simulate some crashes (like power failure) in both configurations and report back about existence of corruption. Unfortunately, it is not always possible for a simulation to catch all possible modalities. The hard truth is that the only _absolute guarantee_ to prevent database corruption is FW = ON. Sean
RE: [firebird-support] Forced write, page size and buffer size
Given this scenario my questions are the following: 1) Does it makes sense to activate forced writes on Linux ? Different versions of Linux file systems provide better protection than others. Personally, I believe that forced write = ON is necessary for all OS -- but I suspect that I am in the minority (some will swear they have never had a problem/db corruption, but I am too chicken to take the risk -- I like sleeping at night). 2) Is increasing the page size the right approach to improve performance ? Which are the drawbacks of setting page size to 16K ? There is no universally true answer, it depends. 3) The same question stands for cache pages: is it useful ( or has disadvantages ) to have so many cache pages as there are on disk pages ( provided there is enough RAM size ) ? As long as you have the RAM there is no problem. Sean
Re: [firebird-support] Forced write, page size and buffer size
On 2014-08-18 14:51, Aldo Caruso aldo.car...@argencasas.com [firebird-support] wrote: After that I noticed that some massive updates ( 100K records ) took nearly 5 minutes, whereas with async writes it used to take 10 seconds. Of course, we don't know how much work actually took place during that 10 second period when forced writes were off. Could be that very little of the update was actually on disk at that point. So do you consider the massive update actually done at that point? Also while forced writes are off, when is all that 5 minutes worth of work actually done? When the database is closed? When the server was shut down? Thats's the problem with forced writes being off - you never know. -- Cheers! Rich Saunders
Re: [firebird-support] Forced write, page size and buffer size
Usually, if you turn FW = ON on Linux, and your filesystem has barrier enabled, it will affect performance of batch updates really badly. You would either accept the performance degradation, or disable one of them (FW or barrier). Carlos Firebird Performance in Detail - http://videos.firebirddevelopersday.com www.firebirdnews.org - www.FireBase.com.br ACacacfs Hello, ACacacfsFor reliability reasons, I decided to turn on forced writes on a ACacacfs database running on Linux. ACacacfsAfter that I noticed that some massive updates ( 100K records ) took ACacacfs nearly 5 minutes, whereas with async writes it used to take 10 seconds. ACacacfsOne solution is, of course, disabling sync writes when doing massive ACacacfs updates. Unfortunately not always massive updates are under database ACacacfs admin control ( some end users actions can lead to massive updates, ACacacfs indirectly, by means of triggers ). ACacacfsAnother aproach I tested was augmenting page size from its default ACacacfs value ( 4 KB ) to its maximum allowed value ( 16 KB ). The speed was ACacacfs notably enhaced ( 1 minute for the update + 10 seconds for the commit, ACacacfs but sometimes 2 seconds for the update and 40 seconds for the commit). ACacacfsIt should be pointed out that 4 KB was fine, taking into account ACacacfs record size ( max. 300 bytes ) and index max depth ( always 3 ). ACacacfsGoing one step further, I augmented cached pages from its default ( ACacacfs 2048 ) to 8192. Some small performance improvement was observed, but not ACacacfs very significative. It shoud be noted also that, with a 16 KB page size, ACacacfs the database has 5700 pages on disk, so there are enough cache pages to ACacacfs hold the entire database. ACacacfsGiven this scenario my questions are the following: ACacacfs 1) Does it makes sense to activate forced writes on Linux ? ACacacfs 2) Is increasing the page size the right approach to improve performance ACacacfs ? Which are the drawbacks of setting page size to 16K ? ACacacfs 3) The same question stands for cache pages: is it useful ( or has ACacacfs disadvantages ) to have so many cache pages as there are on disk pages ( ACacacfs provided there is enough RAM size ) ? ACacacfs Thanks in advance for any clue. ACacacfs Aldo ACacacfs ACacacfs ACacacfs ++ ACacacfs Visit http://www.firebirdsql.org and click the Documentation item ACacacfs on the main (top) menu. Try FAQ and other links from the left-side menu there. ACacacfs Also search the knowledgebases at ACacacfs http://www.ibphoenix.com/resources/documents/ ACacacfs ++ ACacacfs ACacacfs Yahoo Groups Links
Re: [firebird-support] Forced write
On Thu, 18 Jul 2013 09:20:58 +1200, Helen Borrie hele...@iinet.net.au wrote: At 08:55 a.m. 18/07/2013, rddymanohar wrote: Hi, We are on Firebird 1.56 and recently encountered database corruption problem with a customer. Found that forced write was turned off on the database and after the database was restored turned it on. However on checking the next day forced write was again turned off. We do daily sweep, reindex and backup on the database as maintenance at the the eod. Will any of this cause forced write to be turned off. None of the above will change the Forced Write setting. However, in the very ancient version of Firebird that you are using, a client can do it, via the connection API. The very first thing I would recommend is to examine closely the connection settings on all client applications. A common source of this problem is Delphi applications that provide properties and/or arrays of properties in the connection component (connection alone, or database component). If this is your case, look at *both* the property in question (ForcedWrites TRUE/FALSE) and *also* the Params property of the connection/database component. For reference this is fixed since Firebird 2.0.5 / 2.1.2 (around 2009): http://www.firebirdsql.org/file/documentation/release_notes/html/rlsnotes207.html#apiods-api-dpb205 Mark
Re: [firebird-support] Forced write
On Thu, 18 Jul 2013 09:20:58 +1200, Helen Borrie hele...@iinet.net.au wrote: At 08:55 a.m. 18/07/2013, rddymanohar wrote: Hi, We are on Firebird 1.56 and recently encountered database corruption problem with a customer. Found that forced write was turned off on the database and after the database was restored turned it on. However on checking the next day forced write was again turned off. We do daily sweep, reindex and backup on the database as maintenance at the the eod. Will any of this cause forced write to be turned off. None of the above will change the Forced Write setting. However, in the very ancient version of Firebird that you are using, a client can do it, via the connection API. The very first thing I would recommend is to examine closely the connection settings on all client applications. A common source of this problem is Delphi applications that provide properties and/or arrays of properties in the connection component (connection alone, or database component). If this is your case, look at *both* the property in question (ForcedWrites TRUE/FALSE) and *also* the Params property of the connection/database component. For reference this is fixed since Firebird 2.0.5 / 2.1.2 (around 2009): Time flies. ;-) But I guess as the majority out there are using SYSDBA even for ordinary application connections, they still need to have an eye on that. http://www.firebirdsql.org/file/documentation/release_notes/html/rlsnotes207.html#apiods-api-dpb205 -- With regards, Thomas Steinmaurer http://www.upscene.com/ Professional Tools and Services for Firebird FB TraceManager, IB LogManager, Database Health Check, Tuning etc.
RE: [firebird-support] Forced write
We are on Firebird 1.56 and recently encountered database corruption problem with a customer. Found that forced write was turned off on the database and after the database was restored turned it on. However on checking the next day forced write was again turned off. We do daily sweep, reindex and backup on the database as maintenance at the the eod. Will any of this cause forced write to be turned off. None of those operations, on their own, will change the forced write flag setting. The setting can be changed thru gfix command or API call. Sean P.S. A restore would not change the setting on its own -- it would restore using the setting that was present from the backup.
Re: [firebird-support] Forced write
At 08:55 a.m. 18/07/2013, rddymanohar wrote: Hi, We are on Firebird 1.56 and recently encountered database corruption problem with a customer. Found that forced write was turned off on the database and after the database was restored turned it on. However on checking the next day forced write was again turned off. We do daily sweep, reindex and backup on the database as maintenance at the the eod. Will any of this cause forced write to be turned off. None of the above will change the Forced Write setting. However, in the very ancient version of Firebird that you are using, a client can do it, via the connection API. The very first thing I would recommend is to examine closely the connection settings on all client applications. A common source of this problem is Delphi applications that provide properties and/or arrays of properties in the connection component (connection alone, or database component). If this is your case, look at *both* the property in question (ForcedWrites TRUE/FALSE) and *also* the Params property of the connection/database component. Helen Borrie, Support Consultant, IBPhoenix (Pacific) Author of The Firebird Book and The Firebird Book Second Edition http://www.firebird-books.net __
Re: [firebird-support] Forced Write turning off
On Tue, 12 Feb 2013 21:53:24 +0100, Thomas Steinmaurer t...@iblogmanager.com wrote: We recently had database corruptions and had turned on forced write on our databases on windows environment but we found that after we run our client application gstat was no more showing the forced write status. We are using Delphi and IBO components but not sure what in the application is resetting this status. Has anyone come across this problem or know what could be doing this. Yes. What IBO version are you using? AFAIR older IBO versions can switch the TIB_Connection.ForcedWrites property from dpbDefault to something else by accident, e.g. dpbFalse, when you double-click the TIB_Connection component to open the connection editor and switch to the characteristic tab (or even without that) Check out the ForcedWrites property before deploying the application. The use of these dpb items is no longer possible for users other than SYSDBA or owner in Firebird 2.5: http://www.firebirdsql.org/file/documentation/release_notes/html/rlsnotes252.html#rnfb25-apiods-api-tighter A very good reason to not use SYSDBA or owner for connecting to a database for production work, I'd say. Mark
Re: [firebird-support] Forced Write turning off
On Tue, 12 Feb 2013 21:53:24 +0100, Thomas Steinmaurer t...@iblogmanager.com wrote: We recently had database corruptions and had turned on forced write on our databases on windows environment but we found that after we run our client application gstat was no more showing the forced write status. We are using Delphi and IBO components but not sure what in the application is resetting this status. Has anyone come across this problem or know what could be doing this. Yes. What IBO version are you using? AFAIR older IBO versions can switch the TIB_Connection.ForcedWrites property from dpbDefault to something else by accident, e.g. dpbFalse, when you double-click the TIB_Connection component to open the connection editor and switch to the characteristic tab (or even without that) Check out the ForcedWrites property before deploying the application. The use of these dpb items is no longer possible for users other than SYSDBA or owner in Firebird 2.5: http://www.firebirdsql.org/file/documentation/release_notes/html/rlsnotes252.html#rnfb25-apiods-api-tighter Yes, I know. The thing is, I guess the majority of deployments don't care about separated Firebird user accounts and are running with the single user (SYSDBA / db owner) approach. ;-) -- With regards, Thomas Steinmaurer http://www.upscene.com/
Re: [firebird-support] Forced Write turning off
We recently had database corruptions and had turned on forced write on our databases on windows environment but we found that after we run our client application gstat was no more showing the forced write status. We are using Delphi and IBO components but not sure what in the application is resetting this status. Has anyone come across this problem or know what could be doing this. Yes. What IBO version are you using? AFAIR older IBO versions can switch the TIB_Connection.ForcedWrites property from dpbDefault to something else by accident, e.g. dpbFalse, when you double-click the TIB_Connection component to open the connection editor and switch to the characteristic tab (or even without that) Check out the ForcedWrites property before deploying the application. -- With regards, Thomas Steinmaurer http://www.upscene.com/
