Re: Backup and journal file configurations
Tom, Yup, you're absolutely right. Without automated testing for larger systems the risk of shipping a breaking release must be significant. For an internal system that is bad enough - for those shipping vertical market products it must be the stuff of nightmares. I have been trying to digest a lot of 4D articles over the last few and I'm sure did come across a passing reference to Quality Assurance Automation Services - time to dig deeper into the detail of that, thanks. Regards, Narinder Chandi, ToolBox Systems Ltd. -- -Original Message- From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List <4d_tech@lists.4d.com> Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com> Date: Tuesday, 18 June 2019 at 22:23 To: 4D Tech Mailing List <4d_tech@lists.4d.com> Cc: Tom Benedict Subject: Re: Backup and journal file configurations I am very interested in increasing the level of support for automated testing in 4D. The larger and more complex the code base, the greater the testing burden. It has been my experience that the lack of automated test tool support in 4D is a roadblock to code refactoring. The cost of manual regression testing can easily exceed the programming cost. At the recent 4D World Tour Will Taylor presented a short description of Quality Assurance Automation Services. It looks very interesting and very modern (finally!). Details at https://us.4d.com/quality-assurance-automation-services. Tom Benedict > On Jun 18, 2019, at 11:40, Jim Hays via 4D_Tech <4d_tech@lists.4d.com> wrote: > > Hi Narinder, > > We aren't quite that advanced. We've mostly built test suites that cover > key functionality. > For example: > Run a bunch of reports to Excel xml, or text files. Manually check them > for accuracy, then lock the files. > Nightly, run the same reports and run diffs on them. If any differences, > let us know. > > Jim > > > On Tue, Jun 18, 2019 at 12:14 PM Narinder Chandi via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> Jim, >> >> Hi. I'm curious: >>>> ...in dev-build-test cycle. This is where we run automated, headless >> builds and tests with multiple data files... >> >> Are you able to describe this in more detail? I would quite like to learn >> what kind of build/test can be implemented for 4D. >> >> Having worked quite a lot with PHP frameworks, I have worked with test >> driven development (TDD) and more recently behaviour driven development >> (BDD). Have you been able implement anything like TDD in 4D or an >> approximation? Anybody else care to comment on their experience with this? >> Thanks. >> >> Regards, >> >> Narinder Chandi, >> ToolBox Systems Ltd. >> -- >> >> >> >> ** >> 4D Internet Users Group (4D iNUG) >> Archive: http://lists.4d.com/archives.html >> Options: https://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Backup and journal file configurations
I am very interested in increasing the level of support for automated testing in 4D. The larger and more complex the code base, the greater the testing burden. It has been my experience that the lack of automated test tool support in 4D is a roadblock to code refactoring. The cost of manual regression testing can easily exceed the programming cost. At the recent 4D World Tour Will Taylor presented a short description of Quality Assurance Automation Services. It looks very interesting and very modern (finally!). Details at https://us.4d.com/quality-assurance-automation-services. Tom Benedict > On Jun 18, 2019, at 11:40, Jim Hays via 4D_Tech <4d_tech@lists.4d.com> wrote: > > Hi Narinder, > > We aren't quite that advanced. We've mostly built test suites that cover > key functionality. > For example: > Run a bunch of reports to Excel xml, or text files. Manually check them > for accuracy, then lock the files. > Nightly, run the same reports and run diffs on them. If any differences, > let us know. > > Jim > > > On Tue, Jun 18, 2019 at 12:14 PM Narinder Chandi via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> Jim, >> >> Hi. I'm curious: ...in dev-build-test cycle. This is where we run automated, headless >> builds and tests with multiple data files... >> >> Are you able to describe this in more detail? I would quite like to learn >> what kind of build/test can be implemented for 4D. >> >> Having worked quite a lot with PHP frameworks, I have worked with test >> driven development (TDD) and more recently behaviour driven development >> (BDD). Have you been able implement anything like TDD in 4D or an >> approximation? Anybody else care to comment on their experience with this? >> Thanks. >> >> Regards, >> >> Narinder Chandi, >> ToolBox Systems Ltd. >> -- >> >> >> >> ** >> 4D Internet Users Group (4D iNUG) >> Archive: http://lists.4d.com/archives.html >> Options: https://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Backup and journal file configurations
Hi Narinder, We aren't quite that advanced. We've mostly built test suites that cover key functionality. For example: Run a bunch of reports to Excel xml, or text files. Manually check them for accuracy, then lock the files. Nightly, run the same reports and run diffs on them. If any differences, let us know. Jim On Tue, Jun 18, 2019 at 12:14 PM Narinder Chandi via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Jim, > > Hi. I'm curious: > >> ...in dev-build-test cycle. This is where we run automated, headless > builds and tests with multiple data files... > > Are you able to describe this in more detail? I would quite like to learn > what kind of build/test can be implemented for 4D. > > Having worked quite a lot with PHP frameworks, I have worked with test > driven development (TDD) and more recently behaviour driven development > (BDD). Have you been able implement anything like TDD in 4D or an > approximation? Anybody else care to comment on their experience with this? > Thanks. > > Regards, > > Narinder Chandi, > ToolBox Systems Ltd. > -- > > > > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Backup and journal file configurations
Jim, Hi. I'm curious: >> ...in dev-build-test cycle. This is where we run automated, headless builds >> and tests with multiple data files... Are you able to describe this in more detail? I would quite like to learn what kind of build/test can be implemented for 4D. Having worked quite a lot with PHP frameworks, I have worked with test driven development (TDD) and more recently behaviour driven development (BDD). Have you been able implement anything like TDD in 4D or an approximation? Anybody else care to comment on their experience with this? Thanks. Regards, Narinder Chandi, ToolBox Systems Ltd. -- ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Backup and journal file configurations
We have an answer for our problem, which might help other 4D'ers. How to avoid .journal problems in dev-build-test cycle. This is where we run automated, headless builds and tests with multiple data files, and we don't want them to hang up looking and asking for journal files. 1. get data file (just .4DD) from customer (it will always have journaling turned on) 2. open with 4D Server 3. it asks for journal file - click Create 4. when it asks to do backup now, click Cancel (don't do backup) 5. journal file is not created yet 6. go directly to Database Settings > Backup and uncheck the "use journal" setting - the journal file won't be created now Data file is now ready for use within our dev/build system where we don't want journal files active. HTH, Jim On Tue, Apr 2, 2019 at 3:49 PM Jim Hays wrote: > Sorry I have been silent for a week - tied up with other things. > > One reason for not keeping backups and logs operating in our dev > environment is that we often run databases in single user mode, and there > is no automated backup. If you do a lot of operations on the data, the > journal just keeps growing. I suppose we could put some code in for this > scenario, to always run a backup when it is shut down, and only keep 1 > backup. > > I've found that when we take a copy of a .4DD from a customer, and put it > on our dev server for testing purposes, the server will ask for the journal > at startup. > So the data file knows a journal is in use. > > At that point, we need to disable the journal and backup settings. > If we don't, and then run our automated build and testing system, we run > into trouble. > Just moving the 4DB file over to the "build" folder, brings along the > requirement for a log file, even though the data file used for the build > had everything turned off before. > The automated build is interrupted by the UI looking for a journal file - > you must pick existing, create, or cancel. > So the structure file also knows a journal is in use. > > But there are no controls I know to turn it off programmatically, or by > some xml setting. > > There are also issues if we try to keep all the backup and journal files > all the time. > The backup.xml goes with the data. Each time we get a data file we need > to get the backup.xml as well, and place it in Preferences\Backup next to > our structure. > The backup.xml has a different data file name in it - so in essence, there > is data information in the 4DB structure folder. > > It's quite likely I'm missing something in all this. > Maybe a good solution for us will be to just triage any data file that > comes in before we allow it into the "automated" cycle - whether for > development or testing. > > Thanks for listening! > > Jim > > > > > > On Mon, Mar 25, 2019 at 8:42 PM John DeSoi via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > >> >> > On Mar 25, 2019, at 5:11 PM, Tim Nevels via 4D_Tech < >> 4d_tech@lists.4d.com> wrote: >> > >> > When you are moving data files around between systems, just copy the >> entire data file folder. That will have the .4DD, .4DIndx, .Match and >> .journal files. If they are all in the same folder you can open that data >> file with another structure and it will not ask you for the location of the >> .journal. The key is to have all these files in the same folder and to >> always move the entire folder between machines. >> >> Having the .journal file next to the database file defeats the main >> benefit of having the journal. If the drive crashes, you could could >> recreate the database from the last backup and the journal file (assuming >> both are on another drive). >> >> > Now if you copy that datafile folder back to your development machine, >> you have to copy the .journal file too. I believe .journal file usage is >> internally stored in a .4DD, so once you have it set up, they must stay >> together, or you get the messages you are talking about. >> >> If your log file is on a second volume in production and your development >> machine has only one drive, there is no way to automate the restore of the >> backup without the log file prompts. I also wanted a fully automated >> restore with no log file or prompts to allow users to test experimental >> features. This is possible in every database I'm aware of, except for 4D. >> >> See feature request from Jeff Kain and other discussion here: >> >> https://forums.4d.com/Post/EN/22296877/1/22296878 >> >> >> John DeSoi, Ph.D. >> >> ** >> 4D Internet Users Group (4D iNUG) >> Archive: http://lists.4d.com/archives.html >> Options: https://lists.4d.com/mailman/options/4d_tech >> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com >> ** > > ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options:
Re: Backup and journal file configurations
Sorry I have been silent for a week - tied up with other things. One reason for not keeping backups and logs operating in our dev environment is that we often run databases in single user mode, and there is no automated backup. If you do a lot of operations on the data, the journal just keeps growing. I suppose we could put some code in for this scenario, to always run a backup when it is shut down, and only keep 1 backup. I've found that when we take a copy of a .4DD from a customer, and put it on our dev server for testing purposes, the server will ask for the journal at startup. So the data file knows a journal is in use. At that point, we need to disable the journal and backup settings. If we don't, and then run our automated build and testing system, we run into trouble. Just moving the 4DB file over to the "build" folder, brings along the requirement for a log file, even though the data file used for the build had everything turned off before. The automated build is interrupted by the UI looking for a journal file - you must pick existing, create, or cancel. So the structure file also knows a journal is in use. But there are no controls I know to turn it off programmatically, or by some xml setting. There are also issues if we try to keep all the backup and journal files all the time. The backup.xml goes with the data. Each time we get a data file we need to get the backup.xml as well, and place it in Preferences\Backup next to our structure. The backup.xml has a different data file name in it - so in essence, there is data information in the 4DB structure folder. It's quite likely I'm missing something in all this. Maybe a good solution for us will be to just triage any data file that comes in before we allow it into the "automated" cycle - whether for development or testing. Thanks for listening! Jim On Mon, Mar 25, 2019 at 8:42 PM John DeSoi via 4D_Tech <4d_tech@lists.4d.com> wrote: > > > On Mar 25, 2019, at 5:11 PM, Tim Nevels via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > > > > When you are moving data files around between systems, just copy the > entire data file folder. That will have the .4DD, .4DIndx, .Match and > .journal files. If they are all in the same folder you can open that data > file with another structure and it will not ask you for the location of the > .journal. The key is to have all these files in the same folder and to > always move the entire folder between machines. > > Having the .journal file next to the database file defeats the main > benefit of having the journal. If the drive crashes, you could could > recreate the database from the last backup and the journal file (assuming > both are on another drive). > > > Now if you copy that datafile folder back to your development machine, > you have to copy the .journal file too. I believe .journal file usage is > internally stored in a .4DD, so once you have it set up, they must stay > together, or you get the messages you are talking about. > > If your log file is on a second volume in production and your development > machine has only one drive, there is no way to automate the restore of the > backup without the log file prompts. I also wanted a fully automated > restore with no log file or prompts to allow users to test experimental > features. This is possible in every database I'm aware of, except for 4D. > > See feature request from Jeff Kain and other discussion here: > > https://forums.4d.com/Post/EN/22296877/1/22296878 > > > John DeSoi, Ph.D. > > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: Backup and journal file configurations
> On Mar 25, 2019, at 5:11 PM, Tim Nevels via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > When you are moving data files around between systems, just copy the entire > data file folder. That will have the .4DD, .4DIndx, .Match and .journal > files. If they are all in the same folder you can open that data file with > another structure and it will not ask you for the location of the .journal. > The key is to have all these files in the same folder and to always move the > entire folder between machines. Having the .journal file next to the database file defeats the main benefit of having the journal. If the drive crashes, you could could recreate the database from the last backup and the journal file (assuming both are on another drive). > Now if you copy that datafile folder back to your development machine, you > have to copy the .journal file too. I believe .journal file usage is > internally stored in a .4DD, so once you have it set up, they must stay > together, or you get the messages you are talking about. If your log file is on a second volume in production and your development machine has only one drive, there is no way to automate the restore of the backup without the log file prompts. I also wanted a fully automated restore with no log file or prompts to allow users to test experimental features. This is possible in every database I'm aware of, except for 4D. See feature request from Jeff Kain and other discussion here: https://forums.4d.com/Post/EN/22296877/1/22296878 John DeSoi, Ph.D. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Backup and journal file configurations
I'm having trouble understanding how to control the use of backups and journal files. Here are some things we do, or want to do: - Deployed servers have backups fully configured, with logs, recovery settings etc. We're doing this, and have not needed mirrors or anything of that level over the years. - Development server should NOT have backup and journal files. We switch data files a lot and it becomes a mess of files and requests to find the journal, create a journal, or errors with the journal. There should be no interruptions when running automated builds and tests - Single user instances should not have backups and journals. We use single user development, and for built application demos. The data is generally disposable or for demo purposes How do we manage this? - is there a tech note about this sort of thing? - programmatically turn settings on/off? - where is the "USE LOG FILE" setting kept? (why is it not in backup.xml?) - edit or externally manage the backup.xml file Thanks for any tips! Jim Hays ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
