Re: [U2] Unidata 6.1 Replication
My accuracy may be technically off on the minute details there, but regardless, UV dynamic files are not compatible with OS-level replication. This is from an email sent to me by Stephen O'Neal of IBM U2 Lab Services on 3-18-09: --- At the exact instance that a Dynamic File expands or contracts, or a static file increases overflow, there are multiple writes that occur. If any of the writes don't make it to disk, you will end up with a broken file. When you run FIXTOOL to fix the file, it will either: + Fix the file, or - Truncate the File where the mismatch occurred. --- Basically, any U2 file update operation that can't be done in a single write can lead to file corruption when using OS-level replication. Your options with UV are to implement transaction logging or use the workarounds I mentioned earlier. -John -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Anthony W. Youngman Sent: Wednesday, December 16, 2009 5:08 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Unidata 6.1 Replication In message e6179e13392ec14aabcd5272c3aedd61112ce...@exchangesvr.momtex.com, John Hester jhes...@momtex.com writes There are also issues with UV dynamic files (don't know about UD) because some of the file structure information is cached in memory and not immediately flushed to disk. This is no longer true. Can't remember which IBM'er it was, but it was confirmed that UV flushes everything at once, and it's been that way for quite a while now. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Do you replicate every file on the server ? --- On Wed, 12/16/09, John Hester jhes...@momtex.com wrote: From: John Hester jhes...@momtex.com Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Wednesday, December 16, 2009, 1:00 AM Brad is right. You can get bitten by OS-level replication with U2 because hashed files require more than one write when a record has to go to overflow space. If the primary box crashes when only one of these writes is done, the secondary box will have a corrupted file. There are also issues with UV dynamic files (don't know about UD) because some of the file structure information is cached in memory and not immediately flushed to disk. We do OS-level replication on UV and got around these issues by sizing all files to use no overflow and only using dynamic files for temp report work files. -John -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Address Sent: Tuesday, December 15, 2009 8:12 PM To: U2 Users List Subject: Re: [U2] Unidata 6.1 Replication Double-Take replicates byte by byte. --- On Tue, 12/15/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Tuesday, December 15, 2009, 10:56 PM Someone needs to write an i/o agent, so U2 can be used with products like Double-Take and keep record level integrity. Anyone with c++ experience out there? ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Brad and John, [ad] Since, we work with both UD and UV we found replication is not the same. We wrote a simple Java program that runs using UOJ and does the replication to one or many sites. The software can even replicate from Universe to Unidata or the other way. Additionally, we added support for TCL/ECL commands such as CREATE.FILE, BASIC, and most others. We don't have any problems with any release of Unidata 6.0 or greater or Universe 9.6 or greater. We don't have any issues with dynamic or hashed files since everything is replicated at the record level. This is all part of our amazing U2WebLink(tm) product that runs on Apache Tomcat. We even added native support for PHP. The product handles thousand upon thousand of transactions per hour without even a hiccup at our clients for many many years. [/ad] Regards, Doug www.u2logic.com/tools.html BTW: We heard that soon UD and UV will be using the same replication engine. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of John Hester Sent: Tuesday, December 15, 2009 11:01 PM To: U2 Users List Subject: Re: [U2] Unidata 6.1 Replication Brad is right. You can get bitten by OS-level replication with U2 because hashed files require more than one write when a record has to go to overflow space. If the primary box crashes when only one of these writes is done, the secondary box will have a corrupted file. There are also issues with UV dynamic files (don't know about UD) because some of the file structure information is cached in memory and not immediately flushed to disk. We do OS-level replication on UV and got around these issues by sizing all files to use no overflow and only using dynamic files for temp report work files. -John ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Good info to have. I'll have to evaluate your solution when it comes time to upgrade the UV servers again. -John -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Doug Sent: Wednesday, December 16, 2009 8:46 AM To: 'U2 Users List' Subject: Re: [U2] Unidata 6.1 Replication Brad and John, [ad] Since, we work with both UD and UV we found replication is not the same. We wrote a simple Java program that runs using UOJ and does the replication to one or many sites. The software can even replicate from Universe to Unidata or the other way. Additionally, we added support for TCL/ECL commands such as CREATE.FILE, BASIC, and most others. We don't have any problems with any release of Unidata 6.0 or greater or Universe 9.6 or greater. We don't have any issues with dynamic or hashed files since everything is replicated at the record level. This is all part of our amazing U2WebLink(tm) product that runs on Apache Tomcat. We even added native support for PHP. The product handles thousand upon thousand of transactions per hour without even a hiccup at our clients for many many years. [/ad] Regards, Doug www.u2logic.com/tools.html BTW: We heard that soon UD and UV will be using the same replication engine. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
In message e6179e13392ec14aabcd5272c3aedd61112ce...@exchangesvr.momtex.com, John Hester jhes...@momtex.com writes There are also issues with UV dynamic files (don't know about UD) because some of the file structure information is cached in memory and not immediately flushed to disk. This is no longer true. Can't remember which IBM'er it was, but it was confirmed that UV flushes everything at once, and it's been that way for quite a while now. Cheers, Wol -- Anthony W. Youngman pi...@thewolery.demon.co.uk 'Yings, yow graley yin! Suz ae rikt dheu,' said the blue man, taking the thimble. 'What *is* he?' said Magrat. 'They're gnomes,' said Nanny. The man lowered the thimble. 'Pictsies!' Carpe Jugulum, Terry Pratchett 1998 Visit the MaVerick web-site - http://www.maverick-dbms.org Open Source Pick ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Dave, That's good to hear: went thu lots of pain with replication at last place. However, can it be restarted after a crash, or does it keep enough data in the pipe to recover and start replicating again? thanks, Brad - Original Message - From: David Wolverton dwolv...@flash.net To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Wednesday, December 09, 2009 12:23 PM Subject: Re: [U2] Unidata 6.1 Replication Replication is QUITE useful at 7.1.20 and later I know -- we have lots of people using it now! Before this time, it had issues with large records, multiple overwrites one the same key, etc. At 7.1.20 they also improved the monitoring. Don't bother at less than 7.1.20 IMHO David W. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
For redundancy, it's hard to beat triple mirrored drives on a SAN, with RFS on a 4th i/o channel. You break the mirror between drives 2 3, and then drive 1 2 remain mirrored. Back up off #3. When finished, reestablish the mirror between 23. RFS can be then write to 2 drives with a 0,1 configuration sequentially - which is fast. However, RFS can take 25% to 50% of resources, so have a fast box!!! - Original Message - From: David Wolverton dwolv...@flash.net To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Wednesday, December 09, 2009 12:25 PM Subject: Re: [U2] Unidata 6.1 Replication Doubletake is an O/S answer -- but you cannot use the 'DoubleTake' data to host a reporting server, can you? UniData Replication is great for being able to have the database in two places at once with for either backup, hot spare, or 'query only' access. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Address Sent: Wednesday, December 09, 2009 8:33 AM To: U2 Users List Subject: Re: [U2] Unidata 6.1 Replication We are using Double-Take to replicate. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Someone needs to write an i/o agent, so U2 can be used with products like Double-Take and keep record level integrity. Anyone with c++ experience out there? - Original Message - From: Address mrparkl...@yahoo.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wednesday, December 09, 2009 9:33 AM Subject: Re: [U2] Unidata 6.1 Replication We are using Double-Take to replicate. --- On Wed, 12/9/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Wednesday, December 9, 2009, 8:33 AM Kevin, I would not use replication for U2 for 6x. I was given the mandate to use then IBM replication at a company I used to work for, and it didn't work. It wasn't on problem, but a whole hosts of problem. I set up two test boxes four different times and tested it. The 4th time I had 30 people from then IT staff (I was part of the staff then) hammer on it. Then we went live. It worked for 4 days then crashed the whole system. Then it worked for a few hours. Then, it kept crashing (hanging the servers). We had to pull it out. We were on Unidata, but it's the same code base. I can only say that if you want to go ahead with it, find someone who's using it successfully, and find out what environment they're in, version, ect. It's based on Unix and has different issues on different O/Ss. But, personally, I'd try RFS first, or write your own. That's what we ended up doing. And it worked. We used triggers and hammered the system 24/7. There were some issues, but we worked them all out, and we're able to process millions of transactions per day. What we ended up doing is finding all the weak spots in Unidata and avoiding them. The biggest weak spot was locks. The system couldn't hanldle the million+ locks per day. I went to double buffering, avoiding locks (an old GUI technique!). Also, I had to convert types of records to hex, and the UD hex conversion is slow. It's amazing what works in UD and scales, and what doesn't. To write one's own takes time and is expensive. I think that using mirrored drives is better!!! RFS??? Brad - Original Message - From: Kevin King precisonl...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Tuesday, December 08, 2009 12:22 PM Subject: [U2] Unidata 6.1 Replication Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Double-Take replicates byte by byte. --- On Tue, 12/15/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Tuesday, December 15, 2009, 10:56 PM Someone needs to write an i/o agent, so U2 can be used with products like Double-Take and keep record level integrity. Anyone with c++ experience out there? - Original Message - From: Address mrparkl...@yahoo.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wednesday, December 09, 2009 9:33 AM Subject: Re: [U2] Unidata 6.1 Replication We are using Double-Take to replicate. --- On Wed, 12/9/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Wednesday, December 9, 2009, 8:33 AM Kevin, I would not use replication for U2 for 6x. I was given the mandate to use then IBM replication at a company I used to work for, and it didn't work. It wasn't on problem, but a whole hosts of problem. I set up two test boxes four different times and tested it. The 4th time I had 30 people from then IT staff (I was part of the staff then) hammer on it. Then we went live. It worked for 4 days then crashed the whole system. Then it worked for a few hours. Then, it kept crashing (hanging the servers). We had to pull it out. We were on Unidata, but it's the same code base. I can only say that if you want to go ahead with it, find someone who's using it successfully, and find out what environment they're in, version, ect. It's based on Unix and has different issues on different O/Ss. But, personally, I'd try RFS first, or write your own. That's what we ended up doing. And it worked. We used triggers and hammered the system 24/7. There were some issues, but we worked them all out, and we're able to process millions of transactions per day. What we ended up doing is finding all the weak spots in Unidata and avoiding them. The biggest weak spot was locks. The system couldn't hanldle the million+ locks per day. I went to double buffering, avoiding locks (an old GUI technique!). Also, I had to convert types of records to hex, and the UD hex conversion is slow. It's amazing what works in UD and scales, and what doesn't. To write one's own takes time and is expensive. I think that using mirrored drives is better!!! RFS??? Brad - Original Message - From: Kevin King precisonl...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Tuesday, December 08, 2009 12:22 PM Subject: [U2] Unidata 6.1 Replication Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Brad is right. You can get bitten by OS-level replication with U2 because hashed files require more than one write when a record has to go to overflow space. If the primary box crashes when only one of these writes is done, the secondary box will have a corrupted file. There are also issues with UV dynamic files (don't know about UD) because some of the file structure information is cached in memory and not immediately flushed to disk. We do OS-level replication on UV and got around these issues by sizing all files to use no overflow and only using dynamic files for temp report work files. -John -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Address Sent: Tuesday, December 15, 2009 8:12 PM To: U2 Users List Subject: Re: [U2] Unidata 6.1 Replication Double-Take replicates byte by byte. --- On Tue, 12/15/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Tuesday, December 15, 2009, 10:56 PM Someone needs to write an i/o agent, so U2 can be used with products like Double-Take and keep record level integrity. Anyone with c++ experience out there? ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Kevin, I would not use replication for U2 for 6x. I was given the mandate to use then IBM replication at a company I used to work for, and it didn't work. It wasn't on problem, but a whole hosts of problem. I set up two test boxes four different times and tested it. The 4th time I had 30 people from then IT staff (I was part of the staff then) hammer on it. Then we went live. It worked for 4 days then crashed the whole system. Then it worked for a few hours. Then, it kept crashing (hanging the servers). We had to pull it out. We were on Unidata, but it's the same code base. I can only say that if you want to go ahead with it, find someone who's using it successfully, and find out what environment they're in, version, ect. It's based on Unix and has different issues on different O/Ss. But, personally, I'd try RFS first, or write your own. That's what we ended up doing. And it worked. We used triggers and hammered the system 24/7. There were some issues, but we worked them all out, and we're able to process millions of transactions per day. What we ended up doing is finding all the weak spots in Unidata and avoiding them. The biggest weak spot was locks. The system couldn't hanldle the million+ locks per day. I went to double buffering, avoiding locks (an old GUI technique!). Also, I had to convert types of records to hex, and the UD hex conversion is slow. It's amazing what works in UD and scales, and what doesn't. To write one's own takes time and is expensive. I think that using mirrored drives is better!!! RFS??? Brad - Original Message - From: Kevin King precisonl...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Tuesday, December 08, 2009 12:22 PM Subject: [U2] Unidata 6.1 Replication Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
We are using Double-Take to replicate. --- On Wed, 12/9/09, BraDav broadri...@comcast.net wrote: From: BraDav broadri...@comcast.net Subject: Re: [U2] Unidata 6.1 Replication To: U2 Users List u2-users@listserver.u2ug.org Date: Wednesday, December 9, 2009, 8:33 AM Kevin, I would not use replication for U2 for 6x. I was given the mandate to use then IBM replication at a company I used to work for, and it didn't work. It wasn't on problem, but a whole hosts of problem. I set up two test boxes four different times and tested it. The 4th time I had 30 people from then IT staff (I was part of the staff then) hammer on it. Then we went live. It worked for 4 days then crashed the whole system. Then it worked for a few hours. Then, it kept crashing (hanging the servers). We had to pull it out. We were on Unidata, but it's the same code base. I can only say that if you want to go ahead with it, find someone who's using it successfully, and find out what environment they're in, version, ect. It's based on Unix and has different issues on different O/Ss. But, personally, I'd try RFS first, or write your own. That's what we ended up doing. And it worked. We used triggers and hammered the system 24/7. There were some issues, but we worked them all out, and we're able to process millions of transactions per day. What we ended up doing is finding all the weak spots in Unidata and avoiding them. The biggest weak spot was locks. The system couldn't hanldle the million+ locks per day. I went to double buffering, avoiding locks (an old GUI technique!). Also, I had to convert types of records to hex, and the UD hex conversion is slow. It's amazing what works in UD and scales, and what doesn't. To write one's own takes time and is expensive. I think that using mirrored drives is better!!! RFS??? Brad - Original Message - From: Kevin King precisonl...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Tuesday, December 08, 2009 12:22 PM Subject: [U2] Unidata 6.1 Replication Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Replication is QUITE useful at 7.1.20 and later I know -- we have lots of people using it now! Before this time, it had issues with large records, multiple overwrites one the same key, etc. At 7.1.20 they also improved the monitoring. Don't bother at less than 7.1.20 IMHO David W. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Doubletake is an O/S answer -- but you cannot use the 'DoubleTake' data to host a reporting server, can you? UniData Replication is great for being able to have the database in two places at once with for either backup, hot spare, or 'query only' access. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Address Sent: Wednesday, December 09, 2009 8:33 AM To: U2 Users List Subject: Re: [U2] Unidata 6.1 Replication We are using Double-Take to replicate. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
Awesome, great feedback my friends. Appreciate it very much. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
[U2] Unidata 6.1 Replication
Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Unidata 6.1 Replication
7.1.8 is the minimum release I would recommend for using UniData Data Replication. Wally Terhune U2 Support Architect Rocket Software 4700 S. Syracuse Street, Suite 400 **Denver, CO 80237 **USA Tel: +1.720.475.8055 Email: wterh...@rs.com Web: www.rocketsoftware.com/u2 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Kevin King Sent: Tuesday, December 08, 2009 10:23 AM To: U2 Users List Subject: [U2] Unidata 6.1 Replication Is replication solid and functional in Unidata 6,1? Anyone using it with roaring success? -Kevin http://www.PrecisOnline.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
