When you get files from an external source no amount of checking is enough.
Yechiel Adar Mehish ----- Original Message ----- To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> Sent: Tuesday, October 29, 2002 6:19 PM Actually, I am sending and receiving files. They will be handling the FTP of the files, and making sure it has FTPed correctly. I just have to have a sanity check of the file. Basically, I decided to prefix each data line with 'DAT', and the "CRC line" with 'CRC'. The flat file is "read into the database" via an external table. I query the CRC record, get the expected record count and then count how many rows were actually sent. And then if there is a number column, then I sum that up and check it against the "CRC" expected sum. Perhaps all this is overkill, but I know that the odds of data corruption are slim to none. I don't like making assumptions, and I can't assume the file is ok. Many thanks to everyone that responded!!! -----Original Message----- Sent: Monday, October 28, 2002 11:54 AM To: Multiple recipients of list ORACLE-L But he was talking about sending, not receiving. and he says that the ftp is assured to work ok. Yechiel Adar Mehish ----- Original Message ----- To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> Sent: Friday, October 25, 2002 2:48 AM > Unfortunately it is a trust issue... > > Trust me when I say a file can get scrambled. I have seen it happen. > In our wierdest scenario two received files appeared to be merged into > a single file - on the source system they had two intact files, on our system > 1.5 files merged into a single file and .5 of a file missing. > > We could never replicate it, we had extensive testing on ftp > processes, etc, all we know is that it happened and our validation > techniques saw it and saved us a lot of greif. > > > > > > "Yechiel Adar" > <adar76@inter. To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> > net.il> cc: > Sent by: Subject: Re: Flat file generation integrity ideas... > [EMAIL PROTECTED] > om > > > 25/10/2002 > 06:14 > Please respond > to ORACLE-L > > > > > > > I do not see how the file can get "scrambled". > You write it out ok. > The ftp is guaranteed. > So what is the problem. > > I will go along with the suggestion to zip it. It saves on the ftp > time and > also gives you some protection. > > Yechiel Adar > Mehish > ----- Original Message ----- > From: Grabowy, Chris > To: Multiple recipients of list ORACLE-L > Sent: Thursday, October 24, 2002 7:34 PM > Subject: Flat file generation integrity ideas... > > > > I have to create packages that will generate several flat files of > data from tables that will be sent to other systems to be processed. > > > I am looking for ideas on how to ensure data integrity in the flat > files. > > > For example, the expected record count is stored on the first line of > the file to ensure that the correct amount of records was received. > > > The systems group is chartered to ensure the flat files are correctly > FTPed between systems, so that's covered. > > > I just worry that if "somehow" a flat file is scrambled then the scrambled > data is loaded into the database, therefore corrupting it. > > > At this phase, XML is not an option > > > I keep thinking that some sort of CRC should be stored with each line > in the flat file. And then before the line is loaded into the > database, the CRC is compared against the generated CRC of the just > read line. Has anyone done anything like this? Any examples out > there? > > > Many TIA!! > > > > > > > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> >>>> > Privileged/Confidential information may be contained in this message. > If you are not the addressee indicated in this message > (or responsible for delivery of the message to such person), > you may not copy or deliver this message to anyone. In > such case, you should destroy this message and kindly notify the sender > by reply e-mail or by telephone on (61 3) 9612-6999. > Please advise immediately if you or your employer does not consent to > Internet e-mail for messages of this kind. > Opinions, conclusions and other information in this message > that do not relate to the official business of > Transurban City Link Ltd > shall be understood as neither given nor endorsed by it. > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> >>>> > > -- > Please see the official ORACLE-L FAQ: http://www.orafaq.com > -- > Author: Mark Richard > INET: [EMAIL PROTECTED] > > Fat City Network Services -- 858-538-5051 http://www.fatcity.com > San Diego, California -- Mailing list and web hosting services > --------------------------------------------------------------------- > To REMOVE yourself from this mailing list, send an E-Mail message > to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in > the message BODY, include a line containing: UNSUB ORACLE-L (or the > name of mailing list you want to be removed from). You may also send > the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Yechiel Adar INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Grabowy, Chris INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Yechiel Adar INET: [EMAIL PROTECTED] Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).
