I think "Flakey wireless LAN" in my brain read "Outside world" or "Not Client"
I wasn't suggesting that wireless was changing byte structure, but if you are streaming data, and your datastream gets disconnected, then you could end up with an incomplete transfer. I'm not 100% sure that midas catches all scenarios when working off a remote data instance ? Note we use dbx4mysql + midas. Note also that I cannot rule out the drivers either and also could be the data out of the db server, its a tricky one to track down, as you basically have a black box from db -> dbxmysql + midas... What I do know is that its very rare. e.g. maybe 1 in 10,000 usages corrupts the file and has occurred in more than 1 location, so it does not appear to be station specific. Logically with these stats, I can only put that down to a flakey connection, otherwise the error rate would be more often. Matt. > Hmmmm. > > "Flakey wireless LAN".... ? > > Surely wifi (or any LAN tech for that matter) has mechanisms built in to > the > protocol to ensure this sort of thing does not occur.... certainly even > with the flakiest of wifi connections I have never experienced corruption. > The wifi connection is either made, however slowly as a result of the > flakiness, or not made at all. > > > Are you positive that the "corruption" is not occurring directly in the > output of the remote server, rather than in the communication of the data > ? > > > Because if that's the case then checksuming will of course be a waste of > time > > > -----Original Message----- > From: delphi-boun...@delphi.org.nz [mailto:delphi-boun...@delphi.org.nz] > On > Behalf Of Todd > Sent: Monday, 17 January 2011 14:22 > To: NZ Borland Developers Group - Delphi List > Subject: Re: [DUG] Validating CDS files > > Hi Matthew > > It sounds like comparing a MD5 hash of the CDS file prior to sending and > subsequent to receiving the data stream would provide a sufficient check > on its integrity. > > Todd. >> Hi Todd, >> >> The ClientDatasets initially get populated via a remote server, and it >> is >> this process which in rare cases causes the corruption (e.g. customer on >> flakey wireless lan or similar). >> >> Longterm we will replace this mechanism so the dataset is populated >> server >> side and send back with a hash which will maintain integrity. >> >> For the moment, we are stuck with a mechanism which populates this cds >> data and then writes to file. >> >> Appending some metadata at this point is a valid option. Ill test and >> see >> if it circumvents the issue. The main issue though is that I'm not sure >> if >> the ClientDataset at this point knows that the data is corrupt, and >> therefore we could be exporting a corrupt data chunk packed with some >> metadata, which does not help. >> >> Thats why we really need some protection on load. >> >> Cheers, >> >> Matt. >> >> >> >> >>> Hi Matthew >>> >>> Are the CDS files being stored as disk files or in a database? How are >>> they being corrupted? Faulty back up media? Perhaps you could add some >>> meta-data to each file as it is saved. >>> >>> Todd. >>> >>>> The driver for the question, is that we have some application client >>>> datasets which are put into a defaulted state if a corrupt cds file is >>>> loaded. >>>> >>>> Yes with XML, we can just validate the XML, but we use the binary >>>> format >>>> so that solution does not apply. >>>> >>>> At present we basically have two solutions. >>>> >>>> 1. Load into a test clientdataset as suggested by Alistair. This is a >>>> valid solution but does add considerable load time into the startup. >>>> >>>> 2. Can load into application clientdatasets, and dispose and reload if >>>> error encountered. This is ok also but does require additional loading >>>> in >>>> case of error. >>>> >>>> What I'm really after is a file level test to check that file should >>>> even >>>> be attempted. e.g. open file stream seek start and seek end and check >>>> a >>>> couple of bytes... that type of thing. >>>> >>>> Matt. >>>> >>>> >>>> >>>> >>>> >>>>> What does "validate" mean for a CDS file ? >>>>> >>>>> For XML it obviously/intuitively means "is a valid XML representation >>>>> of a >>>>> CDS file" but for a CDS file, if you dumped it from a client data set >>>>> in >>>>> the >>>>> first place, what additional validation would be needed to ensure >>>>> that >>>>> it >>>>> is >>>>> a CDS file ? >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: delphi-boun...@delphi.org.nz >>>>> [mailto:delphi-boun...@delphi.org.nz] >>>>> On >>>>> Behalf Of Matthew Comb >>>>> Sent: Monday, 17 January 2011 10:27 >>>>> To: NZ Borland Developers Group - Delphi List >>>>> Subject: Re: [DUG] Validating CDS files >>>>> >>>>> Hey Cam, for space size reasons, we're using the CDS format not the >>>>> XML >>>>> format as the files get pretty large. >>>>> >>>>> Matt >>>>> >>>>> >>>>> >>>>>> If you have dumped the CDS as XML then you can validate it using >>>>>> XSD. >>>>>> I >>>>>> haven't come across a schema for it but it shouldn't be hard to >>>>>> make. >>>>>> >>>>>> Cameron Hart | Development Manager | Flow Software Limited >>>>>> P: +64 9 476 3569 x910 | M: +64 21 222 3569 | E: >>>>>> cameron.h...@flowsoftware.co.nz >>>>>> PO Box 305-237, Triton Plaza, Auckland 0757, New Zealand | >>>>>> www.flowsoftware.co.nz >>>>>> >>>>>> This message is intended for the addressee named above. It may >>>>>> contain >>>>>> privileged or confidential information. If you are not the intended >>>>>> recipient of this message you must not use, copy, distribute or >>>>>> disclose >>>>>> it to anyone. >>>>>> >>>>>> Please consider the environment before printing this email >>>>>> >>>>>> -----Original Message----- >>>>>> From: delphi-boun...@delphi.org.nz >>>>>> [mailto:delphi-boun...@delphi.org.nz] >>>>>> On Behalf Of Matthew Comb >>>>>> Sent: Monday, 17 January 2011 10:03 a.m. >>>>>> To: NZ Borland Developers Group - Delphi List >>>>>> Cc: NZ Borland Developers Group - Delphi List >>>>>> Subject: [DUG] Validating CDS files >>>>>> >>>>>> Does anyone know if there is a way to Validate a CDS file (midas) >>>>>> before >>>>>> attempting to load it into the dataset ? >>>>>> >>>>>> Matt >>>>>> >>>>>> _______________________________________________ >>>>>> NZ Borland Developers Group - Delphi mailing list >>>>>> Post: delphi@delphi.org.nz >>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>>>>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>>>>> Subject: >>>>>> unsubscribe >>>>>> >>>>>> _______________________________________________ >>>>>> NZ Borland Developers Group - Delphi mailing list >>>>>> Post: delphi@delphi.org.nz >>>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>>>>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>>>>> Subject: >>>>>> unsubscribe >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> NZ Borland Developers Group - Delphi mailing list >>>>> Post: delphi@delphi.org.nz >>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>>>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>>>> Subject: >>>>> unsubscribe >>>>> >>>>> _______________________________________________ >>>>> NZ Borland Developers Group - Delphi mailing list >>>>> Post: delphi@delphi.org.nz >>>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>>>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>>>> Subject: >>>>> unsubscribe >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> NZ Borland Developers Group - Delphi mailing list >>>> Post: delphi@delphi.org.nz >>>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>>> Subject: >>>> unsubscribe >>>> >>>> >>> _______________________________________________ >>> NZ Borland Developers Group - Delphi mailing list >>> Post: delphi@delphi.org.nz >>> Admin: http://delphi.org.nz/mailman/listinfo/delphi >>> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with >>> Subject: >>> unsubscribe >>> >>> >> >> _______________________________________________ >> NZ Borland Developers Group - Delphi mailing list >> Post: delphi@delphi.org.nz >> Admin: http://delphi.org.nz/mailman/listinfo/delphi >> Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: > unsubscribe >> > > _______________________________________________ > NZ Borland Developers Group - Delphi mailing list > Post: delphi@delphi.org.nz > Admin: http://delphi.org.nz/mailman/listinfo/delphi > Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: > unsubscribe > > _______________________________________________ > NZ Borland Developers Group - Delphi mailing list > Post: delphi@delphi.org.nz > Admin: http://delphi.org.nz/mailman/listinfo/delphi > Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: > unsubscribe > _______________________________________________ NZ Borland Developers Group - Delphi mailing list Post: delphi@delphi.org.nz Admin: http://delphi.org.nz/mailman/listinfo/delphi Unsubscribe: send an email to delphi-requ...@delphi.org.nz with Subject: unsubscribe
