Re: Python to do CDC on XML files
Bruce Kirk wrote: > Does anyone know of any existing projects on how to generate a change data > capture on 2 very large xml files. > > The xml structures are the same, it is the data within the files that may > differ. > > I need to take a XML file from yesterday and compare it to the XML file > produced today and not which XML records have changed. > > I have done a google search and I am not able to find much on the subject > other than software vendors trying to sell me their products. :-) There is http://www.logilab.org/project/xmldiff As an alternative you may try to log the changes as they occur instead of inspecting the result. If the application generating the file is not under your control, does it offer other output formats, e. g. csv? Or if the xml file is basically a sequence of one type of node you may convert it to a database (sqlite will do) to match and compare the "records". -- https://mail.python.org/mailman/listinfo/python-list
Re: Python to do CDC on XML files
On Thu, Mar 24, 2016 at 10:57 AM, Bruce Kirkwrote: > I agree, the challenge is the volume of the data to compare is 13. Million > records. So it needs to be very fast 13M records is a good lot. To what extent can the data change? You may find it easiest to do some sort of conversion to text, throwing away any information that isn't "interesting", and then use the standard 'diff' utility to compare the text files. It's up to you to figure out what differences are "uninteresting"; it'll depend on your exact data. As long as you can do the conversion-to-text in a simple and straight-forward way, the overall operation will be reasonably fast. If this is a periodic thing (eg you're constantly checking today's file against yesterday's), saving the dumped text file will mean you generally need to just convert one file, halving your workload. This isn't a solution so much as a broad pointer... hope it's at least a start! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python to do CDC on XML files
I agree, the challenge is the volume of the data to compare is 13. Million records. So it needs to be very fast Sent from my iPad > On Mar 23, 2016, at 4:47 PM, Bob Gailerwrote: > > > On Mar 23, 2016 4:20 PM, "Bruce Kirk" wrote: > > > > Does anyone know of any existing projects on how to generate a change data > > capture on 2 very large xml files. > > > > The xml structures are the same, it is the data within the files that may > > differ. > > > It should not be too difficult to write a program that locates the tags > delimiting each record, then compare them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python to do CDC on XML files
On Mar 23, 2016 4:20 PM, "Bruce Kirk"wrote: > > Does anyone know of any existing projects on how to generate a change data capture on 2 very large xml files. > > The xml structures are the same, it is the data within the files that may differ. > It should not be too difficult to write a program that locates the tags delimiting each record, then compare them. -- https://mail.python.org/mailman/listinfo/python-list
Python to do CDC on XML files
Does anyone know of any existing projects on how to generate a change data capture on 2 very large xml files. The xml structures are the same, it is the data within the files that may differ. I need to take a XML file from yesterday and compare it to the XML file produced today and not which XML records have changed. I have done a google search and I am not able to find much on the subject other than software vendors trying to sell me their products. :-) Regards -- https://mail.python.org/mailman/listinfo/python-list