Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 28-7-2011 9:24, michael.vancann...@wisa.be wrote:
On Thu, 28 Jul 2011, Reinier Olislagers wrote:
On 27-7-2011 20:52, michael.vancann...@wisa.be wrote:
Code like
ftWideString: //fixed length or at least max length string
begin
TDOMElement(ColumnNode).SetAttribute('fieldtype', 'string');
TDOMElement(ColumnNode).SetAttribute('WIDTH',
UTF8Decode(string(ExportFields.Fields[ItemCounter].Field.Size)));
end;
Regarding the snippet above: for Unicode-enabled Delphis, using
Clientdataset savetofile as UTF8-based XML (dfXMLUTF8), what do I need
to output for a ftString:
- Field.Size (number of characters in string)
- Field.DataSize (number of bytes in string field)
As Michael indicated, it probably is Field.Size, but I'd like to have
that confirmed. (Can't seem to find any documentation on ClientDataset
XML format on Embarcadero's Delphi help at
http://docwiki.embarcadero.com/VCL/en/DBClient.TCustomClientDataSet.SaveToFile)
Thanks,
Reinier
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On Mon, 15 Aug 2011, Reinier Olislagers wrote:
On 28-7-2011 9:24, michael.vancann...@wisa.be wrote:
On Thu, 28 Jul 2011, Reinier Olislagers wrote:
On 27-7-2011 20:52, michael.vancann...@wisa.be wrote:
Code like
ftWideString: //fixed length or at least max length string
begin
TDOMElement(ColumnNode).SetAttribute('fieldtype', 'string');
TDOMElement(ColumnNode).SetAttribute('WIDTH',
UTF8Decode(string(ExportFields.Fields[ItemCounter].Field.Size)));
end;
Regarding the snippet above: for Unicode-enabled Delphis, using
Clientdataset savetofile as UTF8-based XML (dfXMLUTF8), what do I need
to output for a ftString:
- Field.Size (number of characters in string)
- Field.DataSize (number of bytes in string field)
As Michael indicated, it probably is Field.Size, but I'd like to have
that confirmed. (Can't seem to find any documentation on ClientDataset
XML format on Embarcadero's Delphi help at
http://docwiki.embarcadero.com/VCL/en/DBClient.TCustomClientDataSet.SaveToFile)
I created a dataset with 2 fields. A ftString field with size 33, and a
ftWideString field with size 44.
Based on the following output from SaveToFile, I would say that you'll need to put
Size*SizeOf(CharTypeUsedInfield), which should be DataSize.
Michael.
?xml version=1.0 encoding=UTF-8 standalone=yes?
DATAPACKET Version=2.0
METADATA
FIELDS
FIELD attrname=stringveld fieldtype=string WIDTH=33/
FIELD attrname=WideStringVeld fieldtype=string.uni WIDTH=44/
/FIELDS
PARAMS/
/METADATA
ROWDATA
/ROWDATA
/DATAPACKET
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 15-8-2011 11:25, Michael Van Canneyt wrote: I created a dataset with 2 fields. A ftString field with size 33, and a ftWideString field with size 44. Based on the following output from SaveToFile, I would say that you'll need to put Size*SizeOf(CharTypeUsedInfield), which should be DataSize. Michael. ?xml version=1.0 encoding=UTF-8 standalone=yes? DATAPACKET Version=2.0 METADATA FIELDS FIELD attrname=stringveld fieldtype=string WIDTH=33/ FIELD attrname=WideStringVeld fieldtype=string.uni WIDTH=44/ /FIELDS PARAMS/ /METADATA ROWDATA /ROWDATA /DATAPACKET Thanks, Michael, Set to .DataSize... I'm ready for uploading this for inclusion in FPC, I can open a bugreport and attach the patch if you're happy with that. Fixes/changes since first commit to FPC === 1. Code cleanup to make it more modular and maintainable: split out code per export format and moved class-level variables there. 2. Delphi Clientdataset export now works thanks to FPC XML fixes by Sergei Gorelkin and tests/improvements by Ludo Brands. 3. Extended test coverage with all field types in a BufDataset so tests cover more. 4. Fixes for invalid data: Access, ADO.Net, Excel date limits respected (generate NULL or text fields respectively) 5. Decimalseparator setting in Access export should now work. 6. Variants are saved as blobs, not strings, because of their variable nature. 7. Fix for proper export of null values to Access NULL values (instead of empty/default values). 8. Changed description of class from Unicode XML file with XSD to Unicode XML file to better match unit function. 9. Compiled on Windows x86, Linux x64 without warnings using fpc -vweh fpxmlxsdexport.pp. Compared output. - ftVarBytes gave different output on Windows Linux. Disregarded as I don't fill these fields in the test because of bug 19930 10. Added option KeepFilesAfterTest = true in xmlxsdexporttestcase1; can be switched to false to delete export test files after use. 11. Lower-cased source code file names. 12. Fixes for Linux invalid date export to Excel by using clocale and specifying dateformat (restored at end of export). 13 .Datasize instead of .Size for string/widestring Delphi fields. Regards, Reinier ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 15-8-2011 11:44, Reinier Olislagers wrote: On 15-8-2011 11:25, Michael Van Canneyt wrote: I created a dataset with 2 fields. A ftString field with size 33, and a ftWideString field with size 44. Based on the following output from SaveToFile, I would say that you'll need to put Size*SizeOf(CharTypeUsedInfield), which should be DataSize. Michael. ?xml version=1.0 encoding=UTF-8 standalone=yes? DATAPACKET Version=2.0 METADATA FIELDS FIELD attrname=stringveld fieldtype=string WIDTH=33/ FIELD attrname=WideStringVeld fieldtype=string.uni WIDTH=44/ /FIELDS PARAMS/ /METADATA ROWDATA /ROWDATA /DATAPACKET Thanks, Michael, Set to .DataSize... I'm ready for uploading this for inclusion in FPC, I can open a bugreport and attach the patch if you're happy with that. Decided to upload it - I think it's a great improvement compared to the original code. Unfortunately that has more to do with my skills and other peoples' help, but well... better code is better code :) Uploaded as issue 19987. Thanks, Reinier ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On Mon, 15 Aug 2011, Reinier Olislagers wrote: On 15-8-2011 11:44, Reinier Olislagers wrote: On 15-8-2011 11:25, Michael Van Canneyt wrote: I created a dataset with 2 fields. A ftString field with size 33, and a ftWideString field with size 44. Based on the following output from SaveToFile, I would say that you'll need to put Size*SizeOf(CharTypeUsedInfield), which should be DataSize. Michael. ?xml version=1.0 encoding=UTF-8 standalone=yes? DATAPACKET Version=2.0 METADATA FIELDS FIELD attrname=stringveld fieldtype=string WIDTH=33/ FIELD attrname=WideStringVeld fieldtype=string.uni WIDTH=44/ /FIELDS PARAMS/ /METADATA ROWDATA /ROWDATA /DATAPACKET Thanks, Michael, Set to .DataSize... I'm ready for uploading this for inclusion in FPC, I can open a bugreport and attach the patch if you're happy with that. Decided to upload it - I think it's a great improvement compared to the original code. Unfortunately that has more to do with my skills and other peoples' help, but well... better code is better code :) Uploaded as issue 19987. I applied the patch, thank you very much for this work :-). But please: - Try to split patches in smaller chunks. It's better for bugfixing. If you break something it's harder to track what caused the break with these huge patches. - Please send diffs. - Don't change casing of files in between patches. (should be no problem if you send diffs, actually) Michael. ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 15-8-2011 12:33, michael.vancann...@wisa.be wrote: I applied the patch, thank you very much for this work :-). But please: - Try to split patches in smaller chunks. It's better for bugfixing. If you break something it's harder to track what caused the break with these huge patches. Ok - and I thought it would be less work for you guys to apply it if I submitted one big one ;) - Please send diffs. Ok. - Don't change casing of files in between patches. (should be no problem if you send diffs, actually) Mmm, tried to help there, by lowercasing everything... Thanks a lot, my itch has been scratched for now - back to developing the program that was actually meant to use this functionality. Of course, bug reports/improvement suggestions still welcome; I'll try and submit small patches for them ;) Regards, Reinier ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On Thu, 28 Jul 2011, Reinier Olislagers wrote:
On 27-7-2011 20:52, michael.vancann...@wisa.be wrote:
On Wed, 27 Jul 2011, Reinier Olislagers wrote:
Update: includes first stab at Delphi Clientdataset. Please test with
various datatypes as I haven't been able to map all of them...
I added your implementation to SVN.
Thanks, Michael.
I noticed your remark on a failing conversion of a longint to a string -
that code wouldn't have worked. Obviously a priority to increase the
test coverage.
But you may want to check your implementation on 64-bit prior to
submitting.
I will compile test on Linux x64 before submitting an upgrade
Code like
ftWideString: //fixed length or at least max length string
begin
TDOMElement(ColumnNode).SetAttribute('fieldtype', 'string');
TDOMElement(ColumnNode).SetAttribute('WIDTH',
UTF8Decode(string(ExportFields.Fields[ItemCounter].Field.Size)));
end;
Is dead wrong. It passes on 32-bit (but generating wrong data), but
fails to compile on 64-bit.
Hmmm, hadn't thought about size. (And size obviously matters here ;)
Just to make sure I understand correctly: in a string field, Field.Size
indicates the number of characters that can be stored in a field, while
Field.DataSize indicates the number of bytes, right?
Yes.
Complication is that e.g. Access expect character size for their size
specification. Delphi Clientdataset will probably require datasize.
I don't think so, but I'm not sure. I would have to test that.
Thanks, I'll look into it.
The compiler has a warning for this, so checking your warnings while
compiling for 32-bit may also be a good idea.
Michael.
New plans:
1. Expand test coverage: additional datatypes, fixed string width fully
filled with East Asian characters.
Use the output of this coverage as baseline to verify code commits still
deliver the proper export.
2. Incorporate fixes, e.g. those mentioned by Ludo Brands. Please keep
those reports coming ;)
3. Clean up code (needless class-level variables), look into possibility
of splitting entire procedures according to export format, instead of
multiple case...of statements inside a procedure body.
I would appreciate that one. I think with some refactoring, the code could
be made much simpler.
I'll keep a Mercurial repository on bitbucket to coordinate with others
and upload a new patch when done.
Let me know when you're ready for a new patch :-)
Michael.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: RE : [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 28-7-2011 6:34, Reinier Olislagers wrote: On 27-7-2011 20:49, Ludo Brands wrote: Some problems with Delphi Clientdataset: -Delphi has the dfXMLUTF8 and dfXML format for exporting data but no way to specify this for reading. The difference is in the first line: encoding=UTF-8. Was afraid of that. FCL XML has no support for specifying encoding. I might need to add a bug and a hackish workaround (if possible, opening the XML file we just created and rewriting the first line) Submitted in FPC bugtracker, issue 19827 ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
Update: includes first stab at Delphi Clientdataset. Please test with various datatypes as I haven't been able to map all of them... Hi list ( Michael Van Canneyt ;), Earlier I announced I was writing another XML export module for FCL-DB. After some interesting hours with Excel, Access, .Net and FreePascal, and thanks to Ludo Brands for testing and bugfixes, it seems the work is done. What is it: an FCL-DB export module that can export in various Unicode XML formats: Access/SQL Server, ADO.Net, Excel and Delphi ClientDataset. Suggestions for additional XML export formats are welcome. Is it tested: yep: export and import on Access XP, import by ADO.Net, import on Excel XP. Earlier versions of the code were tested with import on Access 2010. (Delphi Clientdataset needs to be tested though) The code comes with some simple tests (basically testing if the various export formats actually finish without error) and a readme. I think it is stable enough to include in FPC 2.5.1 for more wide scale use and testing. I've uploaded it to bug 19790: files: 20110727Final_XMLXSDExport_Code.zip: the code, patch for fpc makefile I've annotated the code with Lazarus todo entries to indicate possible improvements/tests. 20110727Final_XMLXSDExport_Tests.zip: the test code. Note: I haven't patched any makefiles, don't know if that's necessary Could an administrator remove the rest of the files as they are older, superfluous and will confuse users? Thanks. Hope it can be applied, of course I'm available for any necessary changes/improvements, Thanks for all the help, Reinier ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
RE : [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
Update: includes first stab at Delphi Clientdataset. Please
test with various datatypes as I haven't been able to map all
of them...
Some problems with Delphi Clientdataset:
-METADATA comes after ROWDATA and Delphi 2006 complains about missing
metadata. Inverting the 2 blocks solves the problem. Moved the METADATA
block from 2555 to 2462
- datetime and timestamp are missing time part as mentioned in the comments.
Formatdatetime('mmddThh:nn:ss', EF.Field.AsDateTime) solves that.
- bcd: invalid field type
- the mysql text types (tinytext,text, longtext, etc) are exported as
bin.hex with subtype binary. Perhaps a problem in the mysql code.
- when a string field contains the character, the row is not imported but
the previous one is duplicated.
-Delphi has the dfXMLUTF8 and dfXML format for exporting data but no way to
specify this for reading. The difference is in the first line:
encoding=UTF-8.
I'll investigate further and let you know by tomorrow.
Ludo
Ludo
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On Wed, 27 Jul 2011, Reinier Olislagers wrote:
Update: includes first stab at Delphi Clientdataset. Please test with
various datatypes as I haven't been able to map all of them...
I added your implementation to SVN.
But you may want to check your implementation on 64-bit prior to submitting.
Code like
ftWideString: //fixed length or at least max length string
begin
TDOMElement(ColumnNode).SetAttribute('fieldtype', 'string');
TDOMElement(ColumnNode).SetAttribute('WIDTH',
UTF8Decode(string(ExportFields.Fields[ItemCounter].Field.Size)));
end;
Is dead wrong. It passes on 32-bit (but generating wrong data),
but fails to compile on 64-bit.
The compiler has a warning for this, so checking your warnings while
compiling for 32-bit may also be a good idea.
Michael.
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: RE : [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 27-7-2011 20:49, Ludo Brands wrote:
Update: includes first stab at Delphi Clientdataset. Please
test with various datatypes as I haven't been able to map all
of them...
Some problems with Delphi Clientdataset:
-METADATA comes after ROWDATA and Delphi 2006 complains about missing
metadata. Inverting the 2 blocks solves the problem. Moved the METADATA
block from 2555 to 2462
I'll look into that.
- datetime and timestamp are missing time part as mentioned in the comments.
Formatdatetime('mmddThh:nn:ss', EF.Field.AsDateTime) solves that.
I'll look into that.
- bcd: invalid field type
Ok, what field type do we need ;) I'll wait for your further investigations.
- the mysql text types (tinytext,text, longtext, etc) are exported as
bin.hex with subtype binary. Perhaps a problem in the mysql code.
Ok.
Am looking at test coverage, might be a good idea for medium term to use
current sqldb testing framework so we can use eg Firebird employee
sample database or an mysql database.
For now, I'll wait for your further investigations.
- when a string field contains the character, the row is not imported but
the previous one is duplicated.
Strange, shouldn't this be handled by the XML export code?
-Delphi has the dfXMLUTF8 and dfXML format for exporting data but no way to
specify this for reading. The difference is in the first line:
encoding=UTF-8.
Was afraid of that. FCL XML has no support for specifying encoding. I
might need to add a bug and a hackish workaround (if possible, opening
the XML file we just created and rewriting the first line)
I'll investigate further and let you know by tomorrow.
Ludo
Hi Ludo,
https://bitbucket.org/reiniero/fpc_laz_patch_playground/issue/2/delphi-createdataset-issues
Thanks, looking forward to your results...
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] Patch: new XML export for FCL-DB ready for inclusion, issue 19790 - includes Delphi Clientdataset
On 27-7-2011 20:52, michael.vancann...@wisa.be wrote:
On Wed, 27 Jul 2011, Reinier Olislagers wrote:
Update: includes first stab at Delphi Clientdataset. Please test with
various datatypes as I haven't been able to map all of them...
I added your implementation to SVN.
Thanks, Michael.
I noticed your remark on a failing conversion of a longint to a string -
that code wouldn't have worked. Obviously a priority to increase the
test coverage.
But you may want to check your implementation on 64-bit prior to
submitting.
I will compile test on Linux x64 before submitting an upgrade
Code like
ftWideString: //fixed length or at least max length string
begin
TDOMElement(ColumnNode).SetAttribute('fieldtype', 'string');
TDOMElement(ColumnNode).SetAttribute('WIDTH',
UTF8Decode(string(ExportFields.Fields[ItemCounter].Field.Size)));
end;
Is dead wrong. It passes on 32-bit (but generating wrong data), but
fails to compile on 64-bit.
Hmmm, hadn't thought about size. (And size obviously matters here ;)
Just to make sure I understand correctly: in a string field, Field.Size
indicates the number of characters that can be stored in a field, while
Field.DataSize indicates the number of bytes, right?
Complication is that e.g. Access expect character size for their size
specification. Delphi Clientdataset will probably require datasize.
Thanks, I'll look into it.
The compiler has a warning for this, so checking your warnings while
compiling for 32-bit may also be a good idea.
Michael.
New plans:
1. Expand test coverage: additional datatypes, fixed string width fully
filled with East Asian characters.
Use the output of this coverage as baseline to verify code commits still
deliver the proper export.
2. Incorporate fixes, e.g. those mentioned by Ludo Brands. Please keep
those reports coming ;)
3. Clean up code (needless class-level variables), look into possibility
of splitting entire procedures according to export format, instead of
multiple case...of statements inside a procedure body.
I'll keep a Mercurial repository on bitbucket to coordinate with others
and upload a new patch when done.
Thanks,
Reinier
___
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal
