Re: [fpc-devel] TFieldDef.Size vs TField.Size
One way is: TField.Size := (TFieldDef.Size div 2) if it's a WideString, and div 4 if unicode is enabled. Or the TDataset descendant has to correct the value for WideStrings when creating fields. (How?) Very simple/primitive Idea No1: add property DataSize: integer read GetDataSize write FDataSize; into TFieldDef class add next parameter ADataSize to Constructor TFieldDef.Create(AOwner: TFieldDefs; const AName: string; ADataType: TFieldType; ASize: Integer; ARequired: Boolean; AFieldNo: Longint ADataSize: Integer=0); which is used in every TSQLConnection descendant to create fielddefs So connectors can supply ASize=character length and also ADataSize=byte length , which they require to write/read character data into local buffer (each connector should be aware of this information ... i.e. encoding of character data which exchanges with DB engine) If not specified , then default =0 means use ASize+1 for ftString, (ASize+1)*2 for ftWideString ... for backward compatibility, for most others datatypes ignore it (because they are internaly stored in fixed length data structures). Each TSQLConnection has CharSet property and each DB engine supports limited number of connection charsets. So SQLConnection can read CharSet property and determine character width. For example if CharSet='UTF-8' then char_width:=3 (or 4) if CharSet='UCS-2' then char_width:=2 if CharSet='UTF-16' then char_width:=4; So with conjuction with Column size (ASize) can easy compute buffer size (ADataSize) or Constructor TFieldDef.Create(AOwner: TFieldDefs; const AName: string; ADataType: TFieldType; ASize: Integer; ARequired: Boolean; AFieldNo: Longint ACharEncoding: TCharEncoding); TCharEncoding=(cheUnknown, cheAnsi, cheUTF8, cheUTF16); and then ADataSize can be computed inside TFieldDef.Create and stored into FDataSize for later usage.(as I wrote before TFieldDef.GetDataSize will replace TCustomBufDataset.GetFieldSize) Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thursday 24 February 2011 08:02:20 LacaK wrote: So also here we can see, that FieldDef.Size is expected to be number of characters not bytes. So IMHO logical conclusion will be say, that TFieldDef.Size for string fields has same menaing as Field.Size, so it is number of characters (so documentation is wrong in this case ... also fast test in Delphi shows, that FieldDef.Size=Field.Size=number of characters) Agreed. In MSEgui tmsestringfield.size is the maximum allowed character count for the field. 0 = no limit. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. With some databases it is a little bit tricky to get the field size in characters instead in bytes. In case you don't know MSEgui: http://developer.berlios.de/projects/mseide-msegui/ Martin ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thu, 24 Feb 2011, LacaK wrote: Hi, I am writting here to discuss bug http://bugs.freepascal.org/view.php?id=17268 (I do not want reopen bug and writte there because I am not sure about my arguments) IMHO root of problem is in different definition of TFieldDef.Size and TField.Size Documentation says, that 1. TFieldDef.Size should be For string and byte fields, *Size* is the number of bytes reserved http://docwiki.embarcadero.com/VCL/en/DB.TFieldDef.Size 2. TField.Size should be the maximum number of characters in the string (for ftString and I guess that also for ftWideString) http://docwiki.embarcadero.com/VCL/en/DB.TField.Size http://docwiki.embarcadero.com/VCL/en/DB.TStringField.Size And now let's go look into fields.inc: 1. Function TFieldDef.CreateField copies Result.Size:=FSize (so TField.Size:=TFieldDef.Size) 2. function TStringField.GetDataSize returns for ftString ... Size+1 3. function TWideStringField.GetDataSize returns for ftWideString ... (Size + 1) * 2 Points 2,3 are clear Size is expected to be number of characters, DataSize number of bytes. But point 1 says, that TFieldDef.Size and TField.Size should be the same And now let's go look into bufdataset.pas: 4. function TCustomBufDataset.GetFieldSize works with TFieldDef and returns number of bytes needed in record buffer case FieldDef.DataType of ftString, ftGuid, ftFixedChar: result := FieldDef.Size + 1; ftFixedWideChar, ftWideString:result := (FieldDef.Size + 1)*2; ... So also here we can see, that FieldDef.Size is expected to be number of characters not bytes. So IMHO logical conclusion will be say, that TFieldDef.Size for string fields has same menaing as Field.Size, so it is number of characters Correct, if you assume 1 char = 1 byte. (so documentation is wrong in this case ... also fast test in Delphi shows, that FieldDef.Size=Field.Size=number of characters) Your story is confusing to me. Where do you see a problem ? Do you see an implementation problem or a documentation problem ? Note that nowhere in the documentation pages you mention, widestrings are mentioned (which is convenient, since that's where 1 char 1 byte). Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thu, 24 Feb 2011, Martin Schreiber wrote: On Thursday 24 February 2011 08:02:20 LacaK wrote: So also here we can see, that FieldDef.Size is expected to be number of characters not bytes. So IMHO logical conclusion will be say, that TFieldDef.Size for string fields has same menaing as Field.Size, so it is number of characters (so documentation is wrong in this case ... also fast test in Delphi shows, that FieldDef.Size=Field.Size=number of characters) Agreed. In MSEgui tmsestringfield.size is the maximum allowed character count for the field. 0 = no limit. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. But here you implicitly assume that you have a fixed number of bytes per character. You should always be explicit about such things, since this is a non-trivial assumption. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thursday 24 February 2011 09:49:51 michael.vancann...@wisa.be wrote: Agreed. In MSEgui tmsestringfield.size is the maximum allowed character count for the field. 0 = no limit. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. But here you implicitly assume that you have a fixed number of bytes per character. You should always be explicit about such things, since this is a non-trivial assumption. I don't understand. Martin ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thu, 24 Feb 2011, Martin Schreiber wrote: On Thursday 24 February 2011 09:49:51 michael.vancann...@wisa.be wrote: Agreed. In MSEgui tmsestringfield.size is the maximum allowed character count for the field. 0 = no limit. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. But here you implicitly assume that you have a fixed number of bytes per character. You should always be explicit about such things, since this is a non-trivial assumption. I don't understand. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. If you say fixed record layout, this means you assume that each character uses the same amount of bytes, and that the size of the string is limited, otherwise I fail to see how you can have fixed record layout. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thursday 24 February 2011 10:16:43 michael.vancann...@wisa.be wrote: But here you implicitly assume that you have a fixed number of bytes per character. You should always be explicit about such things, since this is a non-trivial assumption. I don't understand. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. If you say fixed record layout, this means you assume that each character uses the same amount of bytes, and that the size of the string is limited, otherwise I fail to see how you can have fixed record layout. I really should learn English, but I fear i am too old for the task... What I meant: Original tbufdataset uses a fixed record layout where for every stringfield memory is reserved to hold the maximum possible amount of bytes of the field. MSEgui tmsebufdataset has a fixed record layout too but stores a UnicodeString which actually is a pointer and uses sizeof(pointer) memory space in record layout. The string data fetched/posted from/to the database is converted from/to system encoding or utf-8 (can be selected with an option flag in the dataset or the connection component) to utf-16 while fetching/posting the data. Because MSEgui completely works with UnicodeString the users don't need to convert the DB-encoding. Martin ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
Hi, I am writting here to discuss bug http://bugs.freepascal.org/view.php?id=17268 (I do not want reopen bug and writte there because I am not sure about my arguments) IMHO root of problem is in different definition of TFieldDef.Size and TField.Size Documentation says, that 1. TFieldDef.Size should be For string and byte fields, *Size* is the number of bytes reserved http://docwiki.embarcadero.com/VCL/en/DB.TFieldDef.Size 2. TField.Size should be the maximum number of characters in the string (for ftString and I guess that also for ftWideString) http://docwiki.embarcadero.com/VCL/en/DB.TField.Size http://docwiki.embarcadero.com/VCL/en/DB.TStringField.Size And now let's go look into fields.inc: 1. Function TFieldDef.CreateField copies Result.Size:=FSize (so TField.Size:=TFieldDef.Size) 2. function TStringField.GetDataSize returns for ftString ... Size+1 3. function TWideStringField.GetDataSize returns for ftWideString ... (Size + 1) * 2 Points 2,3 are clear Size is expected to be number of characters, DataSize number of bytes. But point 1 says, that TFieldDef.Size and TField.Size should be the same And now let's go look into bufdataset.pas: 4. function TCustomBufDataset.GetFieldSize works with TFieldDef and returns number of bytes needed in record buffer case FieldDef.DataType of ftString, ftGuid, ftFixedChar: result := FieldDef.Size + 1; ftFixedWideChar, ftWideString:result := (FieldDef.Size + 1)*2; ... So also here we can see, that FieldDef.Size is expected to be number of characters not bytes. So IMHO logical conclusion will be say, that TFieldDef.Size for string fields has same menaing as Field.Size, so it is number of characters Correct, if you assume 1 char = 1 byte. (so documentation is wrong in this case ... also fast test in Delphi shows, that FieldDef.Size=Field.Size=number of characters) Your story is confusing to me. Where do you see a problem ? See please http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/packages/fcl-db/src/sqldb/odbc/odbcconn.pas?r1=16094r2=16988 And look at ftWideString, ftFixedWideChar cases What happens if we have NVARCHAR(20) column on MS SQL Server (uses UCS-2 1char=2bytes): 1. in AddFieldDefs method is created TFieldDef object with Size=20*2 (because FieldSize:=ColumnSize*sizeof(Widechar)) 2. later when TField objects are created in TFieldDef.CreateField method, TWideString object is created with Size=40 (because Size is copied from TFieldDef object) ... and this is wrong, because: 2.1 TWideString.Size should IMO return max.number of characters ... Size is used for example in data-aware controls to limit MaxLength 2.2 TWideStringField.GetDataSize will return 82, which is also wrong (because (40+1)*2) 2.3 in record buffer will be allocated also 82 bytes (because of TCustomBufDataset.GetFieldSize) So IMO correct will be in step 1 FieldSize:=ColumnSize not FieldSize:=ColumnSize*sizeof(WideChar) (like in my original fix in bug report) Note also, that Size is used (also in Delphi) to limit the number of characters in data-aware controls (for ftString, ftWideString), so it must be number of characters in both cases. Do you see an implementation problem or a documentation problem ? Implementation problem at first but also Documentation is confusing, because do not clearly says about menaing of TWideStringField.Size (only indirect guess can be done, that it is inherited from TStringField.Size, so it is number of characters) See, that also other users are confusing by it https://forums.embarcadero.com/thread.jspa?messageID=188792 Note that nowhere in the documentation pages you mention, widestrings are mentioned (which is convenient, since that's where 1 char 1 byte). Yes, but in Delphi6 help is for TWideStringField.DataSize: Inspect DataSize to determine the number of bytes required to store the fields value. Use DataSize to determine the size needed for a buffer when working with the GetData and SetData methods. For wide string fields, DataSize is the value of the Size property plus one (for the trailing NULL character), multiplied by two. So we can estimate, that Size is number of characters, while DataSize=(Size+1)*2 Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thu, 24 Feb 2011, Martin Schreiber wrote: On Thursday 24 February 2011 10:16:43 michael.vancann...@wisa.be wrote: But here you implicitly assume that you have a fixed number of bytes per character. You should always be explicit about such things, since this is a non-trivial assumption. I don't understand. tmsebufdataset stores string data as UnicodeString instead to use a fixed record layout. If you say fixed record layout, this means you assume that each character uses the same amount of bytes, and that the size of the string is limited, otherwise I fail to see how you can have fixed record layout. I really should learn English, but I fear i am too old for the task... What I meant: Original tbufdataset uses a fixed record layout where for every stringfield memory is reserved to hold the maximum possible amount of bytes of the field. MSEgui tmsebufdataset has a fixed record layout too but stores a UnicodeString which actually is a pointer and uses sizeof(pointer) memory space in record layout. The only drawback from this system is memory fragmentation for all the strings. This is the advantage of the TBufDataset. In one of the forms of our application, the users load up to 200.000 records. each containing at minimum 3-4 strings (don't ask why..., they just do) with your system, much more memory would be allocated. As for encoding: if ever the unicode string is finished, encoding issues can be dealt with in the DB components 'automatically'. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
Please, be patient. I'm working on it, as you can see in the bug reports and commits. ok, of course I did not know your plans, ideas, thoughts etc. As I said earlier, widestringfields aren't available in fpc yet. Someone wrote some code for it, but it is never properly tested and probably only worked by accident. I did not know that, I was thinking about TWideStringFields as a implemented, supported thing But I don't commit everything in one go, it's work-in-progress. And I need to write tests for everything that I commit. You mixed two things in one patch, so I had to split it into pieces. ok I did some tests with different Delphi versions, with and without unicode. And beside finding some bugs in XE's string-db code I think I know a solution for all problems regarding strings and unicode. But let me thing it out, write tests, write code and test if it works indeed. ok if you can inform about your plans, order of tasks etc. ... it will be welcomed if I can help you, let me know ... for example I have fix for TODBCConnection.SetParameters ... there is missing ftTime and ftWideString,ftWideFixedChar cases ... as revealed by testing with ODBC/MySQL IMHO root of problem is in different definition of TFieldDef.Size and TField.Size Documentation says, that 1. TFieldDef.Size should be For string and byte fields, Size is the number of bytes reserved [?]http://docwiki.embarcadero.com/VCL/en/DB.TFieldDef.Size I don't care what the documentation of others say. It is about how we implemented it and how we document things. ok I've read the documentation and did some tests. The documentation is simply wrong here. The behavior changed with different versions of Delphi, the documentation is never adapted. As I wrote in the bug-report, TFieldDef.Size should contain the number of bytes to hold the string, without the leading 0. why do you think so ? If we do not care about Delphi documentation, it seems to me more consistent, that TFieldDef.Size and TField.Size says about equal things (this is similar story as TField.AsBCD: TBCD and TParam.AsBCD: Currency ... same name, different meaning ... in my opinion something very strange) Note that with the unicode-versions of Delphi have in TFieldDef.Size the amount of characters * 4 in TFieldDef.Size. For WideStrings and normal strings. I'm thinking about adding a property to T(Buf)Dataset in which you can specify if you want to use unicode-length strings, or 'normal' lengths. So that we can add backwards-compatilibity, which Delphi doesn't. would not be better add such property to TSQLConnection class, as IMO all SQLQueries connected to one DB will work with same type of strings. So users can set it once and not for every TCustomSQLQuery object individualy ? 2. TField.Size should be the maximum number of characters in the string (for ftString and I guess that also for ftWideString) [?]http://docwiki.embarcadero.com/VCL/en/DB.TField.Size [?]http://docwiki.embarcadero.com/VCL/en/DB.TStringField.Size That's simply true. Also for widestrings. (tested) And now let's go look into fields.inc: 1. Function TFieldDef.CreateField copies Result.Size:=FSize (so TField.Size:=TFieldDef.Size) 2. function TStringField.GetDataSize returns for ftString ... Size+1 3. function TWideStringField.GetDataSize returns for ftWideString ... (Size + 1) * 2 Those functions have to be adapted, yes. I have the changes here but I need to fine-tune the tests, clean up etc. So also here we can see, that FieldDef.Size is expected to be number of characters not bytes. This was the case in the past. And it didn't lead to problems as widestrings were never used properly. (That's why no-one bothered how it was implemented for widestrings) So IMHO logical conclusion will be say, that TFieldDef.Size for string fields has same menaing as Field.Size, so it is number of characters (so documentation is wrong in this case ... also fast test in Delphi shows, that FieldDef.Size=Field.Size=number of characters) No, our implementation is wrong and we have to adapt our code. That way it is also possible to fix the unicode-problems you've reported. ps: if you want to know what the Delphi-XE-bug is. when using the TClientDataset, for widestrings, TFieldDef.Size is four times the amount of characters (ok), but when you get TWideSTringField.Datasize the answer is two times TfieldDef.Size +2. Yes same as in FPC GetDataSize:=(Size+1)*2 So if you have a wide-string of 40 characters. TFieldDef.Size is 160, but TField.Datasize 322. (obviously wrong) Yes, TField.Size = TFieldDef.Size, TField.DataSize = (TField.Size+1)*2 Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
On Thu, 2011-02-24 at 12:59 +0100, LacaK wrote: if you can inform about your plans, order of tasks etc. ... it will be welcomed The order.. well, that's difficult.. What comes first, comes first. :) if I can help you, let me know ... for example I have fix for TODBCConnection.SetParameters ... there is missing ftTime and ftWideString,ftWideFixedChar cases ... as revealed by testing with ODBC/MySQL I'm adding some ftTime tests. All widestring-issues are postponed until I found a way to deal with them, unicode, and have tests to test them. but now the TFmtBcd field is finally implemented, you can add support for that? Now it only works with sqlite, maybe you can add it for the other databases that you use? (odbc,mysql?) As I wrote in the bug-report, TFieldDef.Size should contain the number of bytes to hold the string, without the leading 0. why do you think so ? If we do not care about Delphi documentation, it seems to me more consistent, that TFieldDef.Size and TField.Size says about equal things It depends on how they are used. I think it is very usefull for a programmer to know the maximum amount of characters. (TField.Size) And a programmer should be able to detect how many bytes are reserverd to store it (TFieldDef.Size). This is especially important if we are supporting unicode-strings, where both values can differ. So when you are getting/setting data from the server, you use TFieldDef.Size, and then you want to check what the maximum amount of characters is, TField.Size. Note that with the unicode-versions of Delphi have in TFieldDef.Size the amount of characters * 4 in TFieldDef.Size. For WideStrings and normal strings. I'm thinking about adding a property to T(Buf)Dataset in which you can specify if you want to use unicode-length strings, or 'normal' lengths. So that we can add backwards-compatilibity, which Delphi doesn't. would not be better add such property to TSQLConnection class, as IMO all SQLQueries connected to one DB will work with same type of strings. So users can set it once and not for every TCustomSQLQuery object individualy ? Maybe. but I have to think about it better. Because if code in fields.inc depends on it, we have to add the property to TDataset itself. So if you have a wide-string of 40 characters. TFieldDef.Size is 160, but TField.Datasize 322. (obviously wrong) Yes, TField.Size = TFieldDef.Size, TField.DataSize = (TField.Size+1)*2 There are more ways to solve that. And we need to find how the Delphi datasets which do not have this problem do that. One way is: TField.Size := (TFieldDef.Size div 2) if it's a WideString, and div 4 if unicode is enabled. Or the TDataset descendant has to correct the value for WideStrings when creating fields. (How?) Joost. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] TFieldDef.Size vs TField.Size
I'm adding some ftTime tests. Did you noticed, that I already posted such tests http://bugs.freepascal.org/view.php?id=18763 ? Then let me know, I send you fix for ftTime for TODBCConnection ... but depends on http://bugs.freepascal.org/view.php?id=18773 All widestring-issues are postponed until I found a way to deal with them, unicode, and have tests to test them. ok but now the TFmtBcd field is finally implemented, please look at http://bugs.freepascal.org/view.php?id=18809 and IMO also this bugs must be solved first to continue with implementing TFmtBcdField into connectors http://bugs.freepascal.org/view.php?id=18388 and http://bugs.freepascal.org/view.php?id=18807 you can add support for that? Now it only works with sqlite, yes, but there is missing support for parameter ftFmtBCD in Procedure TSQLite3Cursor.bindparams maybe you can add it for the other databases that you use? (odbc,mysql?) yes i can do it, but would wait for fixing above mentioned bugs, to not touch code twice if something in fmtbcd unit changes As I wrote in the bug-report, TFieldDef.Size should contain the number of bytes to hold the string, without the leading 0. why do you think so ? If we do not care about Delphi documentation, it seems to me more consistent, that TFieldDef.Size and TField.Size says about equal things It depends on how they are used. I think it is very usefull for a programmer to know the maximum amount of characters. (TField.Size) yes And a programmer should be able to detect how many bytes are reserverd to store it (TFieldDef.Size). yes, but I would preffer introduce new TFieldDef.DataSize property, which will serve for byte length. So we will have TFieldDef.Size, TField.Size=character length and TFieldDef.DataSize, TField.DataSize=byte length Then we will be able use TFieldDef.DataSize instead of TCustomBufDataset.GetFieldSize (we can completely drop this) and all fields stuff will be in one unit fields.inc This is especially important if we are supporting unicode-strings, where both values can differ. yes So when you are getting/setting data from the server, you use TFieldDef.Size, and then you want to check what the maximum amount of characters is, TField.Size. yes, my only objection is, that same name is used for different things, which is confusing, non-intuitive and will/can lead to misunderstandings Note that with the unicode-versions of Delphi have in TFieldDef.Size the amount of characters * 4 in TFieldDef.Size. For WideStrings and normal strings. I'm thinking about adding a property to T(Buf)Dataset in which you can specify if you want to use unicode-length strings, or 'normal' lengths. So that we can add backwards-compatilibity, which Delphi doesn't. would not be better add such property to TSQLConnection class, as IMO all SQLQueries connected to one DB will work with same type of strings. So users can set it once and not for every TCustomSQLQuery object individualy ? Maybe. but I have to think about it better. Because if code in fields.inc depends on it, we have to add the property to TDataset itself. yes I also must think about it more So if you have a wide-string of 40 characters. TFieldDef.Size is 160, but TField.Datasize 322. (obviously wrong) Yes, TField.Size = TFieldDef.Size, TField.DataSize = (TField.Size+1)*2 There are more ways to solve that. And we need to find how the Delphi datasets which do not have this problem do that. One way is: TField.Size := (TFieldDef.Size div 2) if it's a WideString, and div 4 if unicode is enabled. Or the TDataset descendant has to correct the value for WideStrings when creating fields. (How?) hm, we must think about it more Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
