Re: [fpc-devel] Implementing TFmtBCDField - ftFmtBCD and SQLite
Simply said: So Sqlite doesn't support real BCD values. Yes, SQLite does not support them native So we can't support it either for Sqlite. It was my question. We can (if we want) partialy add work-arounds: DECIMAL(x,y) if y=0 then map to ftLargeInt (64bit integer) elseif y=4 then map to ftBCD (as it is now) else map to ftFloat (IMHO better than raising exception) More difficult: Sqlite is not a 'normal' sql-database. It has all kind of strange issues, it's own definitions and options. It is impossible to support all these gimmicks in sqldb. I agree, and most horrible is, that in one column you can have any datatype in different rows (in 1st row character data, in 2nd integer and in 3th real numbers and in 4th blob) Sqlite only has very few data-types (the 'affinities') but to be gentle to our users we do support more, only based on the field-type name. Yes and this is IMO good. If user defines column datatype as numeric or integer or float it can be expected, that also data in that column will be in that format So as long as users can't make tables with numeric fields that support more then 15 numbers, we don't have to support bcd-fields. So map decimal to BCD (not float, it will be invalid for values within the supported precision range). If users want to store numbers with an higher precision, that is simply impossible in sqlite. Yes it is impossible in native way ... but we can help him and do necessary conversion in the background ... question is what user expect, when defines column like DECIMAL(15,7) ? If they want to store their numbers in blob-fields, they can simply define a blob field and do so... Yes but BLOB is mapped to TBlobField and there are no methods like AsFloat, AsBCD, AsFMTBCD ... so user must handle all this in their code ... retrieve value convert them etc. , use TField.OnGetText etc. ... it is doable, but not very confort. So where's the problem? (I see the problem of Sqlite not supporting bcd-fields, Yes it is primary problem but the sqldb-problem? What to do in case when user defines column NUMERIC, DECIAML (x,y) where y4 or x15 ? (I do not like raise exception if there is way how to handle this situation ... and also user IMHO does not expect exception) As I wrote we should at least do: DECIMAL(x,y) if y=0 --- ftLargeInt elseif y=4 --- ftBCD else --- ftFloat (or ftString with Size=x+1 ?) But we can go far (this is relative simple to implement and gives maximum transparency and easy to use): if x 15 or y4 --- ftFmtBCD (and here write as BLOB into SQLite3 database in background) elseif y=0 --- ftLargeInt else --- ftBCD Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] The future of fpmake
Joost van der Sluis schrieb: That's why I mentioned or any other structured format. It's not a bad idea to see if we can make a structured (and readable) format to create our fpmake.pp files. I was also thinking about letting fpmake detect from the compiler-output if a dependency is missing, and give a warning if this happens, so it can be added. Naive question: is this all about dependency determination? If so, I'd attack the problem from the other end: an efficient scan of the searchpath (both source and compiled), from which the locations of all later required ppu and source files can be determined easily. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] The future of fpmake
Marco van de Voort schrieb: In a new approach I'd provide the interpretation of existing MakeFiles, and extend it to the specific needs and capabilities of the FPC/Lazarus project (package...) model. Personally I don't think the core makefile principles are worth preserving at all: ACK. A build system based on the FPC/Lazarus project management, and Pascal uses, can operate on a higher level (comparable to automake source files, instead of final makefiles). 1. they are mostly generated anyway. The .fpc files are the info, not the makefiles themselves. [that's the mentioned automake level] 2. they totally pass-by certain core FPC principles (like being to able to compile multiple files in one go. In the packages this is worked around using buildunits) 3. For e.g. a multithreaded compiler, the number of files to be handed to the compiler should increase, not decrease (*) (*) there are multiple issues here to fix: - compiler is not mutithreaded. After my tries I'm not sure whether parallel compilation will really speed up compilation. In most cases all but one modules are in suspended state, while their uses clauses are evaluated by the compiler. When a (Lazarus) project uses packages, then all required packages can be compiled in parallel. This doesn't require an multi-threaded compiler, when every package can be compiled by a new compiler process. - module system is scheduled for rewrite for about a decade now Does there exist a roadmap or wishlist, of the intended changes? - packaging (also static, iow DCP) could decrease number of files and speed up compiler. But what about backend tools? Packages must be recompiled as a whole, if only one contained unit must be recompiled. Disk I/O (file cache) usage could be reduced by a map file with the options of all compiled units. But when used units must be loaded anyhow, even if they deserve no recompilation, the benefit of such an added map is questionable. Okay, it could be given a try, in a renewed module system. WRT time consuming jobs, observed e.g. in building FPC itself, I'd integrate cleaning of the output directories, together with means to reduce the number of such directories, as far as possible. I'd also add FPC as an integrated module, with a reusable file cache to prevent excessive directory scans and loading of configs and modules. This only works when integrated with FPC, as the compiler finds its own files. But that is all far, far away. The point is more that the current transition horse makeshift situation (fpmake) should be adapatable in the future. I dunno about platform restrictions, but a shared file pool looks feasable to me. Perhaps fpmake (or Lazarus) can hold a module map or entire modules in an MMF, shareable by all compiler instances or processes? At least currently its data is fairly abstract (though I'm not entirely happy having to specify fileinfo manually. That should be based on compiler feedback. But this is very difficult in practice since the exact buildprocess is OS dependant) One big step of fpmake is the killing of tools outside the FPC project like make. ACK! The compiler module can be used in the Lazarus IDE as well, either as a built-in module or as a shared library, with beforementioned ability to share the module cache with the application. Unfortunately parallel compilation will never become available with FPC, so that other chances for parallel processing should be explored. As long as the build tasks are based on input/output files (disk I/O), a shared file cache looks like the most promising way for speeding up an build process. Such plans are on the decade scale. If you are interested in it, I'd start creating as many internal assemblers and linkers as possible and further the module-rewrite, and the organizing of units into larger concepts (packages or whatever) and reducing the proliferation of files. My experience with (im)porting tools is, that only the interface should be modified, from reading commandline-arguments to a straight forward configuration of the internal variables. The worker code should stay as is, so that bugfixes and other improvements deserve not more than a recompilation of the updated sources. Have e.g. a look on Abbrevia vs. 7zip. Abbrevia never comes to an end with even the maintenance of a few archive formats, while 7zip (as a shell) can be extended easily by only adding an interface to every new compression module. The same concept could be applied to fpc, so that it can be integrated tightly into fpmake - at least for non-cross-platform builds. When fpmake consists of dedicated modules for various tasks, these modules can be used to build other applications, like test suite runners or profile editors. I think the core bit of fpmake/fppkg should be (1) killing off external tools, and the compromises to package metadata that they force upon up. (2) start developing a logical
Re: [fpc-devel] Implementing TFmtBCDField - ftFmtBCD and SQLite
On Fri, 2011-04-01 at 11:13 +0200, LacaK wrote: Simply said: So Sqlite doesn't support real BCD values. Yes, SQLite does not support them native Then it's up to the 'user' (in our case the user is a programmer) to implement workarounds. If we add some work-around, nobody will know that that work-around exist. And other sqlite tools will not be able to handle it, and the 'user' will be puzzled. So we can't support it either for Sqlite. It was my question. We can (if we want) partialy add work-arounds: DECIMAL(x,y) if y=0 then map to ftLargeInt (64bit integer) elseif y=4 then map to ftBCD (as it is now) else map to ftFloat (IMHO better than raising exception) No, map to ftfmtbcd, as it should. That will work fine as long as the values are within the sqlite-range. More difficult: Sqlite is not a 'normal' sql-database. It has all kind of strange issues, it's own definitions and options. It is impossible to support all these gimmicks in sqldb. I agree, and most horrible is, that in one column you can have any datatype in different rows (in 1st row character data, in 2nd integer and in 3th real numbers and in 4th blob) Sqlite only has very few data-types (the 'affinities') but to be gentle to our users we do support more, only based on the field-type name. Yes and this is IMO good. If user defines column datatype as numeric or integer or float it can be expected, that also data in that column will be in that format What we could do is add a possibility for users to override the field-type definition. Add an event or something which is called to determine the field-type. But make this generic, so it works for all datasets. That could be a very strong feature. So as long as users can't make tables with numeric fields that support more then 15 numbers, we don't have to support bcd-fields. So map decimal to BCD (not float, it will be invalid for values within the supported precision range). If users want to store numbers with an higher precision, that is simply impossible in sqlite. Yes it is impossible in native way ... but we can help him and do necessary conversion in the background ... question is what user expect, when defines column like DECIMAL(15,7) ? What other db-tools do when you use sqlite: work as long as you don't really require the specified precision. If they want to store their numbers in blob-fields, they can simply define a blob field and do so... Yes but BLOB is mapped to TBlobField and there are no methods like AsFloat, AsBCD, AsFMTBCD ... so user must handle all this in their code ... retrieve value convert them etc. , use TField.OnGetText etc. ... it is doable, but not very confort. Yes, but it's a sqlite limitation. If users choose to use a tool that is not suited for their goal, it's not the task of sqldb to mask that, imho. So where's the problem? (I see the problem of Sqlite not supporting bcd-fields, Yes it is primary problem but the sqldb-problem? What to do in case when user defines column NUMERIC, DECIAML (x,y) where y4 or x15 ? (I do not like raise exception if there is way how to handle this situation ... and also user IMHO does not expect exception) (Well, show me the bug: as it is now, he won't get an exception? Just map to ftmbcd) the user will only get an invalid value when he stores a value outside the sqlite-precision scope. Exactly what he would expect when he uses sqlite. As I wrote we should at least do: DECIMAL(x,y) if y=0 --- ftLargeInt elseif y=4 --- ftBCD else --- ftFloat (or ftString with Size=x+1 ?) But we can go far (this is relative simple to implement and gives maximum transparency and easy to use): if x 15 or y4 --- ftFmtBCD (and here write as BLOB into SQLite3 database in background) elseif y=0 --- ftLargeInt else --- ftBCD No, but if you need whit work-around in your own projects, implement a general event to override this behavior. I think that's best. Joost. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Implementing TFmtBCDField - ftFmtBCD and SQLite
No, map to ftfmtbcd, as it should. That will work fine as long as the values are within the sqlite-range. ok. in reading phase no problem (ATM we read using sqlite3_column_text (so SQLite converts all storage classes (integer,real, blob) to string) and then converting to TBCD ... ok (Well, show me the bug: as it is now, he won't get an exception? Just map to ftmbcd) yes now no exception it is ok as is the user will only get an invalid value when he stores a value outside the sqlite-precision scope. Exactly what he would expect when he uses sqlite. Yes. I can live with this, but do not forget, that ATM there is not implemented writting of ftFmtBCD parameters in Procedure TSQLite3Cursor.bindparams(AParams : TParams); So writting never occurs! So there we must add at least ftFmtBCD case, (behind ftbcd) But If we add this code (excuse me, I must offer it) ;-): //it is only pseudo code ftFmtBCD: begin if P.AsFMTBCD.Precision 15 then //we are out-of-range begin str1:=BCDTOStr(P.AsFMTBCD, SQLFormatSettings); checkerror(sqlite3_bind_blob(fstatement,I,pcharstr(str1), length(str1),@freebindstring)); end else begin do1:=P.AsFloat; checkerror(sqlite3_bind_double(fstatement,I,do1)); end; end; // all. Then all cases will be covered and no additional work around are needed. Laco. P.S.: P.AsFMTBCD: http://bugs.freepascal.org/view.php?id=18809 ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Re: Implementing TFmtBCDField - ftFmtBCD and SQLite
Let me please once more explain what I meant (I am not sure id I explained it enought)1.) When we decalre column like DECIMAL or NUMERIC then this columns will have numeric affinity2.) Columns with numeric affinity can hold only INTEGER, REAL or BLOB data (in theory they can hold also TEXT data, but such text data can not be a valid numbers, only generic character data, which can not be translated to any number, for example 'abc')3.) if we ommit BLOBs then we do do not need ftBCD, nor ftFmtBCD, because in SQLite database can be stored only INTEGRE (ftLargeInt) or REAL (ftFloat) values. 4.) so when we read such data, SQLite translates their internal representation (storage class) into requested type (sqlite3_column_double, int, text, blob)5.) so only reason to introduce ftFmtBCD is to expect, that stored data are in BLOB storage class, so this is only storage class, which allows numeric values with precision 15 (X'12345678923456789.123456789')6.) so when we are ready to read BLOBs then we would be ready also write BLOBs (when necessary to save precision)I do not know if I explained it sufficient, what I mean ?Laco. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
[fpc-devel] Changes in Makefile-system
Hi all, I've just committed a change in the Makefiles for the fcl-web package. The Makefile now calls fpmake to build and install the package. It's a first test, so please try and report any problems you have when building the fcl-web package. (Or all packages, or just a complete fpc-checkout) I'll try to fix them as soon as possible. If you are really stuck, the old Makefile.fpc is copied to Makefile.org. Copy the Makefile.org to Makefile.fpc, and run 'fpcmake -TAll' in the fcl-web directory. And don't forget to report your problems. Joost. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Implementing TFmtBCDField - ftFmtBCD and SQLite
LacaK schrieb: So as long as users can't make tables with numeric fields that support more then 15 numbers, we don't have to support bcd-fields. So map decimal to BCD (not float, it will be invalid for values within the supported precision range). If users want to store numbers with an higher precision, that is simply impossible in sqlite. Yes it is impossible in native way ... but we can help him and do necessary conversion in the background ... question is what user expect, when defines column like DECIMAL(15,7) ? He expects to get/put values of that type, i.e. TFmtBCDField if I understand the discussion right. Question: is it assumed that the concrete DB is accessed only by FPC apps? If not, the user is lost. So let's assume that the value can be stored in any format. Now you're free to use either a BLOB or an fixed-length string with the ASCII representation of the number (ToString). The string representation has the advantage that the format can be determined from the stored data, making the whole thing somewhat type-safe. The size overhead of such an encoding should be compared to the overhead of an blob, in order to find out whether the verbose storage is acceptable. If it wastes too much space, the digits may be encoded in some way, that e.g. maps 2 digits into 100 distinct character codes. The sign can be encoded in the first character, either as its ASCII value, or like a digit with '0' for positive and everything else for negative values. Such an encoding can be used for every DB, that doesn't support BCD numbers natively. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
