Re: [Harbour] Re: OT: file size
On Sun, 17 Jan 2010, David Arturo Macias Corona wrote: Hi, > Surprisely, except for source code, there are not reference of > DB_DBFLOCK_VFP, "locking scheme" in doc files and even changelog > file These modification and many others RDD related I made few years ago when I was working on xHarbour CVS code and later they were ported to Harbour without detail replication of all ChangeLog notes. In this week I'll try to make some cleanups in existing locking schemes and then I'll create their description which I'll commit to SVN. best regards, Przemek ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi,
> Viktor Szakáts wrote:
>> so all that needs to be done is fall back to return failure where cairo
>> functions don't exist in the cairo version hbcairo is build against.
>
> I found only 4 functions that needs to be fixed. See, pdf.c and png.c.
>
> I still has a question that to do with this:
>
> HB_FUNC( CAIRO_SURFACE_WRITE_TO_PNG )
> {
> #ifdef CAIRO_HAS_PNG_FUNCTIONS
> cairo_surface_t * pSurface = hb_cairo_surface_param( 1 );
> if( pSurface )
> hb_retni( cairo_surface_write_to_png( pSurface, hb_parc( 2 ) ) );
> #else
> hb_cairo_surface_param( 1 ); /* Parameter validation */
> hb_ret();/* There is no good CAIRO_STATUS_* */ #endif
> }
>
> This function should return error code. 0 means "no error". There is no good
> CAIRO_STATUS_* constant to indicate "no supported" (complete
> http://www.cairographics.org/manual/cairo-error-status.html#cairo-status-t ).
> What action do you prefer return NIL, return -1, return 999 or some number
> larger than existing cairo errors. Something else...
Definitely an integer, I'd vote for -1.
Brgds,
Viktor
___
Harbour mailing list (attachment size limit: 40KB)
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi,
Viktor Szakáts wrote:
so all that needs
to be done is fall back to return failure where cairo functions
don't exist in the cairo version hbcairo is build against.
I found only 4 functions that needs to be fixed. See, pdf.c and png.c.
I still has a question that to do with this:
HB_FUNC( CAIRO_SURFACE_WRITE_TO_PNG )
{
#ifdef CAIRO_HAS_PNG_FUNCTIONS
cairo_surface_t * pSurface = hb_cairo_surface_param( 1 );
if( pSurface )
hb_retni( cairo_surface_write_to_png( pSurface, hb_parc( 2 ) ) );
#else
hb_cairo_surface_param( 1 ); /* Parameter validation */
hb_ret();/* There is no good CAIRO_STATUS_* */
#endif
}
This function should return error code. 0 means "no error". There is no
good CAIRO_STATUS_* constant to indicate "no supported" (complete
http://www.cairographics.org/manual/cairo-error-status.html#cairo-status-t
). What action do you prefer return NIL, return -1, return 999 or some
number larger than existing cairo errors. Something else...
Regards,
Mindaugas
___
Harbour mailing list (attachment size limit: 40KB)
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
> Viktor Szakáts wrote: >> Current hbcairo code uses RTE _right on creation_, which is IMO not ideal >> solution, as it's more difficult to handle it on .prg level. >> So as a general advice, IMO RTE should only be thrown if wrong parameter was >> passed to a function, so code snippet like this one can be made sure to not >> RTE because of problems downstream in calls. >> IF ! Empty( p := MYOBJECT_CREATE( "hello" ) ) >>MYOBJECT_DO_SOMETHING_USEFUL( p, 100 ) >> ELSE >>/* Either the low-level call failed,or low-level call was not >> available. >> Handle it gracefully. */ >> ENDIF >> In current implementation whole above construct needs to be put inside BEGIN >> SEQUENCE to prepare for MYOBJECT_CREATE() failures. > > > Do you have any sample code that shows the above hbcairo behavior? Nope sorry, after second review I couldn't find exactly this case in hbcairo, which means it looks fine. (CAIRO_PATH_ITERATOR_CREATE() looked wrong at first glance) Parameter checks and RTEs seem all in place, so all that needs to be done is fall back to return failure where cairo functions don't exist in the cairo version hbcairo is build against. Brgds, Viktor ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi, Viktor Szakáts wrote: Current hbcairo code uses RTE _right on creation_, which is IMO not ideal solution, as it's more difficult to handle it on .prg level. So as a general advice, IMO RTE should only be thrown if wrong parameter was passed to a function, so code snippet like this one can be made sure to not RTE because of problems downstream in calls. IF ! Empty( p := MYOBJECT_CREATE( "hello" ) ) MYOBJECT_DO_SOMETHING_USEFUL( p, 100 ) ELSE /* Either the low-level call failed, or low-level call was not available. Handle it gracefully. */ ENDIF In current implementation whole above construct needs to be put inside BEGIN SEQUENCE to prepare for MYOBJECT_CREATE() failures. Do you have any sample code that shows the above hbcairo behavior? Regards, Mindaugas ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
RE: [Harbour] Re: OT: file size
David/Przemek, Thanks for discussing this, if even in passing. I too, like David, could not find documentation as to locking schemes either, and therefore read "DB_DBFLOCK_XHB64" as being specific to a 64 bit executable - and as such never even tried it. Upon trying it, however, my problems disappeared on large data file manipulations - no more problems in either the write or read portions of my executables/application. Thanks for jogging my assumptions, and again, great release. F -Original Message- From: harbour-boun...@harbour-project.org [mailto:harbour-boun...@harbour-project.org] On Behalf Of David Arturo Macias Corona Sent: Sunday, January 17, 2010 7:40 AM To: harbour@harbour-project.org Subject: [Harbour] Re: OT: file size Przemek: This is becoming more interesting :-) First thanks for your help >In the past I was answering for such questions over 10 times. >You may find these messages in the Harbour and xHarbour mailing >list archives. A short description I added also to tests/xhb-diff.txt I read xhb-diff.txt and checked it About "over 10 times": I have seen before your info about "locking schemes" and so on, but in that cases was not relevant because does not exist a direct need and we usually do not go deep in those messages if we are not involved For example, nobody have response for my recent messages about hbcurl, hbcairo, hbqt because are considered as irrelevant in this moment Surprisely, except for source code, there are not reference of DB_DBFLOCK_VFP, "locking scheme" in doc files and even changelog file My mail file for Harbour messages reached 105 Mb file size and no one editor in OS/2 can open it ( EPM, E, TEDIT, JEdit ) :-) So I used many hours to split file in many files until it can open under 50 Mb: harbour0 21 Mb, 2006 to 2007 harbour08 26 Mb, 2008 harbour09a 31 Mb, 2009 first six months harbour27 Mb, 2009 rest six months as we can see, 2009 was plenty of messages ... and surprisely too, in messages there are not too much info about dbf file sizes, "locking schemes". Last and detailed message was around 30 Oct 2009 In source code we find info as this: /* LOCK SCHEMES */ #define DB_DBFLOCK_DEFAULT0 #define DB_DBFLOCK_CLIP 1 #define DB_DBFLOCK_CL53 2 #define DB_DBFLOCK_VFP3 #define DB_DBFLOCK_CL53EXT4 #define DB_DBFLOCK_XHB64 5 /* DBF locking schemes */ #define DBF_LOCKPOS_CLIP 10UL #define DBF_LOCKPOS_CL53 10UL #define DBF_LOCKPOS_VFP 0x4000UL #define DBF_LOCKPOS_VFPX 0x7ffeUL #define DBF_LOCKPOS_CL53EXT 40UL #define DBF_LOCKPOS_XHB64 HB_LL( 0x7FFF0001 ) which are difficult to understand for a user without too much knowledge For example, what is CL53EXT, VFPX ? For me was hard to find information about this case Perhaps info is there somewhere but I do not found it, so I suggest you to expand xhb-diff.txt or create a different one to clarify: - DBF file size limit ( currently brief ) - "locking schemes" so users can be oriented to file info in place to repeat in messages >I do not know ADS details so I do not it's limitations. >For sure it does not support all Harbour extenssions so they are >smaller and for sure there is nothing in ADS what exceed Harbour >any limits for DBFs and related files (memos and indexes) so >migration to ADS without changing the format to ADT+ADM+ADI can >only reduce the maximum supported file size or in some cases leave >them on the same level. If someone claims sth different then he >simply does not know what he's talking about and you can ignore him. >If you need bigger files then you have to switch to different format. >Each program which uses OS files is limited by OS API. The exact >limitations can be found in the OS documentation and they depends >on the operating system internals (OS kernel) and used file system >(FS). >Additionally please remember that locking schemes can seriously reduce >maximum file size. On some systems it's not possible to read from >locked area. In such case you have to switch all your applications >accessing the files to locking scheme which do not cause such >limitations, i.e. use Harbour 64bit locking. [...] >They use non POSIX system i.e. DOS or WINDOWS with VFP locking scheme >in which the RLOCK and FLOCK are is at range 0x6E6B27FF : 0x7FFE. >In such systems read/write access from locked region is blocked by >OS if client is not lock owner so the above error appear when other >station [R|F]locked some records in DBF file and user program wants >to read records which are located in the above range. That's all. >It's expected behavior. They should read about locking schemes and >switch to other locking scheme which do not cause such limitation. [...] >And this is result of file corruption, i.e. due to wrong CPs or server/ >client computer crash and FS corruption. I h
Re: [Harbour] Re: OT: file size
Hi, > Viktor Szakáts wrote: >> It seems that OS/2 cairo version has no 'CAIRO_HAS_IMAGE_SURFACE' support, >> and this makes test app break. The correct fix here is to provide Harbour >> level function regardless of cairo version, but return permanent error in >> this case. This is the method used in all other Harbour lib bindings. >> I hope Mindaugas can fix it. > > Let's say cairo function should return surface (GC pointer type). What is > expected behavior of such function, if does not exist in current cairo > version? hb_ret()? EG_UNSUPPORTED? etc IMO it's best if it returns failure just as if the underlying low-level function had failed, which means returning the same type as in normal cases, but HB_FALSE, NULL or other distinctive value. This is hb_retptr( NULL ) for low-level object creation functions. If hb_retptr( NULL ) is returned, all other functions expecting this value should also handle it, by throwing an RTE in param checking phase. Current hbcairo code uses RTE _right on creation_, which is IMO not ideal solution, as it's more difficult to handle it on .prg level. So as a general advice, IMO RTE should only be thrown if wrong parameter was passed to a function, so code snippet like this one can be made sure to not RTE because of problems downstream in calls. IF ! Empty( p := MYOBJECT_CREATE( "hello" ) ) MYOBJECT_DO_SOMETHING_USEFUL( p, 100 ) ELSE /* Either the low-level call failed, or low-level call was not available. Handle it gracefully. */ ENDIF In current implementation whole above construct needs to be put inside BEGIN SEQUENCE to prepare for MYOBJECT_CREATE() failures. Brgds, Viktor ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi, Viktor Szakáts wrote: It seems that OS/2 cairo version has no 'CAIRO_HAS_IMAGE_SURFACE' support, and this makes test app break. The correct fix here is to provide Harbour level function regardless of cairo version, but return permanent error in this case. This is the method used in all other Harbour lib bindings. I hope Mindaugas can fix it. Let's say cairo function should return surface (GC pointer type). What is expected behavior of such function, if does not exist in current cairo version? hb_ret()? EG_UNSUPPORTED? etc Regards, Mindaugas ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
> >Easily possible. The errors you sent are
> >all reported in system headers and libcurl
> >headers. We can't fix those in Harbour.
>
> So is not due to case/otherwise flow for compilers in Harbour
No, seemingly it's incompatibility between libcurl
headers and OS/2 ow headers.
Maybe there exist some #defines for libcurl which
are not automatically activated, but can nevertheless
resolve the problem, the headers should be checked
to tell more.
watcom is notoriously badly supported in projects,
so it's a pain to maintain also in Harbour.
> >Ideally this problem should be reported to
> >libcurl developers.
>
> >Unfortunately there is no way to disable
> >dependency detection for platform/compiler
> >combinations (os2/watcom) in Harbour, so OS/2
> >users will have to make sure not to set
> >HB_WITH_CURL when building for watcom.
>
> You mean not to try build hbcurl ?
Yes. Detection is not fully automatic anyway,
so all that needs to be done is to not configure
it for this target.
> >> - Describe list, done in readme* file
>
> >Lib dependencies are never documented in INSTALL,
> >instead, they are configured in appropriate .hbc files.
> >In INSTALL, we can document _package_ dependencies
> >and packages web links, but never actual lib filenames.
>
> I did it in readme_contrib.os2/readme.os2 and not in INSTALL, which is not
> proper place for this kind of info
> I did not it in *.h* files because syntax is unknown by me
> For example:
> {!mingw}libs=libcurl
> {mingw}libs=curldll
> seem to be mingw or not mingw, but maybe we need "not mingw and os2", I am
> not sure
Yes, each line should be guarded appropriately.
There are dozens of example for this in Harbour
SVN, plus probably the help can help and the
source is there with all supported filter
keywords as a last resort. Usually it's enough
to filter for compilers, compiler groups and
platforms. CPU archs in some rare cases.
> And I am not sure if my lib lists work fine in other OS/2 environments
That's the task to solve.
> As you are expert in *.h* files, you can make proper changes based on my lib
> lists for each contrib of readme_contrib.os2, included in previous message
> [Harbour] Re: Harbour 2.0.0 contrib libs for OS/2-eCS
> Tue Jan 12 22:55:19 EST 2010
That's exactly that I don't want to do.
It's guesswork since I can't test it.
Moreover Harbour won't scale to well if
I have to deal with every detail of every
existing platform. .hbc files are pretty
simple beasts, pls check them, and if you
have questions or patch proposal, I'll
try to help.
> >> - Perhaps this code does not work on OS/2 - Qt
>
> >I can't comment on those. Only that the bool conversion
> >problem could probably be easily fixed (unless it's an
> >OS/2 compiler bug), for someone good in C++.
>
> And C or C++ are not for me :-)
> > - Are valid errors which must be fixed on Harbour
> > - Other compiler switchs required
> > - They are errors related to OS/2-Qt code and not Harbour
>
> Perhaps Pritpal or someone who know hbqt code can review these errors
> If they do not belong to hbqt, I can report them to os2-qt developers
Not much feedback so far, so in such case the
problem falls back to the hand of those who feel
the itch the most. If this is you in this case,
you can try to work it around locally and come
to the list if you've found something. It can
surely help the case.
Or, we can wait until someone fixes it.
Brgds,
Viktor
___
Harbour mailing list (attachment size limit: 40KB)
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi David,
>> I did notice it and thanks for these tests, but I'd suggest
>> to patch (or send patches for) existing .hbc files, after you
>> tested them with hbmk2 successfully using OS/2 specific 3rd
>> party lib names. It's rather inefficient if I edit them without
>> testing and we iterate it endlessly. Plus for me it's impossible
>> to decide which one of the possible true OS/2 lib name setups
>> should be put into SVN .hbc files. Ideally the default here
>> should be what _most_ OS/2 users would expect.
>
>> You should add:
>> {os2}libs=...
>
>> Plus you should exclude existing libs= lines with
>> {!os2} if they are not currently protected.
>
> Messages were for specific cases, not only libs list
>
> a) hbcurl
> * Fail to build with OpenWatcom
> * undefined symbols in os2gcc
> Alternatives:
> - Fix source code
> - or, perhaps hbcurl does not work in OpenWatcom
Easily possible. The errors you sent are
all reported in system headers and libcurl
headers. We can't fix those in Harbour.
Ideally this problem should be reported to
libcurl developers.
Unfortunately there is no way to disable
dependency detection for platform/compiler
combinations (os2/watcom) in Harbour, so OS/2
users will have to make sure not to set
HB_WITH_CURL when building for watcom.
> b) hbcairo
> Missing function in one sample
> Alternatives:
> - Fix source code
> - or, perhaps sample fail in any platform
It seems that OS/2 cairo version has no
'CAIRO_HAS_IMAGE_SURFACE' support, and this
makes test app break. The correct fix here is to
provide Harbour level function regardless of
cairo version, but return permanent error in
this case. This is the method used in all other
Harbour lib bindings.
I hope Mindaugas can fix it.
> c) Contrib libs readme*
> Libs list required for each contrib based on my tests
> Alternatives:
> - Modify *.h* files by myself
> - Describe list, done in readme* file
Lib dependencies are never documented in INSTALL,
instead, they are configured in appropriate .hbc files.
In INSTALL, we can document _package_ dependencies
and packages web links, but never actual lib filenames.
> d) hbqt
> List of source files which have errors on OS/2 and detailed flow of tests
> Alternatives:
> - Fix source code
> - Perhaps this code does not work on OS/2 - Qt
I can't comment on those. Only that the bool conversion
problem could probably be easily fixed (unless it's an
OS/2 compiler bug), for someone good in C++.
>
> But Pritpal is highly focused in hbide now :-)
>
> Most errors are of kind:
> 'A' is not a member of 'B'
> class 'C' has no member named 'D'
> 'E' was not declared in this scope
> expected primary-expression before ')' token
> invalid use of incomplete type 'F'
> incomplete type 'G' used in nested name specifier
> forward declaration of 'H'
> expected type-specifier before 'I'
>
> I do not know if:
> - Are valid errors which must be fixed on Harbour
> - Other compiler switchs required
> - They are errors related to OS/2-Qt code and not Harbour
>
>>> My mail file for Harbour messages reached 105 Mb file size and no one
>>> editor in OS/2 can open it ( EPM, E, TEDIT, JEdit ) :-)
>
>> (if it's a text file you can use 'grep')
>
> But grep does not show context around each found
According to its own help, it does:
---
Context control:
-B, --before-context=NUM print NUM lines of leading context
-A, --after-context=NUM print NUM lines of trailing context
-C, --context=NUM print NUM lines of output context
-NUM same as --context=NUM
--color[=WHEN],
--colour[=WHEN] use markers to highlight the matching strings;
WHEN is `always', `never', or `auto'
---
Brgds,
Viktor
___
Harbour mailing list (attachment size limit: 40KB)
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
Hi David,
> I read xhb-diff.txt and checked it
> About "over 10 times":
> I have seen before your info about "locking schemes" and so on, but in that
> cases was not relevant because does not exist a direct need and we usually do
> not go deep in those messages if we are not involved
> For example, nobody have response for my recent messages about hbcurl,
> hbcairo, hbqt because are considered as irrelevant in this moment
I did notice it and thanks for these tests, but I'd suggest
to patch (or send patches for) existing .hbc files, after you
tested them with hbmk2 successfully using OS/2 specific 3rd
party lib names. It's rather inefficient if I edit them without
testing and we iterate it endlessly. Plus for me it's impossible
to decide which one of the possible true OS/2 lib name setups
should be put into SVN .hbc files. Ideally the default here
should be what _most_ OS/2 users would expect.
You should add:
{os2}libs=...
Plus you should exclude existing libs= lines with
{!os2} if they are not currently protected.
> Surprisely, except for source code, there are not reference of
> DB_DBFLOCK_VFP, "locking scheme" in doc files and even changelog file
>
> My mail file for Harbour messages reached 105 Mb file size and no one editor
> in OS/2 can open it ( EPM, E, TEDIT, JEdit ) :-)
> So I used many hours to split file in many files until it can open under 50
> Mb:
> harbour0 21 Mb, 2006 to 2007
> harbour08 26 Mb, 2008
> harbour09a 31 Mb, 2009 first six months
> harbour27 Mb, 2009 rest six months
(if it's a text file you can use 'grep')
Brgds,
Viktor
___
Harbour mailing list (attachment size limit: 40KB)
Harbour@harbour-project.org
http://lists.harbour-project.org/mailman/listinfo/harbour
Re: [Harbour] Re: OT: file size
On Sat, 16 Jan 2010, David Arturo Macias Corona wrote: > >I have not found a clear answer > >Everyone have his own impression :-) > >What is file size limit in DBF format ? > >and in DBF-(x)Harbour ? > >and in DBF-ADS 6.2x ? In the past I was answering for such questions over 10 times. You may find these messages in the Harbour and xHarbour mailing list archives. A short description I added also to tests/xhb-diff.txt I do not know ADS details so I do not it's limitations. For sure it does not support all Harbour extenssions so they are smaller and for sure there is nothing in ADS what exceed Harbour any limits for DBFs and related files (memos and indexes) so migration to ADS without changing the format to ADT+ADM+ADI can only reduce the maximum supported file size or in some cases leave them on the same level. If someone claims sth different then he simply does not know what he's talking about and you can ignore him. If you need bigger files then you have to switch to different format. Each program which uses OS files is limited by OS API. The exact limitations can be found in the OS documentation and they depends on the operating system internals (OS kernel) and used file system (FS). Additionally please remember that locking schemes can seriously reduce maximum file size. On some systems it's not possible to read from locked area. In such case you have to switch all your applications accessing the files to locking scheme which do not cause such limitations, i.e. use Harbour 64bit locking. [...] > As we have: > 1) DBF files larger than 2 Gb do not fail > 2) Record pointer movement does not fail in these files, including DBSEEK() > 3) APPEND FROM / record pointer movement does not fail Right conclusion. > 4) Users confirm these kind of errors happen crossing 2 Gb, > for example Fred Seyffert: > > I'm having the continuation of a long term problem with large data file > (>2gb) read errors. Mostly I get 1010's from standard errorblock trapping, > but sometimes get They use non POSIX system i.e. DOS or WINDOWS with VFP locking scheme in which the RLOCK and FLOCK are is at range 0x6E6B27FF : 0x7FFE. In such systems read/write access from locked region is blocked by OS if client is not lock owner so the above error appear when other station [R|F]locked some records in DBF file and user program wants to read records which are located in the above range. That's all. It's expected behavior. They should read about locking schemes and switch to other locking scheme which do not cause such limitation. > > Application Internal Error -app.exe > Terminated at: 2010.01.15 12:39:02 > Unrecoverable error 9201: hb_cdxPageSeekKey: wrong parent key. > Called from DBSEEK(0) > [...] And this is result of file corruption, i.e. due to wrong CPs or server/ client computer crash and FS corruption. > Perhaps some kind of memory corruption, limits in some functions ( > DBSEEK(), DBGOTO(), ... ) dealing with values greater than 2**31 > under certain circunstances ? Compiler issues ? I've never heard about such limits in code I created. All limits are result of used file structures and external limitations like used OS or FS. Harbour does not introduce any own limits. Anyhow if you will find sth then I'm very interesting. For sure it will be fixed immediately. best regards, Przemek ___ Harbour mailing list (attachment size limit: 40KB) Harbour@harbour-project.org http://lists.harbour-project.org/mailman/listinfo/harbour
