Re: [fossil-users] CR/NL warning in .pdf
On Sep 6, 2011, at 1:18 AM, Konstantin Khomoutov wrote: The problem is while PDF is considered to be a binary file (and it indeed usually contains compressed regions, it does contain ASCII header and footer (I think it's its PostScript heritage), so it can be considered to be a plain ASCII file by any tool which does not look for its special magic character sequence (in the first line of the header). Probably Fossil does not do that. That's a bit funny, because the pdf files contain a single \r\n ending not within a compressed region. Converting that to \n doesn't seem to break anything, but did I just kill a kitten? And the more important question: should we make fossil to treat all pdf files to be binary? Kind regards, Remigiusz Modrzejewski ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] CR/NL warning in .pdf
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/06/2011 11:33 AM, Remigiusz Modrzejewski wrote: On Sep 6, 2011, at 1:18 AM, Konstantin Khomoutov wrote: The problem is while PDF is considered to be a binary file (and it indeed usually contains compressed regions, it does contain ASCII header and footer (I think it's its PostScript heritage), so it can be considered to be a plain ASCII file by any tool which does not look for its special magic character sequence (in the first line of the header). Probably Fossil does not do that. That's a bit funny, because the pdf files contain a single \r\n ending not within a compressed region. Converting that to \n doesn't seem to break anything, but did I just kill a kitten? Yes. PDF files contain tables of contents that point to byte offsets within the file. Changing line endings like that removes a byte, which shifts everything. Some readers *may*, if a ToC entry points to something that doesn't look like what they expect, search about a bit to find the header. Some might not. Such PDF files are unreliable! And the more important question: should we make fossil to treat all pdf files to be binary? Yes. They are binary files. The fact that they contain lots of ASCII text is misleading - they can't be treated as text files. ABS - -- Alaric Snell-Pym http://www.snell-pym.org.uk/alaric/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5l96kACgkQRgz/WHNxCGpk/gCfcOkct8f1WdqYxbqnDtDTE1qB Fp0An2IjFOw5Yz5UvvTv1zt8PlCX3CHx =bIsi -END PGP SIGNATURE- ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Pushing the limits; SQL error: string or blob too big
I am continuing to test the limits of fossil and I've run into an
interesting case where I can't merge. I'd like to know how I could recover
from this. Note, this is from running the scripts (two in parallel) that I
put at :http://chiselapp.com/user/kiatoa/repository/testfossil with more and
larger files. The files are around 1-2 meg in size and the total data is 1.2
Gig over 70 files. One file (the problem file) is 1G in size, probably due
to a bug in my test scripts.
Note 1. Stats from the repository are at the bottom of this email.
Note 2. I do not necessarily consider this a concern over performance with
fossil but I do want to understand what it means.
Note 3. Delete or modify the offending file is a fine answer but I'm
interested in hearing other ideas and views.
chlr11723 fossil leaves
=== 2011-09-06 ===
16:45:17 [e343a05970] *CURRENT* Manually syncing (user: mrwellan tags:
trunk)
16:43:48 [700fc9b91b] Manually resyncing (user: mrwellan tags: trunk)
chlr11723 fossil merge 700fc9b91b
MERGE files/aa
MERGE files/ab
* 1 merge conflicts in files/ab
MERGE files/ac
* 1 merge conflicts in files/ac
MERGE files/ad
* 1 merge conflicts in files/ad
MERGE files/ae
MERGE files/af
MERGE files/ag
* 2 merge conflicts in files/ag
MERGE files/ah
MERGE files/ba
MERGE files/bb
MERGE files/bc
MERGE files/bd
fossil: error code 18: statement aborts at 9: [INSERT OR IGNORE INTO
undo(pathname,redoflag,existsflag,isExe,content)
VALUES('files/bd',0,1,0,:c)] string or blob too big
fossil: SQL error: string or blob too big
If you have recently updated your fossil executable, you might
need to run fossil all rebuild to bring the repository
schemas up to date.
Rolling back prior filesystem changes...
UNDO files/aa
UNDO files/ab
UNDO files/ac
UNDO files/ad
UNDO files/ae
UNDO files/af
UNDO files/ag
UNDO files/ah
UNDO files/ba
UNDO files/bb
UNDO files/bc
fossil: SQLITE_BUSY: statement aborts at 2: [ROLLBACK] cannot rollback
transaction - SQL statements in progress
chlr11723
stats=
Repository Size: 558905344 bytes Number Of Artifacts: 197747 (stored as 4863
full text and 192884 delta blobs) Uncompressed Artifact Size: 690049 bytes
average, 1070741214 bytes max, 136454435349 bytes total Compression Ratio:244:1
Number Of Check-ins: 119289 Number Of Files: 239 Number Of Wiki Pages: 0
Number Of Tickets: 0 Duration Of Project: 6 days or approximately 0.02 years
Project ID: 549128601262c7f7d8110788aee02608db3feb8b Server
ID:e73c5cb7f5bafb88ce4b9760ed635d36fc89b9b7
Fossil Version: 2011-05-12 14:56:52 [d8221b9863] (gcc-4.2.2) SQLite
Version:2011-04-27 19:54:44 [f55156c519] (3.7.6.1)
Database Stats: 545806 pages, 1024 bytes/page, 21776 free pages, UTF-8, wal
mode
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Pushing the limits; SQL error: string or blob too big
On Tue, Sep 6, 2011 at 7:42 PM, Matt Welland estifo...@gmail.com wrote:
I am continuing to test the limits of fossil and I've run into an
interesting case where I can't merge. I'd like to know how I could recover
from this. Note, this is from running the scripts (two in parallel) that I
put at :http://chiselapp.com/user/kiatoa/repository/testfossil with more
and larger files. The files are around 1-2 meg in size and the total data is
1.2 Gig over 70 files. One file (the problem file) is 1G in size, probably
due to a bug in my test scripts.
The error occurs when Fossil tries to save the old content of the file as a
BLOB in the UNDO table, so that it can undo the merge (restoring the
original content of the file) if you later ask it too. Apparently I have
the BLOB size limit set at 1GiB.
Note 1. Stats from the repository are at the bottom of this email.
Note 2. I do not necessarily consider this a concern over performance with
fossil but I do want to understand what it means.
Note 3. Delete or modify the offending file is a fine answer but I'm
interested in hearing other ideas and views.
chlr11723 fossil leaves
=== 2011-09-06 ===
16:45:17 [e343a05970] *CURRENT* Manually syncing (user: mrwellan tags:
trunk)
16:43:48 [700fc9b91b] Manually resyncing (user: mrwellan tags: trunk)
chlr11723 fossil merge 700fc9b91b
MERGE files/aa
MERGE files/ab
* 1 merge conflicts in files/ab
MERGE files/ac
* 1 merge conflicts in files/ac
MERGE files/ad
* 1 merge conflicts in files/ad
MERGE files/ae
MERGE files/af
MERGE files/ag
* 2 merge conflicts in files/ag
MERGE files/ah
MERGE files/ba
MERGE files/bb
MERGE files/bc
MERGE files/bd
fossil: error code 18: statement aborts at 9: [INSERT OR IGNORE INTO
undo(pathname,redoflag,existsflag,isExe,content)
VALUES('files/bd',0,1,0,:c)] string or blob too big
fossil: SQL error: string or blob too big
If you have recently updated your fossil executable, you might
need to run fossil all rebuild to bring the repository
schemas up to date.
Rolling back prior filesystem changes...
UNDO files/aa
UNDO files/ab
UNDO files/ac
UNDO files/ad
UNDO files/ae
UNDO files/af
UNDO files/ag
UNDO files/ah
UNDO files/ba
UNDO files/bb
UNDO files/bc
fossil: SQLITE_BUSY: statement aborts at 2: [ROLLBACK] cannot rollback
transaction - SQL statements in progress
chlr11723
stats=
Repository Size: 558905344 bytes Number Of Artifacts: 197747 (stored as
4863 full text and 192884 delta blobs) Uncompressed Artifact Size: 690049
bytes average, 1070741214 bytes max, 136454435349 bytes total
Compression Ratio: 244:1 Number Of Check-ins: 119289 Number Of Files: 239
Number Of Wiki Pages: 0 Number Of Tickets: 0 Duration Of Project: 6 days
or approximately 0.02 years Project
ID:549128601262c7f7d8110788aee02608db3feb8b
Server ID: e73c5cb7f5bafb88ce4b9760ed635d36fc89b9b7 Fossil Version:2011-05-12
14:56:52 [d8221b9863] (gcc-4.2.2)
SQLite Version: 2011-04-27 19:54:44 [f55156c519] (3.7.6.1) Database
Stats:545806 pages, 1024 bytes/page, 21776 free pages, UTF-8, wal mode
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
--
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] blob_zero, empty_blob and blob_reset
2011/9/5 Lluís Batlle i Rossell virik...@gmail.com Yesterday I achieved quite a working code, through the use of empty_blob (to overcome the assertion), changing some blob operations (solving some leaks), and freeing that allocated never freed. Hi, Lluís! A comment on your commit message: [ef8266b710] Leaf: Implementation of a linked list to solve the memory leak described in a TODO in e2ebb1f5cae8.This code is slower than having the memory leak, and at the end, it was not a big memory leak. Let's say, 10 byte per revision involved in a file annotate. If a file has 3 revisions, it may go around 300KB then.For this leak to be noticeable ... (user: viriketo, tags: label_linkedlist) We must not forget that 10 bytes for us clients doesn't count the internal bytes used by the allocator to track the allocation (i'm guessing 4-16 per allocation, depending on the platform, though there are special-case allocators with only 1 bit/alloc overhead), so a 300k leak is probably twice that size in reality. Since i'm so pedantic when it comes to having matched malloc/free() pairs, i personally really appreciate your efforts on this. Even if it has no outward effect on most users, i always sleep better knowing that the software i use is all clean and tidy on the inside. :) Happy Hacking! -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] blob_zero, empty_blob and blob_reset
On Tue, Sep 06, 2011 at 09:45:46PM +0200, Stephan Beal wrote: 2011/9/5 Lluís Batlle i Rossell virik...@gmail.com Yesterday I achieved quite a working code, through the use of empty_blob (to overcome the assertion), changing some blob operations (solving some leaks), and freeing that allocated never freed. Hi, Lluís! We must not forget that 10 bytes for us clients doesn't count the internal bytes used by the allocator to track the allocation (i'm guessing 4-16 per allocation, depending on the platform, though there are special-case allocators with only 1 bit/alloc overhead), so a 300k leak is probably twice that size in reality. Since i'm so pedantic when it comes to having matched malloc/free() pairs, i personally really appreciate your efforts on this. Even if it has no outward effect on most users, i always sleep better knowing that the software i use is all clean and tidy on the inside. :) Hello Stephan, thank you. I also like paired malloc/free, but there are some cases that the magic of fork() and exit() allows having a cleaner and more efficient code by simply not pairing malloc/free. In this case, I did not benchmark anything; and I started this work because a public fossil repository was hanging the linux serving it in a machine with 512MiB of RAM, due to the big leak in annotate. I had forgotten requiring anonymous login for the links, and a crawler memorised lots of accesses to [annotate]. :) And the OOM killer successed two or three times, then the kernel hanged. I suspect that the OOM killer does not cope well with processes writing to a tmpfs, but I don't know who to ask about that. If anyone has any idea here, I'd be glad to listen! Bye, Lluís ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] coding style guidelines: c89
Hi,
fossil currently does not build in a ANSI C-89 environment - a requirement of
the coding style guidelines ([1], point 16). There's an obvious build error in
http_ssl.c (C++ style comments), but there's something more in sha1.c...
When adding a -std=c89 to the generated Makefile with gcc as the compiler, I
end up getting (ignoring the C++ style comment at line 87)
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
./bld/sha1_.c:104: error: expected ')' before ':' token
(and many, many, many, many more lines like that). This seems to be because of
the calls to asm() ... I don't know how to fix that. FYI, after preprocessing,
the first instruction of line 104 looks like:
qq[4]+=((qq[1](qq[2]^qq[3]))^qq[3])+(block[0] = (({ unsigned
int y; asm(rorl %1,%0 : =r (y) : I (8), 0 (block[0]));
y; })0xFF00FF00) |(({ unsigned int y; asm(roll %1,%0 :
=r (y) : I (8), 0 (block[0])); y; })0x00FF00FF))+0x5A827999+({
unsigned int y; asm(roll %1,%0 : =r (y) : I (5), 0
(qq[0])); y; });
[1] http://fossil-scm.org/index.html/doc/trunk/www/style.wiki
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] blob_zero, empty_blob and blob_reset
2011/9/6 Lluís Batlle i Rossell virik...@gmail.com In this case, I did not benchmark anything; and I started this work because a public fossil repository was hanging the linux serving it in a machine with 512MiB of RAM, due to the big leak in annotate. I had forgotten requiring anonymous login for the links, and a crawler memorised lots of accesses to [annotate]. :) LOL! That is, in fact, the whole reason which the links are disabled by default (but i think that the ZIP downloads were the primary concern, as opposed to annotate). A related funny fossil experience: some months back a bot vandalized a repo of mine by replacing all my wiki pages except the home page (very clever!) and adding tons of tickets. That went on for a month or more before i noticed. It was my fault - the anonymous user had too much access - but the irony of it was that the bot got in via the auto-captcha button... and i was the one who originally added that button to fossil. Bitten in the ass via my own feature! -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] question about makemake.tcl and non-Unix platforms
Hi, all, i've added a couple files to my local json fork of fossil and i've got a question: the new files are in makemake.tcl, and they're built fine for Make-based builds on Unix, but is there anything special i need to be doing to get them added to the Windows/etc builds? -- - stephan beal http://wanderinghorse.net/home/stephan/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] coding style guidelines: c89
On Tue, Sep 06, 2011 at 04:21:43PM -0400, Martin S. Weber wrote: When adding a -std=c89 to the generated Makefile with gcc as the compiler, I end up getting (ignoring the C++ style comment at line 87) That was part of http://fossil-scm.org/index.html/info/f2ede7da6d70851 I don't get that commit -- both GCC 4.1 and 4.5 certainly do use the rotate instructions. As such I am strongly in favour of just reverting that change... Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] coding style guidelines: c89
On Tue, Sep 6, 2011 at 4:21 PM, Martin S. Weber martin.we...@nist.gov wrote:
(and many, many, many, many more lines like that). This seems to be because
of the calls to asm() ... I don't know how to fix that. FYI, after
preprocessing, the first instruction of line 104 looks like:
qq[4]+=((qq[1](qq[2]^qq[3]))^qq[3])+(block[0] = (({ unsigned
int y; asm(rorl %1,%0 : =r (y) : I (8), 0 (block[0]));
y; })0xFF00FF00) |(({ unsigned int y; asm(roll %1,%0 :
=r (y) : I (8), 0 (block[0])); y; })0x00FF00FF))+0x5A827999+({
unsigned int y; asm(roll %1,%0 : =r (y) : I (5), 0
(qq[0])); y; });
The asm() calls are being used to invoke the rotate instructions. C
does not have operators for peforming bitwise rotation of an operand
and emulating a rotate using shifts is messy and very ineficient:
unsigned int rotateLeft(unsigned int x, unsigned int n)
{
unsigned int t;
t = x (SIZEOFINT - n);
return ((x n) | t);
}
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] coding style guidelines: c89
On Sep 6, 2011, at 11:19 PM, Ron Wilson wrote:
The asm() calls are being used to invoke the rotate instructions. C
does not have operators for peforming bitwise rotation of an operand
and emulating a rotate using shifts is messy and very ineficient:
unsigned int rotateLeft(unsigned int x, unsigned int n)
{
unsigned int t;
t = x (SIZEOFINT - n);
return ((x n) | t);
}
FYI,
$ gcc -O2 -S rotate.c
$ cat rotate.s
…
_rotateLeft:
Leh_func_begin1:
pushq %rbp
Ltmp0:
movq%rsp, %rbp
Ltmp1:
movb%sil, %cl
movl%edi, %eax
roll%cl, %eax
popq%rbp
ret
Leh_func_end1:
..
$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658)
(LLVM build 2335.15.00)
(This is GCC with LLVM backend (default on OS X Lion)).
--
Dmitry Chestnykh
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] coding style guidelines: c89
I guess I'm too used to dealing with dumb compilers.
1. Does it work with the C89 restrictions turned on?
2. Does C89 support inline functions?
3. If not, will it handle the folowing correctly?
#define rotateLeft(x,n) { unsigned int t; t = x (SIZEOFINT -
n); x = (x n) | t; } /* bitwise rotate x by n bits left */
On Tue, Sep 6, 2011 at 5:47 PM, Dmitry Chestnykh
dmi...@codingrobots.com wrote:
FYI,
$ gcc -O2 -S rotate.c
$ cat rotate.s
…
_rotateLeft:
Leh_func_begin1:
pushq %rbp
Ltmp0:
movq %rsp, %rbp
Ltmp1:
movb %sil, %cl
movl %edi, %eax
roll %cl, %eax
popq %rbp
ret
Leh_func_end1:
..
$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658)
(LLVM build 2335.15.00)
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
