Is as_ops.c releasepage patch still needed?
The following patch was posted to the Reiser4 list August 3 by zam. Is it still needed? It solved many problems for me, making my systems able to actually complete full Beagle indexing. But I have not seen this patch show up in the last two mm kernel releases. Did something else fix it or is this patch still needed? Index: linux-2.6-git/fs/reiser4/as_ops.c === --- linux-2.6-git.orig/fs/reiser4/as_ops.c +++ linux-2.6-git/fs/reiser4/as_ops.c @@ -350,6 +350,11 @@ int reiser4_releasepage(struct page *pag if (PageDirty(page)) return 0; + /* extra page reference is used by reiser4 to protect +* jnode-page link from this -releasepage(). */ + if (page_count(page) 3) + return 0; + /* releasable() needs jnode lock, because it looks at the jnode fields * and we need jload_lock here to avoid races with jload(). */ spin_lock_jnode(node); -- Jonathan Briggs [EMAIL PROTECTED] eSoft, Inc. signature.asc Description: This is a digitally signed message part
Reiser4 stress test.
Hello, I've been having problems with Reiser 4 panicking for a few months, and I've recently had time to investigate the matter. I've created a program that can crash my system in a few minutes. It's based on kmail's disk activity and consists of small, separated writes to a file that is also mmapped. === scatteredwrites === #!/usr/bin/python import os import mmap import optparse parser = optparse.OptionParser(description= Creates a file in $CWD and performs a pattern of reads and writes to it in an attempt to trigger fs bugs. The file is broken up into regions: for each region the entire region is read, then some portion of it is written to. \nDistilled from kmail workload.) parser.add_option(--region-size, dest=regionsize, default=65536, type=int, help=Set region size to BYTES, metavar=BYTES) parser.add_option(--region-count, dest=regioncount, default=2048, type=int, help=Set number of regions to COUNT, metavar=COUNT) parser.add_option(--write-offset, dest=writeoffset, default=0, type=int, help=Offset write by BYTES in each region, metavar=BYTES) parser.add_option(--write-size, dest=writesize, default=256, type=int, help=Size of write in each region., metavar=BYTES) options, args = parser.parse_args() f = open(scatteredwrites.%d.tmp % (os.getpid()), w+b) try: writestr = A * options.regionsize for i in xrange(options.regioncount): f.write(writestr) f.close() f = open(scatteredwrites.%d.tmp % (os.getpid()), r+b) writestr = B * options.writesize dummy = mmap.mmap(f.fileno(), options.regionsize * options.regioncount, mmap.MAP_SHARED) while True: for i in xrange(options.regioncount): f.seek(i * options.regionsize, 0) f.read(options.regionsize) f.seek(- options.regionsize + options.writeoffset,1) f.write(writestr) except KeyboardInterrupt: os.unlink(scatteredwrites.%d.tmp % (os.getpid())) == Without fs load this stress test rarely causes problems. But with five instances running in parallel with five instances of a large grep (or patch, or tar), my computer crashes on a timescale of 10 minutes. I've also added a few patches to my kernel to help me debug the problems I've been having: diff -rupN a/fs/reiser4/page_cache.c b/fs/reiser4/page_cache.c --- a/fs/reiser4/page_cache.c 2006-08-19 19:45:57.0 -0400 +++ b/fs/reiser4/page_cache.c 2006-08-19 20:23:43.0 -0400 @@ -489,12 +489,9 @@ static int can_hit_entd(reiser4_context return 1; if (ctx-super != s) return 1; - if (get_super_private(s)-entd.tsk == current) - return 0; - if (!lock_stack_isclean(ctx-stack)) - return 0; - if (ctx-trans-atom != NULL) - return 0; + assert(ajw-1, get_super_private(s)-entd.tsk != current); + assert(ajw-2, lock_stack_isclean(ctx-stack)); + assert(ajw-3, ctx-trans-atom == NULL); return 1; } diff -rupN 2.6.18-rc4-mm1/fs/reiser4/debug.c linux/fs/reiser4/debug.c --- 2.6.18-rc4-mm1/fs/reiser4/debug.c 2006-08-18 19:21:13.0 -0400 +++ linux/fs/reiser4/debug.c2006-08-18 19:24:35.0 -0400 @@ -56,6 +56,9 @@ static char panic_buf[REISER4_PANIC_MSG_ */ static DEFINE_SPINLOCK(panic_guard); +static void print_lock_counters(const char *prefix, +const reiser4_lock_counters_info * info); + /* Your best friend. Call it on each occasion. This is called by fs/reiser4/debug.h:reiser4_panic(). */ void reiser4_do_panic(const char *format /* format string */ , ... /* rest */ ) @@ -74,6 +77,8 @@ void reiser4_do_panic(const char *format vsnprintf(panic_buf, sizeof(panic_buf), format, args); va_end(args); printk(KERN_EMERG reiser4 panicked cowardly: %s, panic_buf); + dump_stack(); + print_lock_counters(,reiser4_lock_counters()); spin_unlock(panic_guard); /* I've also added this bugfix by Alexander Zarochentsev [EMAIL PROTECTED]: Index: linux-2.6-git/fs/reiser4/as_ops.c === --- linux-2.6-git.orig/fs/reiser4/as_ops.c +++ linux-2.6-git/fs/reiser4/as_ops.c @@ -350,6 +350,11 @@ int reiser4_releasepage(struct page *pag if (PageDirty(page)) return 0; + /* extra page reference is used by reiser4 to protect +* jnode-page link from this -releasepage(). */ + if (page_count(page) 3) + return 0; + /* releasable() needs jnode lock, because it looks at the jnode fields * and we need jload_lock here to avoid races with jload(). */ spin_lock_jnode(node); Andrew Wade
problem with reiser3
hi all. I'm having problems checking a reiser3 filesystem. reiserfsck says: bread: Cannot read the block (27887610): (Input/output error). I passed badblocks over the partition and found 61 bad sectors (which I know is bad), then run reiserfsck -B with the bad blocks file and got the same, but also this: create_badblock_bitmap: block number (111550463) points out of fs size (39072080). any hints?
assertion failed: can_hit_entd(ctx, s)
This is the most common panic I've been getting. It looks like this: (2.6.18-rc4-mm1) reiser4 panicked cowardly: reiser4[scatteredwrites(4506)]: reiser4_writepage (fs/reiser4/page_cache.c:522)[]: assertion failed: can_hit_entd(ctx, s) Kernel panic - not syncing: reiser4[scatteredwrites(4506)]: reiser4_writepage (fs/reiser4/page_cache.c:522)[]: assertion failed: can_hit_entd(ctx, s) With the extra patches it looks like this: (2.6.18-rc4-mm2) reiser4 panicked cowardly: reiser4[grep(4918)]: can_hit_entd (fs/reiser4/page_cache.c:494)[ajw-3]: assertion failed: ctx-trans-atom == NULL [c0103870] dump_trace+0x64/0x1ad [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0ccf] reiser4_do_panic+0x4e/0x7b [c01e67b1] reiser4_writepage+0xab/0x1a8 [c013b973] shrink_inactive_list+0x37d/0x6f0 [c013bd94] shrink_zone+0xae/0xcc [c013c265] try_to_free_pages+0x139/0x20d [c0136f12] __alloc_pages+0x189/0x27d [c014c2ce] cache_alloc_refill+0x2d2/0x5a0 [c014bfc7] kmem_cache_alloc+0x70/0xa5 [c01eb68c] reiser4_alloc_inode+0x51/0xfa [c0163adc] alloc_inode+0x14/0x122 [c0164ad5] iget5_locked+0x3f/0x132 [c01f4091] reiser4_iget+0x8b/0x361 [c01fadd8] reiser4_lookup_common+0xef/0x151 [c015aef7] do_lookup+0xa0/0x13d [c015b72f] __link_path_walk+0x79b/0xbd4 [c015bbb6] link_path_walk+0x4e/0xc6 [c015c0e3] do_path_lookup+0x203/0x21d [c015c544] __path_lookup_intent_open+0x44/0x76 [c015c5d2] path_lookup_open+0x10/0x12 [c015c7c7] open_namei+0x61/0x570 [c014e72d] do_filp_open+0x1f/0x35 [c014e83e] do_sys_open+0x3f/0xba [c014e8e5] sys_open+0x16/0x18 [c01028cd] sysenter_past_esp+0x56/0x8d DWARF2 unwinder stuck at sysenter_past_esp+0x56/0x8d Leftover inexact backtrace: [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0ccf] reiser4_do_panic+0x4e/0x7b [c01e67b1] reiser4_writepage+0xab/0x1a8 [c013b973] shrink_inactive_list+0x37d/0x6f0 [c013bd94] shrink_zone+0xae/0xcc [c013c265] try_to_free_pages+0x139/0x20d [c0136f12] __alloc_pages+0x189/0x27d [c014c2ce] cache_alloc_refill+0x2d2/0x5a0 [c014bfc7] kmem_cache_alloc+0x70/0xa5 [c01eb68c] reiser4_alloc_inode+0x51/0xfa [c0163adc] alloc_inode+0x14/0x122 [c0164ad5] iget5_locked+0x3f/0x132 [c01f4091] reiser4_iget+0x8b/0x361 [c01fadd8] reiser4_lookup_common+0xef/0x151 [c015aef7] do_lookup+0xa0/0x13d [c015b72f] __link_path_walk+0x79b/0xbd4 [c015bbb6] link_path_walk+0x4e/0xc6 [c015c0e3] do_path_lookup+0x203/0x21d [c015c544] __path_lookup_intent_open+0x44/0x76 [c015c5d2] path_lookup_open+0x10/0x12 [c015c7c7] open_namei+0x61/0x570 [c014e72d] do_filp_open+0x1f/0x35 [c014e83e] do_sys_open+0x3f/0xba [c014e8e5] sys_open+0x16/0x18 [c01028cd] sysenter_past_esp+0x56/0x8d === : jnode: 0, tree: 0 (r:0,w:0), dk: 0 (r:0,w:0) jload: 0, txnh: 0, atom: 0, stack: 0, txnmgr: 0, ktxnmgrd: 0, fq: 0 inode: 0, cbk_cache: 0 (r:0,w0), eflush: 0, zlock: 0, spin: 0, long: 0 inode_sem: (r:0,w:0) d: 0, x: 0, t: 0 Kernel panic - not syncing: reiser4[grep(4918)]: can_hit_entd (fs/reiser4/page_cache.c:494)[ajw-3]: assertion failed: ctx-trans-atom == NULL reiser4 panicked cowardly: reiser4[scatteredwrites(4245)]: can_hit_entd (fs/reiser4/page_cache.c:494)[ajw-3]: assertion failed: ctx-trans-atom == NULL [c0103870] dump_trace+0x64/0x1ad [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0ccf] reiser4_do_panic+0x4e/0x7b [c01e67b1] reiser4_writepage+0xab/0x1a8 [c013b973] shrink_inactive_list+0x37d/0x6f0 [c013bd94] shrink_zone+0xae/0xcc [c013c265] try_to_free_pages+0x139/0x20d [c0136f12] __alloc_pages+0x189/0x27d [c01388a7] __do_page_cache_readahead+0xcc/0x1d2 [c0138f07] blockable_page_cache_readahead+0x51/0xd9 [c0139010] make_ahead_window+0x81/0xa4 [c013918a] page_cache_readahead+0x157/0x176 [c023aa82] reiser4_read_extent+0x374/0x6ab [c020511f] read_unix_file+0x5c7/0x762 [c014f1e2] vfs_read+0x88/0x134 [c014fa4e] sys_read+0x3b/0x60 [c01028cd] sysenter_past_esp+0x56/0x8d DWARF2 unwinder stuck at sysenter_past_esp+0x56/0x8d Leftover inexact backtrace: [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0ccf] reiser4_do_panic+0x4e/0x7b [c01e67b1] reiser4_writepage+0xab/0x1a8 [c013b973] shrink_inactive_list+0x37d/0x6f0 [c013bd94] shrink_zone+0xae/0xcc [c013c265] try_to_free_pages+0x139/0x20d [c0136f12] __alloc_pages+0x189/0x27d [c01388a7] __do_page_cache_readahead+0xcc/0x1d2 [c0138f07] blockable_page_cache_readahead+0x51/0xd9 [c0139010] make_ahead_window+0x81/0xa4 [c013918a] page_cache_readahead+0x157/0x176 [c023aa82] reiser4_read_extent+0x374/0x6ab [c020511f] read_unix_file+0x5c7/0x762 [c014f1e2] vfs_read+0x88/0x134 [c014fa4e] sys_read+0x3b/0x60 [c01028cd] sysenter_past_esp+0x56/0x8d === : jnode: 0, tree: 0 (r:0,w:0), dk: 0 (r:0,w:0) jload: 0, txnh: 0, atom: 0, stack: 0,
assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY)
This one hasn't recurred, so I don't have a stack trace. I haven't looked into it. (2.6.18-rc4-mm1) reiser4 panicked cowardly: reiser4[patch(9302)]: reiser4_set_page_dirty_internal (fs/reiser4/page_cache.c:475)[]: assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY) Kernel panic - not syncing: reiser4[patch(9302)]: reiser4_set_page_dirty_interna l (fs/reiser4/page_cache.c:475)[]: assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY) Andrew Wade
assertion failed: keyeq(znode_get_rd_key(node), znode_get_ld_key(node-right))
I looked at this one for a bit; I couldn't make any headway. I don't fully understand what the debugging code for the delimiting keys is doing. (2.6.18-rc4-mm1) reiser4 panicked cowardly: reiser4[ent:hdb1!(2167)]: sibling_list_remove (fs/reiser4/tree_walk.c:814)[zam-32245]: assertion failed: keyeq(znode_get_rd_key(node), znode_get_ld_key(node-right)) Kernel panic - not syncing: reiser4[ent:hdb1!(2167)]: sibling_list_remove (fs/reiser4/tree_walk.c:814)[zam-32245]: assertion failed: keyeq(znode_get_rd_key(node), znode_get_ld_key(node-right)) (2.6.18-rc4-mm1) reiser4 panicked cowardly: reiser4[ent:hdb1!(2175)]: sibling_list_remove (fs/reiser4/tree_walk.c:814)[zam-32245]: assertion failed: keyeq(znode_get_rd_key(node), znode_get_ld_key(node-right)) [c0103754] dump_trace+0x64/0x181 [c0103883] show_trace_log_lvl+0x12/0x25 [c0103b79] show_trace+0xd/0x10 [c0103b93] dump_stack+0x17/0x19 [c01a0663] reiser4_do_panic+0x4e/0x7b [c01ee6bd] sibling_list_remove+0x85/0x52e [c01ba97d] forget_znode+0x22b/0x33b [c01b76e0] longterm_unlock_znode+0x268/0x723 [c01da260] handle_pos_on_formatted+0x35c/0x45f [c01da3fc] handle_pos_on_leaf+0x4d/0x61 [c01d6a84] squalloc+0x16/0x52 [c01d89f7] jnode_flush+0x80e/0x99d [c01d8fee] flush_current_atom+0x468/0x722 [c01cf073] flush_some_atom+0x9c3/0xb13 [c01f4216] reiser4_writeout+0x1a6/0x30c [c01f554b] entd+0x1e2/0x3d5 [c0124545] kthread+0xaf/0xde [c03eb415] kernel_thread_helper+0x5/0xb DWARF2 unwinder stuck at kernel_thread_helper+0x5/0xb Leftover inexact backtrace: [c0103883] show_trace_log_lvl+0x12/0x25 [c0103b79] show_trace+0xd/0x10 [c0103b93] dump_stack+0x17/0x19 [c01a0663] reiser4_do_panic+0x4e/0x7b [c01ee6bd] sibling_list_remove+0x85/0x52e [c01ba97d] forget_znode+0x22b/0x33b [c01b76e0] longterm_unlock_znode+0x268/0x723 [c01da260] handle_pos_on_formatted+0x35c/0x45f [c01da3fc] handle_pos_on_leaf+0x4d/0x61 [c01d6a84] squalloc+0x16/0x52 [c01d89f7] jnode_flush+0x80e/0x99d [c01d8fee] flush_current_atom+0x468/0x722 [c01cf073] flush_some_atom+0x9c3/0xb13 [c01f4216] reiser4_writeout+0x1a6/0x30c [c01f554b] entd+0x1e2/0x3d5 [c0124545] kthread+0xaf/0xde [c03eb415] kernel_thread_helper+0x5/0xb === : jnode: 0, tree: 1 (r:0,w:1), dk: 1 (r:0,w:1) jload: 0, txnh: 0, atom: 0, stack: 0, txnmgr: 0, ktxnmgrd: 0, fq: 0 inode: 0, cbk_cache: 0 (r:0,w0), eflush: 0, zlock: 1, spin: 3, long: 1 inode_sem: (r:0,w:0) d: 1, x: 4, t: -1 Kernel panic - not syncing: reiser4[ent:hdb1!(2175)]: sibling_list_remove (fs/reiser4/tree_walk.c:814)[zam-32245]: assertion failed: keyeq(znode_get_rd_key(node), znode_get_ld_key(node-right)) Andrew Wade
Re: problem with reiser3
On Mon, Aug 21, 2006 at 07:24:21PM -0300, Marcos Dione wrote: hi all. I'm having problems checking a reiser3 filesystem. reiserfsck says: bread: Cannot read the block (27887610): (Input/output error). I passed badblocks over the partition and found 61 bad sectors (which I know is bad), then run reiserfsck -B with the bad blocks file and got the same, but also this: create_badblock_bitmap: block number (111550463) points out of fs size (39072080). any hints? forgot to mention: I tried to resize the fs to the partition size and got this: $ sudo resize_reiserfs /dev/sda1 resize_reiserfs 3.6.19 (2003 www.namesys.com) /dev/sda1 already is of the needed size. Nothing to be done
[nikita-1936] assertion failed: reiser4_no_counters_are_held()
This one has only occurred once. I looked fairly carefully at the code for partially converted files under the assumption that the rest was unlikely to be buggy, but nothing stood out at me. reiser4 panicked cowardly: reiser4[fixdep(19237)]: reiser4_done_context (fs/reiser4/context.c:181)[nikita-1936]: assertion failed: reiser4_no_counters_are_held() [c0103754] dump_trace+0x64/0x181 [c0103883] show_trace_log_lvl+0x12/0x25 [c0103b79] show_trace+0xd/0x10 [c0103b93] dump_stack+0x17/0x19 [c01a0663] reiser4_do_panic+0x4e/0x7b [c01bdbc0] reiser4_exit_context+0xa1/0x575 [c0202bc9] release_unix_file+0x1b7/0x1c2 [c014f90b] __fput+0xbe/0x16c [c014f9e7] fput+0x2e/0x33 [c014d3ec] filp_close+0x51/0x5b [c014ddd2] sys_close+0x70/0x93 [c01028a5] sysenter_past_esp+0x56/0x8d DWARF2 unwinder stuck at sysenter_past_esp+0x56/0x8d Leftover inexact backtrace: [c0103883] show_trace_log_lvl+0x12/0x25 [c0103b79] show_trace+0xd/0x10 [c0103b93] dump_stack+0x17/0x19 [c01a0663] reiser4_do_panic+0x4e/0x7b [c01bdbc0] reiser4_exit_context+0xa1/0x575 [c0202bc9] release_unix_file+0x1b7/0x1c2 [c014f90b] __fput+0xbe/0x16c [c014f9e7] fput+0x2e/0x33 [c014d3ec] filp_close+0x51/0x5b [c014ddd2] sys_close+0x70/0x93 [c01028a5] sysenter_past_esp+0x56/0x8d === : jnode: 0, tree: 0 (r:0,w:0), dk: 0 (r:0,w:0) jload: 0, txnh: 0, atom: 0, stack: 0, txnmgr: 0, ktxnmgrd: 0, fq: 0 inode: 0, cbk_cache: 0 (r:0,w0), eflush: 0, zlock: 0, spin: 0, long: 0 inode_sem: (r:0,w:0) d: 1, x: -2, t: -2 Kernel panic - not syncing: reiser4[fixdep(19237)]: reiser4_done_context (fs/reiser4/context.c:181)[nikita-1936]: assertion failed: reiser4_no_counters_are_held() I should be looking for an un-zrelse'd znode for this bug, correct? Andrew Wade
Re: problem with reiser3
Marcos Dione wrote: hi all. I'm having problems checking a reiser3 filesystem. reiserfsck says: bread: Cannot read the block (27887610): (Input/output error). I passed badblocks over the partition and found 61 bad sectors (which I know is bad), did you specify reiserfs block size (via option -b) when creating a list of bad blocks? then run reiserfsck -B with the bad blocks file and got the same, but also this: create_badblock_bitmap: block number (111550463) points out of fs size (39072080). any hints? it would be better to create a backup on a spare bigger partition using dd_rescue (pad not recoverable zones with zeroes), then run fsck on the created image.
Re: problem with reiser3
On Tue, Aug 22, 2006 at 03:38:40AM +0400, Edward Shishkin wrote: Marcos Dione wrote: hi all. I'm having problems checking a reiser3 filesystem. reiserfsck says: bread: Cannot read the block (27887610): (Input/output error). I passed badblocks over the partition and found 61 bad sectors (which I know is bad), did you specify reiserfs block size (via option -b) when creating a list of bad blocks? dunno which is the block size. tried to figure it out, but (from dmesg trying to mount the partition): ReiserFS: sda1: journal params: device sda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 and: $ sudo reiserfstune /dev/sda1 reiserfstune: Filesystem looks not cleanly umounted, check the consistency first. I assume it's the 8192 the kernel reports... it would be better to create a backup on a spare bigger partition using dd_rescue (pad not recoverable zones with zeroes), then run fsck on the created image. unluckly I can't. it's a 160 GiB partition and I don't have spare space.
reiserfs v3.6 problems
Hi,My root partition will not mount...strange problems have built up to this, like not being able to read files as root...and I get the following output frim debugreiserfs /dev/hdb2: debugreiserfs 3.6.19 (2003 www.namesys.com)Filesystem state: consistency is not checked after last mountingReiserfs super block in block 16 on 0x342 of format 3.6 with standard journal Count of blocks on the device: 6359728Number of bitmaps: 195Blocksize: 4096Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 6359728Root block: 0Filesystem is clean Tree height: 0Hash function used to sort names: r5Objectid map size 968, max 972Journal parameters: Device [0x0] Magic [0x482c3fa] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30Blocks reserved by journal: 0Fs state field: 0x3: FATAL corruptions exist. some corruptions exist. sb_version: 2inode generation number: 49892980UUID: 3eeff8ac-8c07-4a98-b126-7bcebd4480c1LABEL:Set flags in SB: ATTRIBUTES CLEANI have run reiserfsck --rebuild-sb and reiserfsck --rebuild-tree based on some forum/message boards regarding similar symptoms/issues, but still no good. Is there anyway I can recover? I'm not very experienced with reiserfs, so any help is very appreciated. Thanks!
Re: problem with reiser3
Marcos Dione wrote: it would be better to create a backup on a spare bigger partition using dd_rescue (pad not recoverable zones with zeroes), then run fsck on the created image. unluckly I can't. it's a 160 GiB partition and I don't have spare space. How much spare space do you have? You may be able to do some tricks with dm_snapshot...
Re: problem with reiser3
On Mon, Aug 21, 2006 at 08:23:30PM -0500, David Masover wrote: it would be better to create a backup on a spare bigger partition using dd_rescue (pad not recoverable zones with zeroes), then run fsck on the created image. unluckly I can't. it's a 160 GiB partition and I don't have spare space. How much spare space do you have? You may be able to do some tricks with dm_snapshot... right now, I have 45 MiB of space in my spare disk. I *could* (should?) make more space, but can't guarrantee anythung.
Re: Reiser4 stress test.
Thanks Andrew, please be patient and persistent with us at this time, as one programmer is on vacation, and the other is only able to work a few hours a day due to an illness. Hans
