Hi Laurent,
> As I like to do something useful when I learn a new language, I
> decided to do a backup system.
OK.
> As you can see, given the progression, it will soon take more time
> to store the meta-data than processing the data itself.
> Does anybody see if I made a mistake in my code?
Yes, the main problem is this +Joint:
> (class +File +Entity)
> ...
> (rel backups (+List +Joint) files (+Backup)) #
>
> (class +Backup +Entity)
> ...
> (rel files (+List +Joint) backups (+File)) #
It causes each time a +File is created to be the list of files
in the +Backup object to be extended:
> (de addFile (Bk P)
> ...
> (put!>
> o
> 'backups
> (append (; 'o backups) Bk) )
Thus, the single +Backup object gets larger and larger.
As a general rule, you can always use '+Joint' and '+Ref +Link'
interchangeably. You want to use a list of '+Joint's only if the list
doesn't get too long (less than, say, 100).
So the most important change is to remove the line
(rel files (+List +Joint) backups (+File))
from the +Backup class, and use
(rel backups (+List +Ref +Link) NIL (+Backup))
in the +File class. This will increase the speed dramatically.
After that, there are a few more places that should be optimized. You'll
notice the differences only when you create a lot more objects.
In the specification of the database block sizes
> (dbs
> (0)
> (0 +Chunk)
> (0 +File)
> (0 (+File pth inode))
> (4 +Backup)
> (0 (+Backup name startDT hostName)) )
the '0's mean that the block size is 64. This is a bit to small for
'+File' and, more important, for the index trees. I would use '2' for
the '+File' and '+Backup' objects, and '4' for the indexes. This gives:
(2 +File)
(4 (+File pth inode))
(2 +Backup)
(4 (+Backup name startDT hostName)) )
Then, the usage of 'request' is not as intended
> (request '(+File)
> 'pth P
> 'size 0
> 'accessTime 1
> 'modifyTime 2
> 'changeTime 3
> 'inode 4
> 'links 1
> 'uid 1001
> 'gid 1002
> 'accessRights 766 )
'request' searches with the given keys for an object, before it decides
whether to use an existing object or to create a new one.
So, typically, if 'pth' is a characteristic key, it would be called as
(let Obj (request '(+File) 'pth P)
(put> Obj ..)
...
However, I suspect that 'request' is not needed here at all, as you
create new +File objects. So 'new' is the way:
(de addFile (Bk P)
(let Obj
(new (db: +File) '(+File)
'pth P
'size 0
'accessTime 1
'modifyTime 2
'changeTime 3
'inode 4
'links 1
'uid 1001
'gid 1002
'accessRights 766 )
## Not necessary (put> Obj 'backups (append (; Obj backups) Bk))
(put> Obj 'backups Bk)
(at (0 . 10000) (commit))
Obj ) )
Note two other changes I made:
- Because 'backups' is a +List relation, the explicit 'append' is not
necessary. Just putting 'Bk' is enough, the list will be created
automatically.
- Calling 'new!', 'put!>' etc., i.e. the functions which call
(dbSync) and then (commit) each time they are called, is very
expensive. For a large-volume input it is better to go into
single-user mode of the DB, avoid (dbSync), and call 'commit' less
often. In the example above, it is called only every 10000th time.
The same applies to the backup function
> (de Backup (rootPath)
> (let obj1
> (request '(+Backup)
> 'name (stamp)
> 'startDT (stamp)
> 'basePath rootPath
> 'hostName (host "localhost") )
> (put! *DB 'currentBackup obj1)
> # now, walk the path
> (let Dir rootPath
> (recur (Dir)
> (for F (dir Dir)
> (let Path (pack Dir "/" F)
> (addFile obj1 Path)
> # note: change this test: it considers a link to a dir as a
> dir!
> (if (=T (car (info Path)))
> (recurse Path) ) ) ) ) )
> (length (get obj1 'files)) ) )
Avoiding 'request' and 'put!' gives basically
(de backup (RootPath)
(let Obj1
(new (db: +Backup) '(+Backup)
'name (stamp)
'startDT (stamp)
'basePath RootPath
'hostName (host "localhost") )
(put *DB 'currentBackup Obj1)
(commit)
# now, walk the path
(let Dir RootPath
(recur (Dir)
(for F (dir Dir)
(let Path (pack Dir "/" F)
(addFile Obj1 Path)
# note: change this test: it considers a link to a dir as
a dir!
(if (=T (car (info Path)))
(recurse Path) ) ) ) ) )
(commit)
(count (tree 'backups '+File)) ) )
: (bench (backup "/home"))
0.460 sec
-> 7125
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
