Kern Sibbald <[email protected]> wrote on 2010/07/24 17:56:46: > > On Saturday 24 July 2010 17:32:58 Joakim Tjernlund wrote: > > Kern Sibbald <[email protected]> wrote on 2010/07/24 16:46:58: > > > On Saturday 24 July 2010 16:22:54 Joakim Tjernlund wrote: > > > > Kern Sibbald <[email protected]> wrote on 2010/07/24 14:21:46: > > > > > On Saturday 24 July 2010 14:01:59 Joakim Tjernlund wrote: > > > > > > Joakim Tjernlund <[email protected]> wrote on > > > > > > 2010/07/20 > > > > > > > > > > 20:02:53: > > > > > > > From: Joakim Tjernlund <[email protected]> > > > > > > > To: [email protected] > > > > > > > Cc: Joakim Tjernlund <[email protected]> > > > > > > > Date: 2010/07/20 20:24 > > > > > > > Subject: [Bacula-devel] [PATCH 1/2] Initial crc32 optimization. > > > > > > > > > > > > > > This is a strait port of the Linux kernel crc32 optimization > > > > > > > I did a while ago. This impl. is several times faster than the > > > > > > > one it replaces. Adapted to bacula usage. > > > > > > > --- > > > > > > > > > > > > > > I have not tested this yet as I went/is on vacation but I > > > > > > > figured perhaps someone else is interrested enough to do that :) > > > > > > > > > > > > > > bacula/src/lib/crc32.c | 386 > > > > > > > ++++++++++++++++++++++++++++++++++++++++-------- 1 files changed, > > > > > > > 325 insertions(+), 61 deletions(-) > > > > > > > > > > > > So it has been a few days and I was hoping for one or two comments > > > > > > by now. > > > > > > > > > > I have been terribly busy with *big* problems and past due deadlines. > > > > > > > > > > > One thing I have noticed is that Bacula seems to rely on dynamic > > > > > > endian detection instead of detecting endian at configure/compile > > > > > > time. > > > > > > > > > > Yes. > > > > > > > > OK. > > > > > > > > > > Is this so and does it need to stay this way? > > > > > > > > > > I am not sure. A Mac developer was trying to create a single binary > > > > > that would run on multiple architectures, which is a good idea if it > > > > > will work. I don't know whether he actually uses it or not. > > > > > > > > ehh, how can a single binary run on multiple archs? > > > > If all platforms use autconf I think we can let autoconf > > > > detect the hosts endian, see AC_C_BIGENDIAN > > > > > > > > We need to figure this one out before I can move on. If it needs > > > > to be dynamic then I must add an init crc32 table routine that > > > > runs "tole" on the crc32 table. Where should this init routine > > > > be called from? > > > > > > For the moment, to keep your life simple, I suggest you assume that the > > > dynamic endian detection is not critical. If it is we can find out when > > > it is necessary, and deal with it by either calling your routine or > > > another one or by generating your tables in the two endian manners in > > > each execution or something. > > > > > > Hopefully, you can proceed without that blocking you and we deal with it > > > later. > > > > OK, lets leave it for now. > > > > > > > > Currently the new crc32 code needs compile time detection of > > > > > > endianess and I like to keep it that way. Furthermore I suspect I > > > > > > won't be able to do much more Bacula work as vacation ends in one > > > > > > week and work at my dayjob is piling up. > > > > > > > > > > The first thing to do is to fill out an FLA and send it in. See > > > > > www.bacula.org -> FLA License > > > > > > > > I rather not, lets just call this "a few lines of code" which won't > > > > require a FLA. As you can see I have not claimed any CopyRight in the > > > > patch I sent. > > > > > > If you release it into the public domain or put a 3 clause BSD copyright > > > on it or LGPL or something like that I will not need the FLA, but I need > > > either public domain (explicit) or some simply copyright to not require > > > the FLA. > > > > I see, how about GPLv2 then, same as Linux ? The I could just import > > test code and table code from there. > > Unfortunately, Bacula's code is not compatible with other people's code > (Linux) licensed under GPLv2. If you are the author of the code, then you > should be able to use any license you want. There are a number of choices to > get it into Bacula: > > 1. Some BSD type license > 2. Put it into the public domain > 3. Put it under the Bacula license and sign an FLA > > If it is not your code, I need to see the original copyright to know how to be > able to include it, but if it is not your code, then it is unlikely we can > use it.
It is my code, as you can see there is nothing left of the original code after applying my patch. Earlier you mentioned LGPL, is that OK? > > > > > > > > Then the next step is to ensure that your new code and the old code > > > > > can be run in unit test and can read files for CRC from STDIN. Then > > > > > we can compare the > > > > > > > > Which unit tests? I cannot find any(note that I have never touched > > > > Bacula code until now so I might have missed it) > > > > > > Most my little routines like that have a #define TEST_PROGRAM at the end > > > that create a main(), ... and when you compile it into a binary, it tests > > > itself, in some very primitive way. It looks like this one doesn't have > > > such a routine. Take a look at src/lib/edit.c for a really simple case, > > > or alist.c, or scan.c ... > > > > > > Actually, the code in src/lib/md5.c may be very close to what you want to > > > change to do what I ask. > > > > OK, will have a look. > > > > > > > results of the old code with the new code and the speed. Based on > > > > > those results we can see if we need to dynamically detect which > > > > > algorithm to use, and if yours is not identical, and is faster, then > > > > > we would need to modify the Volume format to account for this and do > > > > > the dynamic detection based on the volume format. All future > > > > > versions of Bacula *must* be able to correctly read older Volume > > > > > versions. > > > > > > > > Not sure I follow you here but it seems like you thing that the new > > > > crc32 impl. is incompatible with the current one? This is not so, the > > > > end result, the calculated crc32, is the same(barring any impl. bugs) > > > > as the old one. > > > > > > OK, I didn't re-read your previous email, but it seems to me you said you > > > were not sure. In any case, before I can replace the current algorithm, > > > I must > > > > I am not sure I didn't made a mistake in porting the code from Linux into > > Bacula. The general idea has been proven for some time. > > Unfortunately, the way the GPL works, you cannot port code written by someone > else from Linux to Bacula. The licenses are both Open Source GPL but not > compatible. As stated above, it is my code that I have submitted to Linux. > > If you wrote the code in the Linux kernel that you are porting, then you have > the choices to license it as I indicate above ... Yes, I know. > > > > > > prove to myself (or much faster would be you prove to me) by seeing the > > > output that it is the same. I suggest something like the md5.c or > > > sha1.c programs where it reads a file and creates the hash -- or in your > > > case the CRC. > > > > > > > There won't be any need to test which one to use, this impl. is much > > > > faster than the current one for all archs(unless you want to crc one > > > > byte at a time which the Bacula bcrc32 isn't designed for) > > > > > > I am all for much faster code. The current CRC takes a good deal of > > > time, and I recently added the ability to disable it, but I would really > > > like something faster so that no one disables it :-) > > > > Yes, I noticed this 10 years ago when we started to use the JFFS2 file > > system in Linux. JFFS2 crc32 all its data and it took time. At the time the > > Linux crc32 impl. was like yours. I then optimized it significantly and > > sent it in and all was well. Recently I did another optimization pass and > > it was twice as fast on x86 than the previous one. > > If the original Linux version was like mine, then it was derived from "public > domain code" and as long as you made the vast majority of the modifications, > and you so desire, the code would still be "public domain", so you just have > to explicitly put in the code that your changes to the RFC 2083 algoritm are > public domain, then there is no license issue. There are many impl. of crc32 like similar to yours with different licences. Here is what Linux says * Oct 15, 2000 Matt Domsch <[email protected]> * Nicer crc32 functions/docs submitted by [email protected]. Thanks! * Code was from the public domain, copyright abandoned. Code was * subsequently included in the kernel, thus was re-licensed under the * GNU GPL v2. Doesn't matter though, the current code has been completely rewritten since then. > > > > > OT, I am pushing our IT here to use Bacula but there was one thing > > that Bacula could not do(or so they think) that we need. > > We backup to disk for easy access when we need to restore some file. > > then we transfer on disk backup to tape for safe and long time storage. > > The on disk backups gets recycled after some time after which we only > > have the tape copy left. > > How can we do this in Bacula and make Bacula understand where to > > look for files, either on disk if it is still there or on the corresponding > > tape? > > You backup to disk, then when you want it on tape, you run a Copy job, and put > the copy on tape. When the disk copy is purged, Bacula will promote the Copy > to a Backup, and it will use that version. I see, I will relay this info to IT, thanks. ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
