Hi Jeroen,
I read that one of the goals was removing JPA from the project, because > it's too slow. > So I guess that > > 1. you want to write queries yourself without JPA > 2. *OR* > 3. you also want to get rid of the database, which is I believe Apache > Derby? > > What's the alternative? Saving the indexes into plain text files? > The problem is not _saving_ the index, it's the thousands of queries Syncany has to make during the chunking process: for each chunk, the chunking algorithm has to ask the index "is this chunk there yet (SELECT)? If not, add it (INSERT)". Assuming a 400 MB file and 4 KB chunks, that would be 100.000 queries (+ 100.000 INSERTs if the file is new). With an SQL based index, this is slow. In fact, I assume that any SQL based DB is slower than a simple hash table in the memory. And as long as the chunk index doesn't grow too big, we're good with that approach. Take this example: you store 10 GB of data in the Syncany folder. With a chunk size of 4 KB, that's 10 x 1000 x1000 KB / 4 KB = 2.5 million chunks. Assuming a memory usage of 50 byte (checksum of the chunk + references), that's 2.5 million x 50 bytes = 125 MB -- which is not low, but okay compared to some other applications (*cough* Firefox *cough*). Since we need to exchange the index with the other clients anyway, we do need to store it in a custom file format. Good for an answer? :-) Cheers, Philipp > > Sincerely, > > Jeroen Verheye > > -- > Mailing list: https://launchpad.net/~syncany-team > Post to : [email protected] > Unsubscribe : https://launchpad.net/~syncany-team > More help : https://help.launchpad.net/ListHelp > >
-- Mailing list: https://launchpad.net/~syncany-team Post to : [email protected] Unsubscribe : https://launchpad.net/~syncany-team More help : https://help.launchpad.net/ListHelp
