[sqlite] In the case of ZIPVFS
Thank you very much. At 2015-12-23 17:32:26, "Richard Hipp" wrote: >On 12/23/15, ??? <2004wqg2008 at 163.com> wrote: >> HI,all >> SQLite retrieves the compressed records is slower than the uncompressed >> records about 30%. >> How can improve the problem? and anyone any suggustion? >> > >Have you run your system in a profiler to determine where the extra >30% time is being used? If the extra time is inside of inflate() (or >whatever other decompression algorithm you are using) or within >rijndaelDecrypt() (or whatever other decryption algorithm you are >using) then probably your only solution will be to disable encryption >and/or compression. There is an engineering trade-off here. > >Please run your system in a profiler and send us the results. Perhaps >we can spot other opportunities to improve performance. > >Also please send as many details about your system as possible. What >kind of hardware are you using? What operating system? Which >compression and encryption algorithms are you using? How large is >your dataset? What kinds of queries are you running? Which specific >version of the NDS dev-kit are you running? > >Probably you should send us the above information through your secure >NDS support channel, rather than here in this public forum. > >-- >D. Richard Hipp >drh at sqlite.org >___ >sqlite-users mailing list >sqlite-users at mailinglists.sqlite.org >http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] In the case of ZIPVFS
Having just glanced at the documentation for ZIPVFS (didn't even know it existed)... 1) Perhaps using a real-time compression algorithm, something like lz4. It's not clear to me which algorithm ZIPVFS uses by default. 2) Perhaps increasing the cache settings to cache more pages in memory. 3) Perhaps increasing the page size. 4) Perhaps ensure that WAL mode is used. -Original Message- From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-bounces at mailinglists.sqlite.org] On Behalf Of ??? Sent: Wednesday, December 23, 2015 4:05 AM To: sqlite-users at mailinglists.sqlite.org Subject: [sqlite] In the case of ZIPVFS HI,all SQLite retrieves the compressed records is slower than the uncompressed records about 30%. How can improve the problem? and anyone any suggustion? best regards wqg ___ sqlite-users mailing list sqlite-users at mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] In the case of ZIPVFS
HI,all SQLite retrieves the compressed records is slower than the uncompressed records about 30%. How can improve the problem? and anyone any suggustion? best regards wqg
[sqlite] In the case of ZIPVFS
??? <2004wqg2008 at 163.com> wrote: > HI,all > SQLite retrieves the compressed records is slower than the uncompressed > records about 30%. > How can improve the problem? and anyone any suggustion? > > best regards > wqg What is your SQLite page size? If you access small records, you're better off with a small page size (fewer bytes to read and decompress). If you have large BLOBs you can afford large page sizes (better compression, which can result in less I/Os). Which compression algorithm are you using? You can see the compression algorithm the first few bytes of the database file. Some compression algorithms are faster than others. For example, lz4hc decompression is much faster than zlib (about 10x faster I think) at the cost of compressing less than zlib. Are you reading only or also writing to the DB? Some algorithms have very different speed for compression and decompression. And of course, make sure that your queries minimize the number of SQLite pages read, which is always good whether you compress, but especially true with compressed DB. Watch for queries doing full table scan, etc. Regards Dominique
[sqlite] In the case of ZIPVFS
On 12/23/15, ??? <2004wqg2008 at 163.com> wrote: > HI,all > SQLite retrieves the compressed records is slower than the uncompressed > records about 30%. > How can improve the problem? and anyone any suggustion? > Have you run your system in a profiler to determine where the extra 30% time is being used? If the extra time is inside of inflate() (or whatever other decompression algorithm you are using) or within rijndaelDecrypt() (or whatever other decryption algorithm you are using) then probably your only solution will be to disable encryption and/or compression. There is an engineering trade-off here. Please run your system in a profiler and send us the results. Perhaps we can spot other opportunities to improve performance. Also please send as many details about your system as possible. What kind of hardware are you using? What operating system? Which compression and encryption algorithms are you using? How large is your dataset? What kinds of queries are you running? Which specific version of the NDS dev-kit are you running? Probably you should send us the above information through your secure NDS support channel, rather than here in this public forum. -- D. Richard Hipp drh at sqlite.org
