I have some TSV table in .gz format of only 278MB. But the
corresponding sqlite3 database exceeds 1.58GB (without any index). Is
there a way to make the database file of a size comparable (at least
not over 5 times) to the original TSV table in the .gz file? Thanks.
--
Regards,
Peng
_
I know of two options: The proprietary
https://sqlite.org/zipvfs/doc/trunk/www/readme.wiki and this extension that
you have to call on strings yourself:
https://github.com/siara-cc/Shox96_Sqlite_UDF
Furthermore, some filesystems allow transparent compression, like ntfs,
bcachefs, zfs and btrfs. I
39
>To: SQLite mailing list
>Subject: [sqlite] compressed sqlite3 database file?
>
>I have some TSV table in .gz format of only 278MB. But the
>corresponding sqlite3 database exceeds 1.58GB (without any index). Is
>there a way to make the database file of a size comparable (at least
&g
On Apr 9, 2019, at 11:39 PM, Peng Yu wrote:
>
> Is there a way to make the database file of a size comparable (at least
> not over 5 times) to the original TSV table in the .gz file?
Transparent file compression is a feature of several filesystems: NTFS, ZFS,
Btrfs, and more:
https://en.wi
As I said in my previous email, I have a 13GB database that transparently
compresses to 800MB. Not sure if it got through, didn't get replies to my
last two emails.
Wout.
On Wed., Apr. 10, 2019, 5:04 p.m. Warren Young wrote:
> On Apr 9, 2019, at 11:39 PM, Peng Yu wrote:
> >
> > Is there a way
A database trades off space for performance and functionality. It is
expected that a database with indexes that you can randomly access is going
to take more space than the raw data, let alone a compressed version of the
raw data.
On Wed, Apr 10, 2019 at 12:39 AM Peng Yu wrote:
> I have some TSV
https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos
I work on Mac. Would this be worthwhile to try? Does the transparent
compression work at the file system level or at the directory level? Would
it have a slight chance to corrupt the existent files
On Apr 10, 2019, at 12:08 PM, Peng Yu wrote:
>
> https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos
>
> I work on Mac. Would this be worthwhile to try?
The first link didn’t work here because it didn’t like the APFS drive I tried
it on. (Symptom
On 10 Apr 2019, at 7:08pm, Peng Yu wrote:
> https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos
>
> I work on Mac. Would this be worthwhile to try?
I do not recommend it. Two reasons:
A) If you copy the file you get an uncompressed result. This
What do you recommend for Mac? Thanks.
On 4/10/19, Simon Slavin wrote:
> On 10 Apr 2019, at 7:08pm, Peng Yu wrote:
>
>> https://softwarerecs.stackexchange.com/questions/45010/transparent-file-compression-apps-for-macos
>>
>> I work on Mac. Would this be worthwhile to try?
>
> I do not recommend
en Young
>Sent: Wednesday, 10 April, 2019 09:05
>To: SQLite mailing list
>Subject: Re: [sqlite] compressed sqlite3 database file?
>
>On Apr 9, 2019, at 11:39 PM, Peng Yu wrote:
>>
>> Is there a way to make the database file of a size comparable (at
>least
>> not
On Wed, Apr 10, 2019 at 3:12 PM Keith Medcalf wrote:
> Why would anyone fart about with added complication and the concomittant
> increased unreliability when storage is so damn cheap?
>
Embedded systems and mobile devices.
But of course those probably don't apply here. :)
_
On Wed, 10 Apr 2019 15:20:32 -0500
Peter da Silva wrote:
> > Why would anyone fart about with added complication and the
> > concomittant increased unreliability when storage is so damn cheap?
>
> Embedded systems and mobile devices.
>
> But of course those probably don't apply here. :)
Are yo
On Apr 10, 2019, at 2:12 PM, Keith Medcalf wrote:
>
> It is far cheaper and much more reliable to just buy some file storage space.
>
If you’re going to buy some more storage, you should put ZFS on it then, too. :)
You get a whole lot more from ZFS than just transparent compression.
You act
On Wed, Apr 10, 2019 at 9:51 PM Peng Yu wrote:
> What do you recommend for Mac? Thanks.
>
Nothing. Apple doesn't want you to have compression, because then you would
take longer to buy a new Mac. The afsctool compression is a laughable hack
that only works on read-only data. Writing to the file
On Wednesday, 10 April, 2019 14:21, Peter da Silva wrote:
>On Wed, Apr 10, 2019 at 3:12 PM Keith Medcalf wrote:
>> Why would anyone fart about with added complication and the
>> concomittant increased unreliability when storage is so damn cheap?
>Embedded systems and mobile devices.
You mean
On 10 Apr 2019, at 8:51pm, Peng Yu wrote:
> What do you recommend for Mac? Thanks.
I do not have a good answer ready for you. These days storage is so cheap that
buying an external 2TB drive is cheaper than spending a lot of time doing
clever programming. But if you want to investigate this
Oy. I've worked on safety-critical systems with hard real-time constraints
too. For the most part they didn't *have* file systems or the file systems
were basically read-only in production. Sticking a relational database any
closer than the SCADA monitoring node would not be a thing that happens,
l
On Wed, 10 Apr 2019 15:14:59 -0600
Warren Young wrote:
> On Apr 10, 2019, at 2:12 PM, Keith Medcalf
> wrote:
> >
> > It is far cheaper and much more reliable to just buy some file
> > storage space.
>
> If you?re going to buy some more storage, you should put ZFS on it
> then, too. :)
That'
On Apr 11, 2019, at 1:27 PM, James K. Lowden wrote:
>
> On Wed, 10 Apr 2019 15:14:59 -0600
> Warren Young wrote:
>
>> If you?re going to buy some more storage, you should put ZFS on it
>> then, too. :)
>
> That's interesting advice for a DBMS mailing list.
>
> ZFS has built-in transactions,
On 12 Apr 2019, at 1:45am, Warren Young wrote:
> ZFS
I've used ZFS. ZFS is a great file system for some purposes. Fully-fledged
databases isn't one of them.
Someone already mentioned the problem of a transaction-based DBMS running on a
transaction-based filesystem. I can add the problems o
On Thu, 11 Apr 2019 18:45:01 -0600
Warren Young wrote:
> Sure, but what *is* on the disk after a crash is always consistent
> with ZFS, so any decent database engine can recover.
It's been some years, but I saw a presentation about running Postgres
on ZFS. Every "victory" was a way to compensat
22 matches
Mail list logo