On Wednesday, 31 October, 2018 13:22, Gerlando Falauto
wrote:
>first of all let me tell you I cannot thank you enough for going
>through all this trouble for me!
No problem. I still really do not know if the answer is correct however it
does let you get data about how much data is actually b
a while for the database to reach
> steady-state ... to see if as it gets bigger the pagechange set per
> checkpoint increases much.
>
> ---
> The fact that there's a Highway to Hell but only a Stairway to Heaven says
> a lot about anticipated traffic volume.
>
> >-Original Mes
..@mailinglists.sqlite.org] On Behalf Of Gerlando Falauto
>Sent: Tuesday, 30 October, 2018 01:46
>To: SQLite mailing list
>Subject: Re: [sqlite] SQLite for datalogging - best practices
>
>On Tue, Oct 30, 2018 at 4:58 AM Keith Medcalf
>wrote:
>
>>
>> If you don't m
On 10/30/18, Gerlando Falauto wrote:
>
> I understand SQLite is perfectly capable of handling huge database files
> without any issues.
> Yet I'm convinced there may be some corner cases where there might be
> legitimate reasons for wanting partitioning (like this flight recorder mode
> of mine).
Please forgive my insistence -- are both those ideas really, really, stupid?
I understand SQLite is perfectly capable of handling huge database files
without any issues.
Yet I'm convinced there may be some corner cases where there might be
legitimate reasons for wanting partitioning (like this fli
On 10/30/18, Mark Wagner wrote:
> Going back to the comments from Dr. Hipp regarding WAL vs DELETE mode on
> F2FS devices, I just wanted to confirm my understanding.
>
> Given a device with F2FS and with sqlite compiled with
> SQLITE_ENABLE_BATCH_ATOMIC_WRITE, writes with DELETE mode will be
> con
On Tue, Oct 30, 2018 at 4:58 AM Keith Medcalf wrote:
>
> If you don't mind me asking, what sort of data are you collecting?
> Are you the master (ie, scanning) or a slave (getting async data pushed to
> you).
> Are you "compressing" the returned data (storing only changes exceeding
> the deadband
Going back to the comments from Dr. Hipp regarding WAL vs DELETE mode on
F2FS devices, I just wanted to confirm my understanding.
Given a device with F2FS and with sqlite compiled with
SQLITE_ENABLE_BATCH_ATOMIC_WRITE, writes with DELETE mode will be
considerably faster than with WAL mode.
But a
If you don't mind me asking, what sort of data are you collecting?
Are you the master (ie, scanning) or a slave (getting async data pushed to you).
Are you "compressing" the returned data (storing only changes exceeding the
deadband) or are you storing every value (or is the source instrument d
On Mon, Oct 29, 2018 at 6:38 PM Keith Medcalf wrote:
>
> See the ext/misc/unionvtab.c extension for "reading" a bunch of databases
> as if they were a single database.
>
> https://www.sqlite.org/src/artifact/0b3173f69b8899da
Cool, indeed.
I also had a look at the CSV file extension:
https://www
>I see. However, giving up on indexes kind of defeats the whole
>purpose of having a database.
>I assume there is no way to leverage the fact that e.g. rows would be
>inherently sorted by timestamp, without recurring to indexes.
There kind of is. You can do this using a bit of funny business by
See the ext/misc/unionvtab.c extension for "reading" a bunch of databases as if
they were a single database.
https://www.sqlite.org/src/artifact/0b3173f69b8899da
---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a
lot about anticipated traffic volume.
_
On Mon, Oct 29, 2018 at 2:28 PM Richard Hipp wrote:
> On 10/28/18, Gerlando Falauto wrote:
> >
> > This is the current approach:
> > - There is just one single-threaded writer process, which also
> periodically
> > issues DELETE statements to remove older data.
> > - In order to prevent long-run
Hi Keith,
thanks for your feedback and your generous insights.
Bottom line is, it's a cheap device (MLC with 3000 P/E cycles) and there's
nothing I can do about it, I just have to make the best of it.
If there's any way to figure out how "crappy" it is, I'd be interested in
hearing that.
>If ther
Hi Simon,
On Mon, Oct 29, 2018 at 1:30 AM Simon Slavin wrote:
> On 28 Oct 2018, at 11:47pm, Gerlando Falauto
> wrote:
>
> > On an SSD, I have no clue what's going on -- will writing a 2 MB block
> one page at a time be much worse than writing an entire block at once?
>
> You do indeed have no c
On 10/28/18, Gerlando Falauto wrote:
>
> This is the current approach:
> - There is just one single-threaded writer process, which also periodically
> issues DELETE statements to remove older data.
> - In order to prevent long-running reading queries from blocking the
> writer, I'm using WAL mode.
>>The next factor is the internal write multiplication factor. Lets
>>say you have a device which is divided into 2 MB blocks. And you update 1
>>sector (512 bytes) somewhere in this block. The device must (a) read out
>>the entire 2MB block (b) update the data within the block then (c) re-
>>w
On Sunday, 28 October, 2018 17:48, Gerlando Falauto
wrote:
>On Sun, Oct 28, 2018 at 6:18 PM Simon Slavin wrote:
>> On 28 Oct 2018, at 2:06pm, Gerlando Falauto
>> wrote:
>>> - the SSD's wear due to continuous writes should be reduced to a minimum
>> I assume your App generates items for the
On 28 Oct 2018, at 11:47pm, Gerlando Falauto wrote:
> On an SSD, I have no clue what's going on -- will writing a 2 MB block one
> page at a time be much worse than writing an entire block at once?
You do indeed have no clue. And so do I unless I use a utility to look at the
low-level formatt
Hi Keith,
thanks for the heads up on thermal stress.
The next factor is the internal write multiplication factor. Lets say you
> have a device which is divided into 2 MB blocks. And you update 1 sector
> (512 bytes) somewhere in this block. The device must (a) read out the
> entire 2MB block (
On Sunday, 28 October, 2018 16:42, Petite Abeille
wrote:
>> On Oct 28, 2018, at 11:32 PM, Keith Medcalf wrote:
>> will last 50 years (which is 10 times the warranty period)
>Thank you.
No problems. What I really mean of course is not that I "expect" the SSD to
last 50 years (when the warr
On Sun, Oct 28, 2018 at 6:18 PM Simon Slavin wrote:
> On 28 Oct 2018, at 2:06pm, Gerlando Falauto
> wrote:
>
> > There are two (apparently) opposing requirements:
> > - in case of a power outage, all data collected up to at most N seconds
> > prior to the power failure should be readable. Ideall
> On Oct 28, 2018, at 11:32 PM, Keith Medcalf wrote:
>
> will last 50 years (which is 10 times the warranty period)
Thank you.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo
rrantied lifetime by 100
times (estimate is over 1000 years).
---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a
lot about anticipated traffic volume.
>-Original Message-
>From: sqlite-users [mailto:sqlite-users-
>boun...@mailinglists.sqlite.org]
On 28 Oct 2018, at 2:06pm, Gerlando Falauto wrote:
> There are two (apparently) opposing requirements:
> - in case of a power outage, all data collected up to at most N seconds
> prior to the power failure should be readable. Ideally N=0, but what's
> important is that the database never gets cor
Hi everyone,
as I mentioned a few months ago, I'm using SQLite to continuously log data
collected from several sources on a linux system:
This is the current approach:
- There is just one single-threaded writer process, which also periodically
issues DELETE statements to remove older data.
- In o
26 matches
Mail list logo
