Help with object serialization please!

@ingo \- this is on the same machine with SQLite: import db_sqlite, times # inMem.nim const n = 1_500_000 let t0 = epochTime() let db = open(":memory:", "", "", "") # In RAM!! db.exec sql"DROP TABLE IF EXISTS bars" db.exec sql"""CREATE TABLE bars (t INTEGER PRIMAR

Help with object serialization please!

**UPDATED BENCHMARK** Task: read 1.5 million price bars off disk and total the volume values Platform: Win 10 Pro Workstation, 2x Xeon X5675 CPUs, 24 gigs memory, SSD @treeform approach - marshalling with flatty and supersnappy libs **Average: 0.14 secs** @cblake approach - using the memfiles

Help with object serialization please!

> Articles like these pop up regularly Hah, yes, I know. In Dutch we'd say "Het is niet goed of het deugd niet" (it's not good nor virtue). So many have tried, even more have failed. SQL is far from perfect, yet what we have works. Use/learn it as it is instead of creating yet another failure o

Help with object serialization please!

@ingo \- just happened to be written yesterday / . While I don't agree 100% with everything written, this excoriating condemnation of SQL as a programming language expresses many problems in a way you may find interesting (and Nimmer's in gene

Help with object serialization please!

> No idea if Windows has this ability. NTFS does support compression but I found Windows Server's block deduplication feature much more useful. > I am also unfamiliar with any port of Windows to a big endian CPU, but I > really don't follow it much. Araq might know. Well Windows runs on ARM an

Help with object serialization please!

@cblake @Scotpip Thanks for the insight.

Help with object serialization please!

I will answer your question with a question: Does SQLite have builtin fancy time series models it can launch directly against full table scans? Doubtful. Whatever it does have will be commonplace and you are unlikely to beat the market with it { you aren't so likely in the first place..but, hey,

Help with object serialization please!

Been there - tried that. Pretty slow for big data sets. Though for read-only, if you set up their in-memory table you can slice and dice the data pretty effectively once you do heave it off the disk. For my use-case, querying the data is not a priority. For what I'm doing (trading backtesting)

Help with object serialization please!

SQLite? (more of a question than an answer)

Help with object serialization please!

I may be a biased judge, but I think the code is **_pretty straightforward_**. My more broad abstract-speak may be harder to follow without background, but I do try to be more concrete. If you give that code some time, I think you'll do fine. :-) People may be interested to hear your perf number

Help with object serialization please!

@cblake Many thanks for your intriguing post. It certainly makes sense, and opens up the prospect of very low-cost and rapid caching to disk which might change my design somewhat. I will cogitate and digest and try to understand the code you have linked. But more broadly I'd ask you to reflect

Help with object serialization please!

Since a lot of people get kind of stuck in their IO ways and for whatever reasons seem to not just trust my word for it and since @Scotpip seems new to Nim and might want some hand holding, here is some **_code to get him/others started_** : import times, os, memfiles, strutils #

Help with object serialization please!

@Scotpip \- while all the @treeform/@planetis advice may be relevant if you need to serialize/deserialize to strings, it sounds like you are doing so **_to a binary file_**. Given that as a hard constraint, you can save much unnecessary work - possibly dozens to hundreds of times more than you n

Help with object serialization please!

Btw try making the `let data = bars.toFlatty()` string 7 bytes shorter, deserialize it with `fromFlatty` and output the last `.v` value. Cheers

Help with object serialization please!

I wonder if these results can be replicated for Nim's upcoming "incremental compilation" feature.

Help with object serialization please!

its simple import std/streams type Bar = object d: Timestamp o: float h: float l: float c: float v: float var bars: seq[Bar] bars.add(Bar( d: parseTs("{year}-{month}-{day}", "2000-01-01"), o: 123.123, h: 12.23

Help with object serialization please!

**Treeform** \- brilliant advice - I'd never have gotten to such an effective solution on my own. For 1.5 million records to and from HDD on my HP Z600 Win 10 Pro workstation, the average benchmarks were: * Json serialisation: 11.6 secs * Flatty serialisation: 1.7 secs * Flatty+Snappy ser

Help with object serialization please!

You know what's more important than speed, right? Well, its error checking! Try out when you have the time.

Help with object serialization please!

The main problem you have is `DateTime`. Because the `DateTime` contains functions its very hard to serialize. What are you going to do serialize the assembly machine code of the functions? Huge security risk! If you instead use a `float64` timestamp or my `chrono` library you would not have th

Help with object serialization please!

Hi folks Newbie question here. I've figured out everything I need for my project except for one showstopping issue, and would very much appreciate a hand. I need a fast way to serialize and deserialize a seq of a relatively simple object to a binary file. The typical sequence will contain a few