> I tried your example and got ~0.45s for json and ~0.37s for my
> implementation;
I have also tried a custom virtual table, and I also get figures about
20-30% faster than with json.
> Interesting: when NOT using "primary key" (I have a flag for this), the
> times are ~ 0.26s vs 0.18s vs 0.47s !
Interesting indeed, I had (incorrectly) assumed that providing the rowid
would help with performance.
> I agree json works fast enough for such simple example, but I don't know
> how you could use it for e.g. 2 fields;
There are two approaches I found that work with varying degrees of
efficiency depending on the data (how many columns, how large...):
* concat the row data into a string, pass those as a json array, then in
sqlite, use json_each, instr and substr and cast to parse back to columns
* stream the rows as a json array of json objects, and parse in sqlite with
json_each and json_extract, using
select json_extract(value, '$.a') a, json_extract(value, '$.b') b
from (select value from json_each('[{"a":1,"b":2},{"a":3,"b":3}]'))
or
select json_extract(value, '$.a') a, json_extract(value, '$.b') b
from json_each('[{"a":1,"b":2},{"a":3,"b":3}]')
from "a mile high", both forms had a similar performance, but I have not
looked at the vdbe nor at the bowels of json_extract/each.
I will likely got with the custom virtual table in my particular case, but
being able to insert straight from json is something I will definitely use
knowing it's faster than doing it with code (all APIs that revolve around
running queries on input data would be a perfect fit).
Eric
On Tue, May 31, 2016 at 3:32 PM, Gabriel Corneanu <[email protected]
> wrote:
> Hi Eric,
>
> As I know you from Delphi related projects, I thought it would be nice to
> share my ideas.
> I am using a home-grown framework for sqlite persistence, optimized for
> storing records into tables.
> Main ideas:
> - using virtual tables (with the same structure) to speed up sqlite inserts
> a write translates to: insert into test("ID") select "ID" from
> temp.tmp_test
>
> - based on RTTI to generate optimized code for accessing/returning data
> - supports all common data (ints, floats, string, blob/TBytes)
>
> I tried your example and got ~0.45s for json and ~0.37s for my
> implementation; I have also tried using simple recursive cte (see code
> below) but it was slower, ~0.66s.
> I think most of the time is spent in packing the integers and building the
> btree(s) (i.e. CPU bound). Enabling the "name" field below ("storage"
> attribute) increased the processing time (my implementation only) to ~0.47,
> which I think confirms my interpretation.
>
> Interesting: when NOT using "primary key" (I have a flag for this), the
> times are ~ 0.26s vs 0.18s vs 0.47s !
> Looks like sqlite is optimized to generate the rowid, and it is slower when
> explicitly set (even if it's the same value).
>
> I agree json works fast enough for such simple example, but I don't know
> how you could use it for e.g. 2 fields;
>
>
> Hope it helps,
> Gabriel
>
> Code:
>
> type
> TTestRecord = record
> [Storage('ID', [sfPrimaryKey])]
> //[Storage('ID')]
> ID : Int32;
> //[Storage('name')]
> name: string;
> end;
>
> procedure TForm2.Button2Click(Sender: TObject);
> var
> db : TSQLiteDatabase;
> tb : TVirtualTableDataArray<TTestRecord>;
> r : TTestRecord;
> i : integer;
> pf, t,t1,t2,t3: int64;
> json: TStringBuilder;
> begin
> tb := TVirtualTableDataArray<TTestRecord>.Create;
> json := TStringBuilder.Create;
> db := TSQLiteDatabase.Create;
> db.OpenWrite(':memory:');
>
> tb.Setup(db.DB.DB, 'test');
> tb.CreateTable;
> tb.VTInit;
>
> json.Append('[');
> for i := 1 to 1000000 do
> begin
> r.ID := i;
> r.name := i.ToString;
> tb.Items.Add(r);
> if i > 1 then
> json.Append(',');
> json.Append(i);
> end;
> json.Append(']');
>
> QueryPerformanceCounter(t);
> db.db.ExecSQL('insert into test(id) select value from json_each(?)',
> [json.ToString]);
> QueryPerformanceCounter(t1);
> t1 := t1-t;
>
> tb.CreateTable;
> QueryPerformanceCounter(t);
> tb.WriteData;
> QueryPerformanceCounter(t2);
> t2 := t2-t;
>
> tb.CreateTable;
> QueryPerformanceCounter(t);
> db.db.ExecSQL('WITH RECURSIVE ' +
> 'cnt(x) AS (VALUES(1) UNION ALL SELECT x+1 FROM cnt WHERE
> x<'+tb.Count.ToString+
> ') insert into test(id) select x from cnt');
> QueryPerformanceCounter(t3);
> t3 := t3-t;
>
>
> QueryPerformanceFrequency(pf);
> ShowMessageFmt('N=%d T1=%.2fs T2=%0.2fs T3=%.2fs', [tb.Count, t1/pf,
> t2/pf, t3/pf]);
>
> tb.Free;
> db.Free;
> json.Free;
> end;
> _______________________________________________
> sqlite-users mailing list
> [email protected]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[email protected]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
