Fastest JSON parser in the world is a D project
JSON parsing in D has come a long way, especially when you look at it from the efficiency angle as a popular benchmark does that has been forked by well known D contributers like Martin Nowak or Sönke Ludwig. The test is pretty simple: Parse a JSON object, containing an array of 1_000_000 3D coordinates in the range [0..1) and average them. The performance of std.json in parsing those was horrible still in the DMD 2.066 days*: DMD : 41.44s, 934.9Mb Gdc : 29.64s, 929.7Mb Python : 12.30s, 1410.2Mb Ruby: 13.80s, 2101.2Mb Then with 2.067 std.json got a major 3x speed improvement and rivaled the popular dynamic languages Ruby and Python: DMD : 13.02s, 1324.2Mb In the mean time several other D JSON libraries appeared with varying focus on performance or API: Medea : 56.75s, 1753.6Mb (GDC) libdjson : 24.47s, 1060.7Mb (GDC) stdx.data.json: 2.76s, 207.1Mb (LDC) Yep, that's right. stdx.data.json's pull parser finally beats the dynamic languages with native efficiency. (I used the default options here that provide you with an Exception and line number on errors.) A few days ago I decided to get some practical use out of my pet project 'fast' by implementing a JSON parser myself, that could rival even the by then fastest JSON parser, RapidJSON. The result can be seen in the benchmark results right now: https://github.com/kostya/benchmarks#json fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) (* Timings from my computer, Haswell CPU, Linux amd64.) -- Marco
Fastest JSON parser in the world is a D project
fast.json usage: UTF-8 and JSON validation of used portions by default: auto json = parseJSONFile("data.json"); Known good file input: auto json = parseTrustedJSONFile("data.json"); auto json = parseTrustedJSON(`{"x":123}`); Work with a single key from an object: json.singleKey!"someKey" json.someKey Iteration: foreach (key; json.byKey) // object by key foreach (idx; json)// array by index Remap member names: @JsonRemap(["clazz", "class"]) struct S { string clazz; } @JsonRemap(["clazz", "class"]) enum E { clazz; } Example: double x = 0, y = 0, z = 0; auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, "y": 2, "z": 3 }, … ] }`); foreach (idx; json.coordinates) { // Provide one function for each key you are interested in json.keySwitch!("x", "y", "z")( { x += json.read!double; }, { y += json.read!double; }, { z += json.read!double; } ); } Features: - Loads double values in compliance with IEEE round-to-nearest (no precision loss in serialization->deserialization round trips) - UTF-8 validation of non-string input (file, ubyte[]) - Currently fastest JSON parser money can buy - Reads strings, enums, integral types, double, bool, POD structs consisting of those and pointers to such structs Shortcomings: - Rejects numbers with exponents of huge magnitude (>=10^28) - Only works on Posix x86/amd64 systems - No write capabilities - Data size limited by available contiguous virtual memory -- Marco
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, "y": 2, "z": 3 }, … ] }`); I assume parseTrustedJSON is not validating? Did you use it in the benchmark? And were the competitors non-validating as well?
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: https://github.com/kostya/benchmarks#json I can't find fast.json here. Where is it?
Re: Fastest JSON parser in the world is a D project
On Wed, Oct 14, 2015 at 9:35 AM, Marco Leise via Digitalmars-d-announce < digitalmars-d-announce@puremagic.com> wrote: > Features: > - Loads double values in compliance with IEEE round-to-nearest > (no precision loss in serialization->deserialization round trips) > - UTF-8 validation of non-string input (file, ubyte[]) > - Currently fastest JSON parser money can buy > - Reads strings, enums, integral types, double, bool, POD > structs consisting of those and pointers to such structs > Does this version handle real world JSON? I've keep getting problems with vibe and JSON because web browsers will automatically make a "1" into a 1 which then causes exceptions in vibe. Does yours do lossless conversions automatically?
Re: Fastest JSON parser in the world is a D project
Am Wed, 14 Oct 2015 07:55:18 + schrieb Idan Arye : > On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: > > auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, > > "y": 2, "z": 3 }, … ] }`); > > I assume parseTrustedJSON is not validating? Did you use it in > the benchmark? And were the competitors non-validating as well? That is correct. For the benchmark parseJSONFile was used though, which validates UTF-8 and JSON in the used portions. That probably renders your third question superfluous. I wouldn't know anyways, but am inclined to think they all validate the entire JSON and some may skip UTF-8 validation, which is a low cost operation in this ASCII file anyways. -- Marco
Re: Fastest JSON parser in the world is a D project
Am Wed, 14 Oct 2015 10:22:37 +0200 schrieb Rory McGuire via Digitalmars-d-announce : > Does this version handle real world JSON? > > I've keep getting problems with vibe and JSON because web browsers will > automatically make a "1" into a 1 which then causes exceptions in vibe. > > Does yours do lossless conversions automatically? No I don't read numbers as strings. Could the client JavaScript be fixed? I fail to see why the conversion would happen automatically when the code could explicitly check for strings before doing math with the value "1". What do I miss? -- Marco
Re: Fastest JSON parser in the world is a D project
Am Wed, 14 Oct 2015 08:19:52 + schrieb Per Nordlöw : > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > https://github.com/kostya/benchmarks#json > > I can't find fast.json here. Where is it? »»» D Gdc Fast 0.34226.7 ««« C++ Rapid 0.79687.1 Granted if he wrote "D fast.json" it would have been easier to identify. -- Marco
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) (* Timings from my computer, Haswell CPU, Linux amd64.) Where's the code?
Re: Fastest JSON parser in the world is a D project
Gary Willoughby via Digitalmars-d-announce napsal Čt, říj 15, 2015 v 10∶08 : On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) (* Timings from my computer, Haswell CPU, Linux amd64.) Where's the code? code.dlang.org
Re: Fastest JSON parser in the world is a D project
Daniel Kozak via Digitalmars-d-announce píše v Čt 15. 10. 2015 v 11:07 +0200: > > > Gary Willoughby via Digitalmars-d-announce remagic.com> napsal Čt, říj 15, 2015 v 10∶08 : > > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > fast: 0.34s, 226.7Mb (GDC) > > RapidJSON: 0.79s, 687.1Mb (GCC) > > > > (* Timings from my computer, Haswell CPU, Linux amd64.) > > > > Where's the code? > code.dlang.org https://github.com/mleise/fast
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) Why not add this to std.experimental?
Re: Fastest JSON parser in the world is a D project
On 10/15/15 12:40 PM, Per Nordlöw wrote: On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) Why not add this to std.experimental? Sure seems like a good question! At the least a more generic generalization (more character and range types etc) should start from Marco's core implementation. -- Andrei
Re: Fastest JSON parser in the world is a D project
In browser JSON.serialize is the usual way to serialize JSON values. The problem is that on D side if one does deserialization of an object or struct. If the types inside the JSON don't match exactly then vibe freaks out. Another problem with most D JSON implementations is that they don't support proper JSON, e.g. outputting nan as though it was a valid value etc... browsers don't like that stuff. For me the best D JSON implementation at the moment is actually jsvar by Adam and its not even a JSON parser, it just has that as a needed feature. It feels slow though I haven't benchmarked, but if I run it over a couple of Gigs of data(Paged by 1000) it takes a long while. On Thu, Oct 15, 2015 at 3:42 AM, Marco Leise via Digitalmars-d-announce < digitalmars-d-announce@puremagic.com> wrote: > Am Wed, 14 Oct 2015 10:22:37 +0200 > schrieb Rory McGuire via Digitalmars-d-announce > : > > > Does this version handle real world JSON? > > > > I've keep getting problems with vibe and JSON because web browsers will > > automatically make a "1" into a 1 which then causes exceptions in vibe. > > > > Does yours do lossless conversions automatically? > > No I don't read numbers as strings. Could the client > JavaScript be fixed? I fail to see why the conversion would > happen automatically when the code could explicitly check for > strings before doing math with the value "1". What do I miss? > > -- > Marco > >
Re: Fastest JSON parser in the world is a D project
Am Thu, 15 Oct 2015 11:09:01 +0200 schrieb Daniel Kozak : > Daniel Kozak via Digitalmars-d-announce píše v Čt 15. 10. 2015 v 11:07 > +0200: > > > > > > Gary Willoughby via Digitalmars-d-announce > > napsal Čt, říj 15, 2015 v > > 10∶08 : > > > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > > fast: 0.34s, 226.7Mb (GDC) > > > RapidJSON: 0.79s, 687.1Mb (GCC) > > > > > > (* Timings from my computer, Haswell CPU, Linux amd64.) > > > > > > Where's the code? > > code.dlang.org > > https://github.com/mleise/fast BTW: Is there a reason why the code is GPL licensed? I understand that people might want to use more restrictive licenses, but isn't LGPL a better replacement for GPL when writing library code? Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? See: http://stackoverflow.com/a/10179181/471401
Re: Fastest JSON parser in the world is a D project
On Thursday, 15 October 2015 at 12:51:58 UTC, Johannes Pfau wrote: BTW: Is there a reason why the code is GPL licensed? I understand that people might want to use more restrictive licenses, but isn't LGPL a better replacement for GPL when writing library code? Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? See: http://stackoverflow.com/a/10179181/471401 It also precludes any of this code being used in Phobos :/
Re: Fastest JSON parser in the world is a D project
Am 15.10.2015 um 13:06 schrieb Rory McGuire via Digitalmars-d-announce: In browser JSON.serialize is the usual way to serialize JSON values. The problem is that on D side if one does deserialization of an object or struct. If the types inside the JSON don't match exactly then vibe freaks out. For float and double fields, the serialization code should actually accept both, floating point and integer numbers: https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1603 https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1804 Do you have a test case for your error?
Re: Fastest JSON parser in the world is a D project
Am 14.10.2015 um 09:01 schrieb Marco Leise: JSON parsing in D has come a long way, especially when you look at it from the efficiency angle as a popular benchmark does that has been forked by well known D contributers like Martin Nowak or Sönke Ludwig. The test is pretty simple: Parse a JSON object, containing an array of 1_000_000 3D coordinates in the range [0..1) and average them. The performance of std.json in parsing those was horrible still in the DMD 2.066 days*: DMD : 41.44s, 934.9Mb Gdc : 29.64s, 929.7Mb Python : 12.30s, 1410.2Mb Ruby: 13.80s, 2101.2Mb Then with 2.067 std.json got a major 3x speed improvement and rivaled the popular dynamic languages Ruby and Python: DMD : 13.02s, 1324.2Mb In the mean time several other D JSON libraries appeared with varying focus on performance or API: Medea : 56.75s, 1753.6Mb (GDC) libdjson : 24.47s, 1060.7Mb (GDC) stdx.data.json: 2.76s, 207.1Mb (LDC) Yep, that's right. stdx.data.json's pull parser finally beats the dynamic languages with native efficiency. (I used the default options here that provide you with an Exception and line number on errors.) From when are the numbers for stdx.data.json? The latest results for the pull parser that I know of were faster than RapidJson: http://forum.dlang.org/post/wlczkjcawyteowjbb...@forum.dlang.org Judging by the relative numbers, your "fast" result is still a bit faster, but if that doesn't validate, it's hard to make an objective comparison.
Re: Fastest JSON parser in the world is a D project
On Thursday, 15 October 2015 at 10:34:16 UTC, Andrei Alexandrescu wrote: On 10/15/15 12:40 PM, Per Nordlöw wrote: On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) Why not add this to std.experimental? Sure seems like a good question! At the least a more generic generalization (more character and range types etc) should start from Marco's core implementation. -- Andrei Would it not be a better use of effort to attempt to merge the efforts here over to Sonkes new stdx.json? I didn't look at either codebase, so I dont know how difficult that'll be.
Re: Fastest JSON parser in the world is a D project
On 2015-10-15 14:51, Johannes Pfau wrote: Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? Yes, it does have that enforcement. -- /Jacob Carlborg
Re: Fastest JSON parser in the world is a D project
On Thursday, 15 October 2015 at 19:40:16 UTC, Jacob Carlborg wrote: On 2015-10-15 14:51, Johannes Pfau wrote: Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? Yes, it does have that enforcement. Then this is practically useless for the vast majority of programmers.
Re: Fastest JSON parser in the world is a D project
On Thursday, October 15, 2015 14:51:58 Johannes Pfau via Digitalmars-d-announce wrote: > BTW: Is there a reason why the code is GPL licensed? I understand that > people might want to use more restrictive licenses, but isn't LGPL a > better replacement for GPL when writing library code? Doesn't the GPL > force everybody _using_ fast.json to also use the GPL license? > > See: http://stackoverflow.com/a/10179181/471401 I think that you might be able to link code with various other compatible, open source licenses against it, but you definitely can't link any proprietary code aganist it. GPL really makes more sense for programs than for libraries for precisely that reason. And most D libraries are likely to be Boost licensed, since that's the license used by Phobos and generally favored by the D community. There's nothing wrong with releasing a library under the GPL if you really want to, but it seriously limits its usefulness. - Jonathan M Davis
Re: Fastest JSON parser in the world is a D project
On Thursday, October 15, 2015 09:40:05 Per Nordlöw via Digitalmars-d-announce wrote: > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > fast: 0.34s, 226.7Mb (GDC) > > RapidJSON: 0.79s, 687.1Mb (GCC) > > Why not add this to std.experimental? I thought that http://code.dlang.org/packages/std_data_json was the json implementation we were looking at adding to Phobos. Or did that fall through? I haven't paid much attention to the discussion on that, though I have used it in one of my own projects. - Jonathan M Davis
Re: Fastest JSON parser in the world is a D project
On 2015-10-16 00:14, Jonathan M Davis via Digitalmars-d-announce wrote: I thought that http://code.dlang.org/packages/std_data_json was the json implementation we were looking at adding to Phobos. Or did that fall through? I haven't paid much attention to the discussion on that, though I have used it in one of my own projects. Yes, that was the plan. But if a better alternative shows up, should we look at that as well? -- /Jacob Carlborg
Re: Fastest JSON parser in the world is a D project
On 2015-10-16 00:12, Jonathan M Davis via Digitalmars-d-announce wrote: I think that you might be able to link code with various other compatible, open source licenses against it, but you definitely can't link any proprietary code aganist it. GPL really makes more sense for programs than for libraries for precisely that reason. And most D libraries are likely to be Boost licensed, since that's the license used by Phobos and generally favored by the D community. There's nothing wrong with releasing a library under the GPL if you really want to, but it seriously limits its usefulness. Yes, that's correct. It would be fine if everything used GPL, but that's not the world we live in. Which makes the license not very practical. -- /Jacob Carlborg
Re: Fastest JSON parser in the world is a D project
On Friday, October 16, 2015 08:21:32 Jacob Carlborg via Digitalmars-d-announce wrote: > On 2015-10-16 00:14, Jonathan M Davis via Digitalmars-d-announce wrote: > > > I thought that http://code.dlang.org/packages/std_data_json was the json > > implementation we were looking at adding to Phobos. Or did that fall > > through? I haven't paid much attention to the discussion on that, though I > > have used it in one of my own projects. > > Yes, that was the plan. But if a better alternative shows up, should we > look at that as well? Sure, but going from std_data_json as being the candidate to talking about putting this other one in std.experimental seems a bit much. It needs to go through the review process first, and if we're doing that, it doesn't make sense to have two winners. They'll have to duke it out (or be merged), and then the one that wins can go in std.experimental. - Jonathan M Davis
Re: Fastest JSON parser in the world is a D project
On 10/15/15 10:40 PM, Jacob Carlborg wrote: On 2015-10-15 14:51, Johannes Pfau wrote: Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? Yes, it does have that enforcement. Then we'd need to ask Marco if he's willing to relicense the code with Boost. -- Andrei
Re: Fastest JSON parser in the world is a D project
On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis wrote: On Thursday, October 15, 2015 14:51:58 Johannes Pfau via Digitalmars-d-announce wrote: BTW: Is there a reason why the code is GPL licensed? I understand that people might want to use more restrictive licenses, but isn't LGPL a better replacement for GPL when writing library code? Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? See: http://stackoverflow.com/a/10179181/471401 I think that you might be able to link code with various other compatible, open source licenses against it, but you definitely can't link any proprietary code aganist it. Yes, you can. GPL only affects distribution of executables to third party, it doesn't affect services. Maybe you are thinking of AGPL, which also affects services. But even AGPL allows internal usage.
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: https://github.com/kostya/benchmarks#json Does fast.json use any non-standard memory allocation patterns or plain simple GC-usage?
Re: Fastest JSON parser in the world is a D project
On 10/16/15 6:20 AM, Ola Fosheim Grøstad wrote: On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis wrote: On Thursday, October 15, 2015 14:51:58 Johannes Pfau via Digitalmars-d-announce wrote: BTW: Is there a reason why the code is GPL licensed? I understand that people might want to use more restrictive licenses, but isn't LGPL a better replacement for GPL when writing library code? Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? See: http://stackoverflow.com/a/10179181/471401 I think that you might be able to link code with various other compatible, open source licenses against it, but you definitely can't link any proprietary code aganist it. Yes, you can. GPL only affects distribution of executables to third party, it doesn't affect services. Maybe you are thinking of AGPL, which also affects services. But even AGPL allows internal usage. No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. -Steve
Re: Fastest JSON parser in the world is a D project
Am Thu, 15 Oct 2015 18:17:07 +0200 schrieb Sönke Ludwig : > Am 15.10.2015 um 13:06 schrieb Rory McGuire via Digitalmars-d-announce: > > In browser JSON.serialize is the usual way to serialize JSON values. > > The problem is that on D side if one does deserialization of an object > > or struct. If the types inside the JSON don't match exactly then vibe > > freaks out. > > For float and double fields, the serialization code should actually > accept both, floating point and integer numbers: > > https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1603 > https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1804 > > Do you have a test case for your error? Well it is not an error. Rory originally wrote about conversions between "1" and 1 happening on the browser side. That would mean adding a quirks mode to any well-behaving JSON parser. In this case: "read numbers as strings". Hence I was asking if the data on the client could be fixed, e.g. the json number be turned into a string first before serialization. -- Marco
Re: Fastest JSON parser in the world is a D project
Am Thu, 15 Oct 2015 18:46:12 +0200 schrieb Sönke Ludwig : > Am 14.10.2015 um 09:01 schrieb Marco Leise: > > […] > > stdx.data.json: 2.76s, 207.1Mb (LDC) > > > > Yep, that's right. stdx.data.json's pull parser finally beats > > the dynamic languages with native efficiency. (I used the > > default options here that provide you with an Exception and > > line number on errors.) > > From when are the numbers for stdx.data.json? The latest results for > the pull parser that I know of were faster than RapidJson: > http://forum.dlang.org/post/wlczkjcawyteowjbb...@forum.dlang.org You know, I'm not surprised at the "D new lazy Ldc" result, which is in the ball park figure of what I measured without exceptions & line-numbers, but the Rapid C++ result seems way off compared to kostya's listing. Or maybe that Core i7 doesn't work well with RapidJSON. I used your fork of the benchmark, made some modifications like adding taggedalgebraic and what else was needed to make it compile with vanilla ldc2 0.16.0. Then I removed the flags that disable exceptions and line numbers. Compilation options are the same as for the existing gdc and ldc2 entries. I did not add " -partial-inliner -boundscheck=off -singleobj ". > Judging by the relative numbers, your "fast" result is still a bit > faster, but if that doesn't validate, it's hard to make an objective > comparison. Every value that is read (as opposed to skipped) is validated according to RFC 7159. That includes UTF-8 validation. Full validation (i.e. readJSONFile!validateAll(…);) may add up to 14% overhead here. -- Marco
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 12:53:09 UTC, Steven Schveighoffer wrote: Yes, you can. GPL only affects distribution of executables to third party, it doesn't affect services. Maybe you are thinking of AGPL, which also affects services. But even AGPL allows internal usage. No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. There was something called the "server loophole." The language of GPLv2 only requires source to be distributed if the binaries are distributed. The Affero GPL was created to close the loophole, requiring source to be made available even if the binaries aren't distributed. IIRC, GPLv3 requires the same.
Re: Fastest JSON parser in the world is a D project
On 10/16/2015 08:53 AM, Steven Schveighoffer wrote: On 10/16/15 6:20 AM, Ola Fosheim Grøstad wrote: On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis wrote: On Thursday, October 15, 2015 14:51:58 Johannes Pfau via Digitalmars-d-announce wrote: BTW: Is there a reason why the code is GPL licensed? I understand that people might want to use more restrictive licenses, but isn't LGPL a better replacement for GPL when writing library code? Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? See: http://stackoverflow.com/a/10179181/471401 I think that you might be able to link code with various other compatible, open source licenses against it, but you definitely can't link any proprietary code aganist it. Yes, you can. GPL only affects distribution of executables to third party, it doesn't affect services. Maybe you are thinking of AGPL, which also affects services. But even AGPL allows internal usage. No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. This is the real reason I'm not a huge fan of *GPL. Nobody can understand it!
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 14:05:50 UTC, Mike Parker wrote: On Friday, 16 October 2015 at 12:53:09 UTC, Steven Schveighoffer wrote: Yes, you can. GPL only affects distribution of executables to third party, it doesn't affect services. Maybe you are thinking of AGPL, which also affects services. But even AGPL allows internal usage. No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. There was something called the "server loophole." The language of GPLv2 only requires source to be distributed if the binaries are distributed. The Affero GPL was created to close the loophole, requiring source to be made available even if the binaries aren't distributed. IIRC, GPLv3 requires the same. Looks like I got my versions wrong. AGPL is a modified GPLv3. http://www.gnu.org/licenses/why-affero-gpl.en.html
Re: Fastest JSON parser in the world is a D project
Am Fri, 16 Oct 2015 11:09:37 + schrieb Per Nordlöw : > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > https://github.com/kostya/benchmarks#json > > Does fast.json use any non-standard memory allocation patterns or > plain simple GC-usage? Plain GC. I found no way in D to express something as "borrowed", but the GC removes the ownership question from the picture. That said the only thing that I allocate in this benchmark is the dynamic array of coordinates (1_000_000 * 3 * double.sizeof), which can be replaced by lazily iterating over the array to remove all allocations. Using these lazy techniques for objects too and calling .borrow() instead of .read!string() (which .idups) should allow GC free usage. (Well, except for the one in parseJSON, where I append 16 zero bytes to the JSON string to make it SSE safe.) -- Marco
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 15:07:17 UTC, Ola Fosheim Grøstad wrote: GPL, on the other hand, gives the same right to users of a service. Typo, "AGPL", not "GPL"...
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 12:53:09 UTC, Steven Schveighoffer wrote: No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. Yes, you can. GPL is a copyright license which says that if you legally obtain a copy of an executable then you also have the right to the source code and the right to make copies. If you don't hand out an executable then there are no obligations at all, for obvious reasons. GPL, on the other hand, gives the same right to users of a service. https://en.wikipedia.org/wiki/Affero_General_Public_License
Re: Fastest JSON parser in the world is a D project
On 10/16/15 11:07 AM, Ola Fosheim Grøstad wrote: On Friday, 16 October 2015 at 12:53:09 UTC, Steven Schveighoffer wrote: No, you cannot link against GPL library without making your code also GPL. "Services" I don't think have anything to do with this, we are talking about binary linking. Yes, you can. GPL is a copyright license which says that if you legally obtain a copy of an executable then you also have the right to the source code and the right to make copies. If you don't hand out an executable then there are no obligations at all, for obvious reasons. You certainly can link with it, and then your code becomes GPL. you don't have to distribute the binary, but it's still now GPL. The question is, can you link a proprietary licensed piece of software against GPL, while having the proprietary software retain its license, and the answer is no. If you want to say GPL is fine if you only want to provide your software as a service, then that is not an answer to the question. Your software is effectively GPL, but you don't have to distribute the source because you didn't distribute the binary. However, you would have no recourse if you mistakenly provided the binary to someone. Ever. This is a poison pill risk that most companies will not swallow. Please give me a json library that's 10% slower and won't ruin my entire business, thanks. -Steve
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 14:09:13 UTC, Nick Sabalausky wrote: This is the real reason I'm not a huge fan of *GPL. Nobody can understand it! It is really simple!! The basic idea is that people shouldn't have to reverse engineer software they use in order to fix it/modify it, so when you receive software you should get the right to the means to modify it (the source code). In addition GPL also gives you the right to distribute copies (if you want to) so that you can let others enjoy your improved version of the program. It doesn't give the public the right to demand source code to be made available, only owners of legally obtained copies get the right to demand the full source to be available for them. It also does not forbid linking against anything, it requires the copyright holder to grant rights to the receiver of the copy (access to source code and making copies to distribute under the same terms). As long as you keep your modifications/derived works for yourself, the only party that has been granted GPL for the derived work is yourself. One dilemma here is that a company with a million employees is treated like a single entity legally. So big companies can embrace the GPL freely for internal use and services without the redistribution GPL clauses coming into effect, whereas smaller companies that exchange software between them cannot restrict redistribution...
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 15:36:26 UTC, Steven Schveighoffer wrote: You certainly can link with it, and then your code becomes GPL. No, the code is code. It is an artifact. The GPL is a legal document. The legal document says what rights you have to the copy you received and what requirements that follows it. You are allowed to modify it and do anything you want with it that is covered under fair use. This varies between jurisdictions. The license primarily comes into effect when you _distribute_ or _publish_, because the legal precedent for putting restrictions on distribution and publishing is much stronger. And WIPO is much more clear there. So, if you build websites for a third party you can use GPL without redistribution by writing the contract in such a way that the third party is using your service. Meaning, you run the software. So circumventing the GPL isn't all that hard if you want to. The AGPL also affects publishing as a service, so it makes such arrangements much more difficult.
Re: Fastest JSON parser in the world is a D project
On 10/16/15 11:56 AM, Ola Fosheim Grøstad wrote: On Friday, 16 October 2015 at 15:36:26 UTC, Steven Schveighoffer wrote: You certainly can link with it, and then your code becomes GPL. No, the code is code. It is an artifact. The GPL is a legal document. The legal document says what rights you have to the copy you received and what requirements that follows it. You are allowed to modify it and do anything you want with it that is covered under fair use. This varies between jurisdictions. The license primarily comes into effect when you _distribute_ or _publish_, because the legal precedent for putting restrictions on distribution and publishing is much stronger. And WIPO is much more clear there. Right, so what happens when you accidentally distribute it? What license is it under? For example, let's say you have a product that doesn't use JSON. It's proprietary, and you distribute it under a proprietary license. You want to include JSON parsing, so you incorporate this GPL'd library. Then you distribute it under your proprietary license. Recipient says "Wait, you used fast.json! That means this is now GPL, I want the source". Then what? So, if you build websites for a third party you can use GPL without redistribution by writing the contract in such a way that the third party is using your service. Meaning, you run the software. So circumventing the GPL isn't all that hard if you want to. Being able to use GPL on SAAS doesn't satisfy the use case here. This is a compiled library, it can be used in any piece of software. -Steve
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 17:38:01 UTC, Steven Schveighoffer wrote: For example, let's say you have a product that doesn't use JSON. It's proprietary, and you distribute it under a proprietary license. You want to include JSON parsing, so you incorporate this GPL'd library. Then you distribute it under your proprietary license. Recipient says "Wait, you used fast.json! That means this is now GPL, I want the source". Then what? The recipient has no say in this, but the original author can demand that you either stop distribution or purchase a compatible license. Being able to use GPL on SAAS doesn't satisfy the use case here. This is a compiled library, it can be used in any piece of software. My point was that you can use GPLed code in a proprietary service. But you can also ship propritary code separately that the end user links with the GPLed code. It is only when you bundle the two that you get a derived work.
Re: Fastest JSON parser in the world is a D project
On 10/16/15 2:24 PM, Ola Fosheim Grøstad wrote: On Friday, 16 October 2015 at 17:38:01 UTC, Steven Schveighoffer wrote: For example, let's say you have a product that doesn't use JSON. It's proprietary, and you distribute it under a proprietary license. You want to include JSON parsing, so you incorporate this GPL'd library. Then you distribute it under your proprietary license. Recipient says "Wait, you used fast.json! That means this is now GPL, I want the source". Then what? The recipient has no say in this, but the original author can demand that you either stop distribution or purchase a compatible license. Exactly my point. Being able to use GPL on SAAS doesn't satisfy the use case here. This is a compiled library, it can be used in any piece of software. My point was that you can use GPLed code in a proprietary service. But you can also ship propritary code separately that the end user links with the GPLed code. It is only when you bundle the two that you get a derived work. And I don't disagree with your point, just that it was not a correct response to "but you definitely can't link any proprietary code aganist [sic] it." -Steve
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 18:53:39 UTC, Steven Schveighoffer wrote: And I don't disagree with your point, just that it was not a correct response to "but you definitely can't link any proprietary code aganist [sic] it." That I don't understand. You can indeed build your executable from a mix of proprietary third party libraries and GPL code, that means you definetively can link. You cannot distribute it together to another third party, but your employer can use it and run a service with it. People attribute way too many limitations to GPL codebases. For many organizations the GPL would be perfectly ok for their software stack.
Re: Fastest JSON parser in the world is a D project
On 10/16/15 3:36 PM, Ola Fosheim Grøstad wrote: On Friday, 16 October 2015 at 18:53:39 UTC, Steven Schveighoffer wrote: And I don't disagree with your point, just that it was not a correct response to "but you definitely can't link any proprietary code aganist [sic] it." That I don't understand. You can indeed build your executable from a mix of proprietary third party libraries and GPL code, that means you definetively can link. You cannot distribute it together to another third party, but your employer can use it and run a service with it. The distribution is implied in the comment. If there isn't distribution, the license taint isn't important, why bring it up? In any case, having a GPL license for a library diminishes its usefulness to proprietary software houses. People attribute way too many limitations to GPL codebases. For many organizations the GPL would be perfectly ok for their software stack. It depends on what you do. Sure, if you are a pure SAAS house, GPL is perfectly fine, but if one day you want to license that as an installable server, you need to re-develop that GPL piece, and make sure your developers never looked at the code. It's not something to take lightly. -Steve
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 21:01:02 UTC, Steven Schveighoffer wrote: The distribution is implied in the comment. If there isn't distribution, the license taint isn't important, why bring it up? That was not implied. You can have a license which is much more limiting, the GPL is fairly liberal. Most software that is written is not for redistribution! In any case, having a GPL license for a library diminishes its usefulness to proprietary software houses. If that's what you you mean, then be explicit about it. It depends on what you do. Sure, if you are a pure SAAS house, GPL is perfectly fine, but if one day you want to license that as an installable server, you need to re-develop that GPL piece, It isn't obvious, you should be able to lease a server without that being considered obtaining a copy? To figure that out you'll need a legal interpretation of the GPL for your jurisdiction. and make sure your developers never looked at the code. It's not something to take lightly. No, having read code in the past does not affect copyright. If you don't translate the GPL code while writing then it isn't a derived work. What you are thinking about is clean room implementation of reverse engineered APIs to hardware where the code only is tiny stubs of machine code that has to be written in a certain way to be compatible.
Re: Fastest JSON parser in the world is a D project
Am 16.10.2015 um 17:11 schrieb Marco Leise: Am Thu, 15 Oct 2015 18:17:07 +0200 schrieb Sönke Ludwig : (...) Do you have a test case for your error? Well it is not an error. Rory originally wrote about conversions between "1" and 1 happening on the browser side. That would mean adding a quirks mode to any well-behaving JSON parser. In this case: "read numbers as strings". Hence I was asking if the data on the client could be fixed, e.g. the json number be turned into a string first before serialization. Okay, I obviously misread that as a once familiar issue. Maybe it indeed makes sense to add a "JavaScript" quirks mode that behaves exactly like a JavaScript interpreter would.
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: - Data size limited by available contiguous virtual memory Mmaping files for sequential reading is a very debatable choice, b/c the common use case is to read a file once. You should at least compare the numbers w/ drop_caches between each run. https://github.com/mleise/fast/blob/69923d5a69f67c21a37e5e2469fc34d60c9ec3e1/source/fast/json.d#L1441
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 08:07:57 UTC, Martin Nowak wrote: On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: - Data size limited by available contiguous virtual memory Mmaping files for sequential reading is a very debatable choice, b/c the common use case is to read a file once. You should at least compare the numbers w/ drop_caches between each run. It's a sensible choice together with appropriate madvise().
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 08:20:33 UTC, Daniel N wrote: On Saturday, 17 October 2015 at 08:07:57 UTC, Martin Nowak wrote: On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: - Data size limited by available contiguous virtual memory Mmaping files for sequential reading is a very debatable choice, b/c the common use case is to read a file once. You should at least compare the numbers w/ drop_caches between each run. It's a sensible choice together with appropriate madvise(). Mmap is very expensive, as it affects all cores, you need a realistic multithreaded aync benchmark on smaller files to see the effect.
Re: Fastest JSON parser in the world is a D project
Am Sat, 17 Oct 2015 09:27:46 +0200 schrieb Sönke Ludwig : > Am 16.10.2015 um 17:11 schrieb Marco Leise: > > Am Thu, 15 Oct 2015 18:17:07 +0200 > > schrieb Sönke Ludwig : > > > >> (...) > >> Do you have a test case for your error? > > > > Well it is not an error. Rory originally wrote about > > conversions between "1" and 1 happening on the browser side. > > That would mean adding a quirks mode to any well-behaving JSON > > parser. In this case: "read numbers as strings". Hence I was > > asking if the data on the client could be fixed, e.g. the json > > number be turned into a string first before serialization. > > > > Okay, I obviously misread that as a once familiar issue. Maybe it indeed > makes sense to add a "JavaScript" quirks mode that behaves exactly like > a JavaScript interpreter would. Ok, but remember: https://www.youtube.com/watch?v=20BySC_6HyY And then think again. :D -- Marco
Re: Fastest JSON parser in the world is a D project
Am Sat, 17 Oct 2015 08:29:24 + schrieb Ola Fosheim Grøstad : > On Saturday, 17 October 2015 at 08:20:33 UTC, Daniel N wrote: > > On Saturday, 17 October 2015 at 08:07:57 UTC, Martin Nowak > > wrote: > >> On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise > >> wrote: > >>> - Data size limited by available contiguous virtual memory > >> > >> Mmaping files for sequential reading is a very debatable > >> choice, b/c the common use case is to read a file once. You > >> should at least compare the numbers w/ drop_caches between > >> each run. The results are: * The memory usage is then fixed at slightly more than the file size. (While it often stays below when using the disk cache.) * It would still be faster than copying the whole thing to a separate memory block. * Depending on whether the benchmark system uses a HDD or SSD, the numbers may be rendered meaningless by a 2 seconds wait on I/O. * Common case yes, but it is possible that you read JSON that had just been saved. > > It's a sensible choice together with appropriate madvise(). Obviously agreed :). Just that in practice (on my HDD system) it never made a difference in I/O bound sequential reads. So I removed posix_madvise. > Mmap is very expensive, as it affects all cores, you need a > realistic multithreaded aync benchmark on smaller files to see > the effect. That's valuable information. It is trivial to read into an allocated block when the file size is below a threshold. I would just need a rough file size. Are you talking about 4K pages or mega-bytes? 64 KiB maybe? -- Marco
Re: Fastest JSON parser in the world is a D project
Am 17.10.2015 um 13:16 schrieb Marco Leise: Am Sat, 17 Oct 2015 09:27:46 +0200 schrieb Sönke Ludwig : Okay, I obviously misread that as a once familiar issue. Maybe it indeed makes sense to add a "JavaScript" quirks mode that behaves exactly like a JavaScript interpreter would. Ok, but remember: https://www.youtube.com/watch?v=20BySC_6HyY And then think again. :D What about just naming it SerializationMode.WAT?
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 09:30:47 UTC, Marco Leise wrote: It is trivial to read into an allocated block when the file size is below a threshold. I would just need a rough file size. Are you talking about 4K pages or mega-bytes? 64 KiB maybe? Maybe, I guess you could just focus on what you think is the primary usage patterns for your library and benchmark those for different parameters. If you want to test processing of many small files combined with computationally/memory intensive tasks then you could try to construct a simple benchmark where you iterate over memory (M*cache 3 size) using a "realistic" pattern like brownian motion in N threads and also repeatedly/concurrently load JSON code for different file sizes so that the CPUs page table mechanisms are stressed by mmap, cache misses and (possibly) page faults.
Re: Fastest JSON parser in the world is a D project
Am Sat, 17 Oct 2015 11:12:08 + schrieb Ola Fosheim Grøstad : > […] you could try to construct a simple benchmark where you > iterate over memory (M*cache 3 size) using a "realistic" pattern > like brownian motion in N threads and also repeatedly/concurrently > load JSON code for different file sizes so that the CPUs page table > mechanisms are stressed by mmap, cache misses and (possibly) page > faults. O.O Are you kidding me? Just give me the correct value already. -- Marco
Re: Fastest JSON parser in the world is a D project
On Friday, 16 October 2015 at 10:08:06 UTC, Andrei Alexandrescu wrote: On 10/15/15 10:40 PM, Jacob Carlborg wrote: On 2015-10-15 14:51, Johannes Pfau wrote: Doesn't the GPL force everybody _using_ fast.json to also use the GPL license? Yes, it does have that enforcement. Then we'd need to ask Marco if he's willing to relicense the code with Boost. -- Andrei I've just crossed my fingers. Piotrek
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 13:09:45 UTC, Marco Leise wrote: Am Sat, 17 Oct 2015 11:12:08 + schrieb Ola Fosheim Grøstad : […] you could try to construct a simple benchmark where you iterate over memory (M*cache 3 size) using a "realistic" pattern like brownian motion in N threads and also repeatedly/concurrently load JSON code for different file sizes so that the CPUs page table mechanisms are stressed by mmap, cache misses and (possibly) page faults. O.O Are you kidding me? Just give me the correct value already. :-P
Re: Fastest JSON parser in the world is a D project
If this is the benchmark I'm remembering, the bulk of the time is spent parsing the floating point numbers. So it isn't a test of JSON parsing in general so much as the speed of scanf.
Re: Fastest JSON parser in the world is a D project
On 10/17/15 6:43 PM, Sean Kelly wrote: If this is the benchmark I'm remembering, the bulk of the time is spent parsing the floating point numbers. So it isn't a test of JSON parsing in general so much as the speed of scanf. In many cases the use of scanf can be replaced with drastically faster methods, as I discuss in my talks on optimization (including Brasov recently). I hope they'll release the videos soon. -- Andrei
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 16:14:01 UTC, Andrei Alexandrescu wrote: On 10/17/15 6:43 PM, Sean Kelly wrote: If this is the benchmark I'm remembering, the bulk of the time is spent parsing the floating point numbers. So it isn't a test of JSON parsing in general so much as the speed of scanf. In many cases the use of scanf can be replaced with drastically faster methods, as I discuss in my talks on optimization (including Brasov recently). I hope they'll release the videos soon. -- Andrei Oh absolutely. My issue with the benchmark is just that it claims to be a JSON parser benchmark but the bulk of CPU time is actually spent parsing floats. I'm on my phone though so perhaps this is a different benchmark--I can't easily check. The one I recall came up a year or so ago and was discussed on D.general.
Re: Fastest JSON parser in the world is a D project
Am Sat, 17 Oct 2015 16:27:06 + schrieb Sean Kelly : > On Saturday, 17 October 2015 at 16:14:01 UTC, Andrei Alexandrescu > wrote: > > On 10/17/15 6:43 PM, Sean Kelly wrote: > >> If this is the benchmark I'm remembering, the bulk of the time > >> is spent > >> parsing the floating point numbers. So it isn't a test of JSON > >> parsing > >> in general so much as the speed of scanf. > > > > In many cases the use of scanf can be replaced with drastically > > faster methods, as I discuss in my talks on optimization > > (including Brasov recently). I hope they'll release the videos > > soon. -- Andrei > > Oh absolutely. My issue with the benchmark is just that it claims > to be a JSON parser benchmark but the bulk of CPU time is > actually spent parsing floats. I'm on my phone though so perhaps > this is a different benchmark--I can't easily check. The one I > recall came up a year or so ago and was discussed on D.general. 1/4 to 1/3 of the time is spent parsing numbers in highly optimized code. You see that in a profiler the number parsing shows up on top, but the benchmark also exercises the structural parsing a lot. It is not a very broad benchmark though, lacking serialization, UTF-8 decoding, validation of results etc. I believe the author didn't realize how over time it became the go-to performance test. The author of RapidJSON has a very in-depth benchmark suite, but it would be a bit of work to get something non-C++ integrated: https://github.com/miloyip/nativejson-benchmark It includes conformance tests as well. -- Marco
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: JSON parsing in D has come a long way, especially when you look at it from the efficiency angle as a popular benchmark does that has been forked by well known D contributers like Martin Nowak or Sönke Ludwig. [...] Slightly OT: You have a std.simd file in your repo, was this written by you or is there a current std.simd proposal that I'm unaware of?
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 09:35:47 UTC, Sönke Ludwig wrote: Am 17.10.2015 um 13:16 schrieb Marco Leise: Am Sat, 17 Oct 2015 09:27:46 +0200 schrieb Sönke Ludwig : Okay, I obviously misread that as a once familiar issue. Maybe it indeed makes sense to add a "JavaScript" quirks mode that behaves exactly like a JavaScript interpreter would. Ok, but remember: https://www.youtube.com/watch?v=20BySC_6HyY And then think again. :D What about just naming it SerializationMode.WAT? At the very least that needs to be an undocumented alias easter egg. :)
Re: Fastest JSON parser in the world is a D project
Am Sun, 18 Oct 2015 03:40:52 + schrieb rsw0x : > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: > > JSON parsing in D has come a long way, especially when you look > > at it from the efficiency angle as a popular benchmark does > > that has been forked by well known D contributers like Martin > > Nowak or Sönke Ludwig. > > > > [...] > > Slightly OT: > You have a std.simd file in your repo, was this written by you or > is there a current std.simd proposal that I'm unaware of? Manu wrote that back in the days with the idea that it would help writing portable SIMD code on many architectures: https://github.com/TurkeyMan/simd Working in the 3D visualization business and having held at least one talk about SIMD it was no coincidence that he was interested in better vector math support. Inclusion into Phobos was planned. DMD needs some upgrading of the somewhat ad hoc SIMD intrinsic implementation though: https://issues.dlang.org/buglist.cgi?keywords=SIMD&resolution=--- Many instructions cannot be expressed outside of inline assembly which doesn't inline. -- Marco
Re: Fastest JSON parser in the world is a D project
On Saturday, 17 October 2015 at 16:27:08 UTC, Sean Kelly wrote: Oh absolutely. My issue with the benchmark is just that it claims to be a JSON parser benchmark but the bulk of CPU time is actually spent parsing floats. Well, most of such language-comparison benchmarks are just for fun/marketing. In the real world big JSON files would be compressed and most likely retrieved over a network connection (like a blob from a database). Pull-parsing of mmap'ed memory is a rather unusual scenario for JSON.
Re: Fastest JSON parser in the world is a D project
Am 16.10.2015 um 18:04 schrieb Marco Leise: Every value that is read (as opposed to skipped) is validated according to RFC 7159. That includes UTF-8 validation. Full validation (i.e. readJSONFile!validateAll(…);) may add up to 14% overhead here. Nice! I see you are using bitmasking trickery in multiple places. stdx.data.json is mostly just the plain lexing algorithm, with the exception of whitespace skipping. It was already very encouraging to get those benchmark numbers that way. Good to see that it pays off to go further.
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote: The test is pretty simple: Parse a JSON object, containing an array of 1_000_000 3D coordinates in the range [0..1) and average them. The performance of std.json in parsing those was horrible still in the DMD 2.066 days*: DMD : 41.44s, 934.9Mb Gdc : 29.64s, 929.7Mb Python : 12.30s, 1410.2Mb Ruby: 13.80s, 2101.2Mb Then with 2.067 std.json got a major 3x speed improvement and rivaled the popular dynamic languages Ruby and Python: DMD : 13.02s, 1324.2Mb In the mean time several other D JSON libraries appeared with varying focus on performance or API: Medea : 56.75s, 1753.6Mb (GDC) libdjson : 24.47s, 1060.7Mb (GDC) stdx.data.json: 2.76s, 207.1Mb (LDC) Yep, that's right. stdx.data.json's pull parser finally beats the dynamic languages with native efficiency. (I used the default options here that provide you with an Exception and line number on errors.) A few days ago I decided to get some practical use out of my pet project 'fast' by implementing a JSON parser myself, that could rival even the by then fastest JSON parser, RapidJSON. The result can be seen in the benchmark results right now: https://github.com/kostya/benchmarks#json fast: 0.34s, 226.7Mb (GDC) RapidJSON: 0.79s, 687.1Mb (GCC) (* Timings from my computer, Haswell CPU, Linux Very impressive. Is this not quite interesting ? Such a basic web back end operation, and yet it's a very different picture from those who say that one is I/O or network bound. I already have JSON files of a couple of gig, and they're only going to be bigger over time, and this is a more generally interesting question. Seems like you now get 2.1 gigbytes/sec sequential read from a cheap consumer SSD today...
Re: Fastest JSON parser in the world is a D project
On Monday, 19 October 2015 at 07:48:16 UTC, Sönke Ludwig wrote: Am 16.10.2015 um 18:04 schrieb Marco Leise: Every value that is read (as opposed to skipped) is validated according to RFC 7159. That includes UTF-8 validation. Full validation (i.e. readJSONFile!validateAll(…);) may add up to 14% overhead here. Nice! I see you are using bitmasking trickery in multiple places. stdx.data.json is mostly just the plain lexing algorithm, with the exception of whitespace skipping. It was already very encouraging to get those benchmark numbers that way. Good to see that it pays off to go further. Is there any chance that new json parser can be include in next versions of vibed? And what need to including its to Phobos?
Re: Fastest JSON parser in the world is a D project
On Wednesday, 21 October 2015 at 04:17:19 UTC, Laeeth Isharc wrote: Seems like you now get 2.1 gigbytes/sec sequential read from a cheap consumer SSD today... Not many consumer drives give more than 500-600 MB/s (SATA3 limit) yet. There are only a couple that I know of that reach 2000 MB/s, like Samsung's SM951, and they're generally a fair bit more expensive than what most consumers tend to buy (but at about $1 / GB, still affordable for businesses certainly).
Re: Fastest JSON parser in the world is a D project
On Wednesday, October 21, 2015 06:36:31 Suliman via Digitalmars-d-announce wrote: > On Monday, 19 October 2015 at 07:48:16 UTC, Sönke Ludwig wrote: > > Am 16.10.2015 um 18:04 schrieb Marco Leise: > >> Every value that is read (as opposed to skipped) is validated > >> according to RFC 7159. That includes UTF-8 validation. Full > >> validation (i.e. readJSONFile!validateAll(…);) may add up to > >> 14% overhead here. > >> > > > > Nice! I see you are using bitmasking trickery in multiple > > places. stdx.data.json is mostly just the plain lexing > > algorithm, with the exception of whitespace skipping. It was > > already very encouraging to get those benchmark numbers that > > way. Good to see that it pays off to go further. > > Is there any chance that new json parser can be include in next > versions of vibed? And what need to including its to Phobos? It's already available on code.dlang.org: http://code.dlang.org/packages/std_data_json For it to get into Phobos, it has to get through the review process and be voted in. It was put up for formal review two or three months ago, but that didn't get to the point that it was voted on (I assume that there was more work that needed to be done on it first; I haven't really read through that thread though, so I don't know - I was too busy when the review started to get involved in it). So, whatever needs to be done for it to be ready for a formal vote needs to be done, and then it can be voted in, but all of that takes time, so if you want to use it soon, you might as well just grab it from code.dlang.org - and it will make it so that you're in a better position to give feedback on it as well so that it will be that much better if/when it makes it into Phobos. - Jonathan M Davis
Re: Fastest JSON parser in the world is a D project
> Nice! I see you are using bitmasking trickery in multiple > places. stdx.data.json is mostly just the plain lexing > algorithm, with the exception of whitespace skipping. It was > already very encouraging to get those benchmark numbers that > way. Good to see that it pays off to go further. Is there any chance that new json parser can be include in next versions of vibed? And what need to including its to Phobos? It's already available on code.dlang.org: http://code.dlang.org/packages/std_data_json Jonatan, I mean https://github.com/mleise/fast :)
Re: Fastest JSON parser in the world is a D project
On Wednesday, 21 October 2015 at 09:59:09 UTC, Kapps wrote: On Wednesday, 21 October 2015 at 04:17:19 UTC, Laeeth Isharc wrote: Seems like you now get 2.1 gigbytes/sec sequential read from a cheap consumer SSD today... Not many consumer drives give more than 500-600 MB/s (SATA3 limit) yet. There are only a couple that I know of that reach 2000 MB/s, like Samsung's SM951, and they're generally a fair bit more expensive than what most consumers tend to buy (but at about $1 / GB, still affordable for businesses certainly). Yes - that's the one I had in mind. It's not dirt cheap, but at GBP280 if you have some money and want speed, the price is hardly an important factor. I should have said consumer grade rather than consumer, but anyway you get my point. That's today, in 2015. Maybe one can do even better than that by striping data, although it sounds like it's not that easy, but still. "The future is here already; just unevenly distributed". Seems like if you're processing JSON, which is not the most difficult task one might reasonably want to be doing, then CPU+memory is the bottleneck more than the SSD. I don't know what outlook is for drive speeds (except they probably won't go down), but data sets are certainly not shrinking. So I am intrigued by the difference between what people say is typical and what seems to be the case, certainly in what I want to do.
Re: Fastest JSON parser in the world is a D project
Could anybody reddit this benchmark?
Re: Fastest JSON parser in the world is a D project
On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote: Could anybody reddit this benchmark? done https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/
Re: Fastest JSON parser in the world is a D project
Am Wed, 21 Oct 2015 17:00:39 + schrieb Suliman : > >> > Nice! I see you are using bitmasking trickery in multiple > >> > places. stdx.data.json is mostly just the plain lexing > >> > algorithm, with the exception of whitespace skipping. It was > >> > already very encouraging to get those benchmark numbers that > >> > way. Good to see that it pays off to go further. > >> > >> Is there any chance that new json parser can be include in > >> next versions of vibed? And what need to including its to > >> Phobos? > > > > It's already available on code.dlang.org: > > http://code.dlang.org/packages/std_data_json > > > Jonatan, I mean https://github.com/mleise/fast :) That's nice, but it has a different license and I don't think Phobos devs would be happy to see all the inline assembly I used and duplicate functionality like the number parsing and UTF-8 validation and missing range support. -- Marco
Re: Fastest JSON parser in the world is a D project
Am Wed, 21 Oct 2015 04:17:16 + schrieb Laeeth Isharc : > Very impressive. > > Is this not quite interesting ? Such a basic web back end > operation, and yet it's a very different picture from those who > say that one is I/O or network bound. I already have JSON files > of a couple of gig, and they're only going to be bigger over > time, and this is a more generally interesting question. > > Seems like you now get 2.1 gigbytes/sec sequential read from a > cheap consumer SSD today... You have this huge amount of Reddit API JSON, right? I wonder if your processing could benefit from the fast skipping routines or even reading it as "trusted JSON". -- Marco
Re: Fastest JSON parser in the world is a D project
On Wednesday, 21 October 2015 at 22:24:30 UTC, Marco Leise wrote: Am Wed, 21 Oct 2015 04:17:16 + schrieb Laeeth Isharc : Very impressive. Is this not quite interesting ? Such a basic web back end operation, and yet it's a very different picture from those who say that one is I/O or network bound. I already have JSON files of a couple of gig, and they're only going to be bigger over time, and this is a more generally interesting question. Seems like you now get 2.1 gigbytes/sec sequential read from a cheap consumer SSD today... You have this huge amount of Reddit API JSON, right? I wonder if your processing could benefit from the fast skipping routines or even reading it as "trusted JSON". The couple of gig were just Quandl metadata for one provider, but you're right I have that Reddit data too. And that's just a beginning. What some have been doing for a while, I'm beginning to do now, and many others will be doing in the next few years - just as soon as they have finished having meetings about what to do... I don't suppose they'll be using python, at least not for long. I am sure it could benefit - I kind of need to get some other parts going first. (For once it truly is a case of Knuth's 97%). But I'll be coming back to look at best way, for json, but text files more generally. Have you thought about writing up your experience with writing fast json? A bit like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd. And actually if you have time, would you mind dropping me an email? laeeth at kaledicassociates.com Thanks. Laeeth.
Re: Fastest JSON parser in the world is a D project
On 10/21/2015 04:38 PM, Laeeth Isharc wrote: On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote: Could anybody reddit this benchmark? done https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/ Getting good press. Congratulations! -- Andrei
Re: Fastest JSON parser in the world is a D project
On 10/21/2015 1:38 PM, Laeeth Isharc wrote: On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote: Could anybody reddit this benchmark? done https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/ It's item 9 on the front page of https://news.ycombinator.com/ too! Link to actual article (don't click on this link, or your upvote will not be counted): https://news.ycombinator.com/item?id=10430951
Re: Fastest JSON parser in the world is a D project
On 10/21/2015 3:40 PM, Laeeth Isharc wrote: Have you thought about writing up your experience with writing fast json? A bit like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd. Yes, Marco, please. This would make an awesome article, and we need articles like that! You've already got this: https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489 so most of it is already written.
Re: Fastest JSON parser in the world is a D project
On 10/21/2015 12:17 AM, Laeeth Isharc wrote: I already have JSON files of a couple of gig, and they're only going to be bigger over time, Geez, if they're that big, is JSON really the best format to be using?
Re: Fastest JSON parser in the world is a D project
On 10/22/2015 09:08 AM, Walter Bright wrote: On 10/21/2015 1:38 PM, Laeeth Isharc wrote: On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote: Could anybody reddit this benchmark? done https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/ It's item 9 on the front page of https://news.ycombinator.com/ too! This has been a homerun. Congratulations for this work and also for publicizing it! (Consider it might have remained just one forum discussion read by all of 80 persons...) -- Andrei
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 17:35:48 UTC, Nick Sabalausky wrote: On 10/21/2015 12:17 AM, Laeeth Isharc wrote: I already have JSON files of a couple of gig, and they're only going to be bigger over time, Geez, if they're that big, is JSON really the best format to be using? Of course not. I don't much like JSON myself anyway. But I am not in control of the format it arrives in. Obviously I will eventually pick something better for the existing dump. But that two gig will be updated at least every week, and that's just one provider of 20 plus (mostly smaller). I have 2 Tb of Reddit JSON too. A one off conversion job, but there will be others.
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei Alexandrescu wrote: On 10/22/2015 09:08 AM, Walter Bright wrote: On 10/21/2015 1:38 PM, Laeeth Isharc wrote: On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote: Could anybody reddit this benchmark? done https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/ It's item 9 on the front page of https://news.ycombinator.com/ too! This has been a homerun. Congratulations for this work and also for publicizing it! (Consider it might have remained just one forum discussion read by all of 80 persons...) -- Andrei We really do need to stop hiding our light under a bushel. Thinking in marketing terms doesn't always come easy to technically minded people, and I understand why, but ultimately the community benefits a great deal from people becoming aware of the very real benefits D has to offer (alas people won't just get it, even if you think they should), and there are personal career benefits too from helping communicate how you have applied D to do useful work. It's hard to find great programmers and showing what you can do will pay off over time.
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc wrote: We really do need to stop hiding our light under a bushel. Thinking in marketing terms doesn't always come easy to technically minded people, and I understand why, but ultimately the community benefits a great deal from people becoming aware of the very real benefits D has to offer (alas people won't just get it, even if you think they should), and there are personal career benefits too from helping communicate how you have applied D to do useful work. It's hard to find great programmers and showing what you can do will pay off over time. Yeah, we don't want to repeat Lisp's mistake.
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc wrote: On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei Alexandrescu wrote: On 10/22/2015 09:08 AM, Walter Bright wrote: [...] This has been a homerun. Congratulations for this work and also for publicizing it! (Consider it might have remained just one forum discussion read by all of 80 persons...) -- Andrei We really do need to stop hiding our light under a bushel. Thinking in marketing terms doesn't always come easy to technically minded people, and I understand why, but ultimately the community benefits a great deal from people becoming aware of the very real benefits D has to offer (alas people won't just get it, even if you think they should), and there are personal career benefits too from helping communicate how you have applied D to do useful work. It's hard to find great programmers and showing what you can do will pay off over time. D has no well defined area to be used in. Everyone knows D, when written in a very specific C-mimicking way, is performant. But nobody is using C# or Scala or Python for performance.
Re: Fastest JSON parser in the world is a D project
Am Thu, 22 Oct 2015 06:10:56 -0700 schrieb Walter Bright : > On 10/21/2015 3:40 PM, Laeeth Isharc wrote: > > Have you thought about writing up your experience with writing fast json? > > A bit > > like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd. > > Yes, Marco, please. This would make an awesome article, and we need articles > like that! > > You've already got this: > > https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489 > > so most of it is already written. There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) -- Marco
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 20:54:01 UTC, Marco Leise wrote: Am Thu, 22 Oct 2015 06:10:56 -0700 schrieb Walter Bright : On 10/21/2015 3:40 PM, Laeeth Isharc wrote: > Have you thought about writing up your experience with > writing fast json? A bit like Walter's Dr Dobbs's article > on wielding a profiler to speed up dmd. Yes, Marco, please. This would make an awesome article, and we need articles like that! You've already got this: https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489 so most of it is already written. There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) That's why we need an official D blog. Perhaps you could publish it in TWID.
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 20:54:01 UTC, Marco Leise wrote: Am Thu, 22 Oct 2015 06:10:56 -0700 schrieb Walter Bright : On 10/21/2015 3:40 PM, Laeeth Isharc wrote: > Have you thought about writing up your experience with > writing fast json? A bit like Walter's Dr Dobbs's article > on wielding a profiler to speed up dmd. Yes, Marco, please. This would make an awesome article, and we need articles like that! You've already got this: https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489 so most of it is already written. There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) The main D forum is as good a place as any. Just start a thread there.
Re: Fastest JSON parser in the world is a D project
On 10/22/2015 1:53 PM, Marco Leise wrote: There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) You can publish it on my site: http://digitalmars.com/articles/index.html But I highly recommend that you create your own web site. It's great for your professional career.
Re: Fastest JSON parser in the world is a D project
On 10/22/2015 9:29 PM, Joakim wrote: The main D forum is as good a place as any. Just start a thread there. No, articles should be more than postings.
Re: Fastest JSON parser in the world is a D project
On 2015-10-22 22:53, Marco Leise wrote: There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) You could have a look at this blog implementation by Dicebot [1]. You still need to host it though. [1] https://github.com/Dicebot/mood -- /Jacob Carlborg
Re: Fastest JSON parser in the world is a D project
On Friday, 23 October 2015 at 19:48:31 UTC, Jacob Carlborg wrote: On 2015-10-22 22:53, Marco Leise wrote: There is at least one hurdle. I don't have a place to publish articles, no personal blog or site I contribute articles to and I don't feel like creating a one-shot one right now. :) You could have a look at this blog implementation by Dicebot [1]. You still need to host it though. [1] https://github.com/Dicebot/mood Mood is very nice, and I plan on using it in the medium term (made a pull request so it would compile using gdc or ldc). But you might want to wait a little while as you want a blog to be stable, and I think there is a problem with segfaulting right now - perhaps to do with the caching of posts, although it shouldn't be hard either to fix that or rewrite it your own way (as I started doing). It's worth setting one up though - what you use doesn't matter (look at Nikola or one of the other static site generators) - and Walter is right.
Re: Fastest JSON parser in the world is a D project
On Thursday, 22 October 2015 at 20:10:36 UTC, rsw0x wrote: On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc wrote: On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei Alexandrescu wrote: On 10/22/2015 09:08 AM, Walter Bright wrote: [...] This has been a homerun. Congratulations for this work and also for publicizing it! (Consider it might have remained just one forum discussion read by all of 80 persons...) -- Andrei We really do need to stop hiding our light under a bushel. Thinking in marketing terms doesn't always come easy to technically minded people, and I understand why, but ultimately the community benefits a great deal from people becoming aware of the very real benefits D has to offer (alas people won't just get it, even if you think they should), and there are personal career benefits too from helping communicate how you have applied D to do useful work. It's hard to find great programmers and showing what you can do will pay off over time. D has no well defined area to be used in. Everyone knows D, when written in a very specific C-mimicking way, is performant. But nobody is using C# or Scala or Python for performance. You reply to my post, but I don't entirely see how it relates. D is very flexible, and that's its virtue. Because splitting a codebase across multiple languages does have a cost, even if it's often worth paying the cost in order to use the right till for the job when those tools are by their nature specialised. I don't think everyone knows D is performant, and I wouldn't say fast JSON is written in a C mimicking way, taken as a whole. Choices are based on making trade-offs, and the relevant data are not static, but constantly shifting. When an SSD in 2015 that isn't especially pricey gives 2.1 Gig a sec throughput and one has many terabytes of text data a month to get through, and that's today and datasets keep growing and what I write today may be in use for years then the right decision will be a very different one to that five years ago. That's not just my perception, but those in other fields where the problems are similar - bioinformatics and advertising data being some of the many others. AdRoll is known for their Python work, but their data scientists use D. And my point, which you didn't really reply to, is that as a community we should do a bit more to share our experiences on how D can be useful in doing real work. As Walter observes, that's also something that pays off personally too.
Re: Fastest JSON parser in the world is a D project
On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: Example: double x = 0, y = 0, z = 0; auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, "y": 2, "z": 3 }, … ] }`); foreach (idx; json.coordinates) { // Provide one function for each key you are interested in json.keySwitch!("x", "y", "z")( { x += json.read!double; }, { y += json.read!double; }, { z += json.read!double; } ); } How can `coordinates` member be known at compile-time when the input argument is a run-time string?
Re: Fastest JSON parser in the world is a D project
On Monday, 26 October 2015 at 20:04:33 UTC, Nordlöw wrote: On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote: Example: double x = 0, y = 0, z = 0; auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, "y": 2, "z": 3 }, … ] }`); foreach (idx; json.coordinates) { // Provide one function for each key you are interested in json.keySwitch!("x", "y", "z")( { x += json.read!double; }, { y += json.read!double; }, { z += json.read!double; } ); } How can `coordinates` member be known at compile-time when the input argument is a run-time string? I suspect through the opDispatch operator overload. http://dlang.org/operatoroverloading.html#dispatch
Re: Fastest JSON parser in the world is a D project
On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote: How can `coordinates` member be known at compile-time when the input argument is a run-time string? I suspect through the opDispatch operator overload. http://dlang.org/operatoroverloading.html#dispatch Yikes, this is such an anti-pattern. https://github.com/rejectedsoftware/vibe.d/issues/634
Re: Fastest JSON parser in the world is a D project
On Tuesday, 27 October 2015 at 14:00:07 UTC, Martin Nowak wrote: On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote: How can `coordinates` member be known at compile-time when the input argument is a run-time string? I suspect through the opDispatch operator overload. http://dlang.org/operatoroverloading.html#dispatch Yikes, this is such an anti-pattern. https://github.com/rejectedsoftware/vibe.d/issues/634 Heh - yeah it is quite problematic. The only time I've needed to use it was when I was reading in Json with some structure like { [ { "timestamp" : { ... timestamp info ... }, "info1" : { ... info ...}, "info2" : { ... info ...}, . . "info 23" : { ... info ...} }, { < more of the above >} ] } and I wanted to be able get a Json[timestamp] map, where the Json is either a info1, info2 etc etc. I didn't want to write 23 different functions "hash_info1", "hash_info2" etc etc. So, opDispatch! Basically I wanted to hash the timestamp and some data. My opDispatch became: @ignore auto opDispatch(string name)(){ static assert(name.startsWith("hash_"), "Error, use StatHosts.hash_XYZ to gather XYZ[timestamp] info"); static assert(name.length > 5); enum dataName = name[5..$]; typeof(mixin("StatDetail."~dataName))[StatTimestampDetail] data; foreach(stat; statistics){ data[stat.timestamp] = mixin("stat."~dataName); } return data; } 23 functions merged into 1... The static assert reduces the number of places it can break things at least, still some weird things can happen but for the most part it's ok. So yes - opDispatch is cool but should be used VERY sparingly.
Re: Fastest JSON parser in the world is a D project
On Wednesday, 28 October 2015 at 11:26:59 UTC, wobbles wrote: So yes - opDispatch is cool but should be used VERY sparingly. I just had a thought, I could check if dataName is in [__traits(allMembers ... )]. That would at least ensure I'm referencing something that exists. Maybe that'd be useful in vibes Bson/Json code. (Except the opposite, you want to check you're referencing something that DOESN'T exist, so you can be sure it's not 'remove' for example).