Re: [Development] State of "binary JSON" in 5.15+?
On Thursday, 15 April 2021 13:49:30 PDT Stottlemyer, Brett (B.S.) wrote: > No, your description is correct. Thanks for helping to clarify. > > Does this help Thiago? I'm not using JSON as an input to sending a QString > or QByteArray over the wire, I want to read a file from disk (quickly) and > find or extract specific elements from the document. Yes, it helps. No, your use-case won't be efficient the way you described. The in-memory representation of QJsonDocument has changed, so attempting to save and reloading as binary JSON will instead now make it worse. Just keep the QJsonDocument / QJsonValue that the thing was about, or keep the CBOR or JSON forms. The memory consumption of the CBOR-based in-memory format should be comparable, at least at first approximation, for reasonably-nested structures. The QCborContainerPrivate structure that is the backend of everything is designed to minimise the overhead and it also keeps your strings in US-ASCII as much as possible (most of your JSON object keys will be US-ASCII). If you REALLY need the binary format, you can try to use the qt5compat module in Qt 5. You may need to rename or namespace the class, since QtCore does export classes by the same name and you don't want an ODR violation. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel DPG Cloud Engineering ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] State of "binary JSON" in 5.15+?
On 4/15/21, 3:45 PM, "Development on behalf of Elvis Stansvik" wrote: With the risk of muddling things even more, but the way I understood > Think geographic data, where I can prefetch at low-priority, and load/process data on-demand as location changes. was that his use case was: 1. In the background, proactively fetch some JSON that might be needed soon, parse it and store it as "binary JSON". 2. When the time comes and the data is needed, it can be loaded fast from disk with fromBinaryJson. And with 5.14, step (2) was faster than loading from JSON text due to the mmapping, but in 5.15 it is slower (due to the change in change in in-memory format). It's quite possible I've misunderstood though. Elvis No, your description is correct. Thanks for helping to clarify. Does this help Thiago? I'm not using JSON as an input to sending a QString or QByteArray over the wire, I want to read a file from disk (quickly) and find or extract specific elements from the document. Thanks, Brett ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] State of "binary JSON" in 5.15+?
Den tors 15 apr. 2021 kl 21:17 skrev Thiago Macieira : > > On Thursday, 15 April 2021 07:36:08 PDT Stottlemyer, Brett (B.S.) wrote: > > On 4/14/21, 12:53 AM, "Development on behalf of Thiago Macieira" > > > > wrote: > > > No, that was it. I assume you're caching templates which you need to > > modify slightly for each reply, not plain-text JSON. > > > > Ahh, no. Sorry for not being clear. I'm the consumer of the REST API, not > > the producer. > > This made it even less clear. > > I was trying to draw the distinction between prepared JSON in a QString or > QByteArray and a template that you needed to fill in. For example, suppose you > need to send a POST with JSON data but you need to fill in some information in > that POST before sending, such as a URI or an ID. > > Because if the contents don't change, there's no need to use QJsonDocument, > binary or not, to cache the request or reply. You can do that in QString or > QByteArray form. With the risk of muddling things even more, but the way I understood > Think geographic data, where I can prefetch at low-priority, and load/process > data on-demand as location changes. was that his use case was: 1. In the background, proactively fetch some JSON that might be needed soon, parse it and store it as "binary JSON". 2. When the time comes and the data is needed, it can be loaded fast from disk with fromBinaryJson. And with 5.14, step (2) was faster than loading from JSON text due to the mmapping, but in 5.15 it is slower (due to the change in change in in-memory format). It's quite possible I've misunderstood though. Elvis > > > In Qt 5, the existing methods in QJsonDocument continue to work like > > they have done since 5.0. > > > > I think you are saying this with a different use-case in mind, and I'd like > > to understand your use-case. For my use-case, I disagree with your > > statement. Qt 5.14 could memcpy and cast a QByteArray into a QJsonDocument > > (untested, but by inspection of the Woboq code and the documentation saying > > binary JSON was "fast for mmap"). Qt 5.15 can't, although the > > fromBinaryJson API is still supported. > > The in-memory format of QJsonDocument and related classes has indeed changed. > But from your point of view that's not relevant. If you cache the prepared > template in the form of a QJsonDocument, the behaviour and performance should > be the same in either 5.14 or 5.15. > > But that also means there's no reason to talk about the binary format any > more. It's not a caching technique. > -- > Thiago Macieira - thiago.macieira (AT) intel.com > Software Architect - Intel DPG Cloud Engineering > > > > ___ > Development mailing list > Development@qt-project.org > https://lists.qt-project.org/listinfo/development ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] State of "binary JSON" in 5.15+?
On Thursday, 15 April 2021 07:36:08 PDT Stottlemyer, Brett (B.S.) wrote: > On 4/14/21, 12:53 AM, "Development on behalf of Thiago Macieira" > > wrote: > No, that was it. I assume you're caching templates which you need to > modify slightly for each reply, not plain-text JSON. > > Ahh, no. Sorry for not being clear. I'm the consumer of the REST API, not > the producer. This made it even less clear. I was trying to draw the distinction between prepared JSON in a QString or QByteArray and a template that you needed to fill in. For example, suppose you need to send a POST with JSON data but you need to fill in some information in that POST before sending, such as a URI or an ID. Because if the contents don't change, there's no need to use QJsonDocument, binary or not, to cache the request or reply. You can do that in QString or QByteArray form. > In Qt 5, the existing methods in QJsonDocument continue to work like > they have done since 5.0. > > I think you are saying this with a different use-case in mind, and I'd like > to understand your use-case. For my use-case, I disagree with your > statement. Qt 5.14 could memcpy and cast a QByteArray into a QJsonDocument > (untested, but by inspection of the Woboq code and the documentation saying > binary JSON was "fast for mmap"). Qt 5.15 can't, although the > fromBinaryJson API is still supported. The in-memory format of QJsonDocument and related classes has indeed changed. But from your point of view that's not relevant. If you cache the prepared template in the form of a QJsonDocument, the behaviour and performance should be the same in either 5.14 or 5.15. But that also means there's no reason to talk about the binary format any more. It's not a caching technique. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel DPG Cloud Engineering ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] State of "binary JSON" in 5.15+?
Apologies if this was already sent, I meant to send yesterday and found it in a still open window. On 4/14/21, 12:53 AM, "Development on behalf of Thiago Macieira" wrote: No, that was it. I assume you're caching templates which you need to modify slightly for each reply, not plain-text JSON. Ahh, no. Sorry for not being clear. I'm the consumer of the REST API, not the producer. Think geographic data, where I can prefetch at low-priority, and load/process data on-demand as location changes. In Qt 5, the existing methods in QJsonDocument continue to work like they have done since 5.0. I think you are saying this with a different use-case in mind, and I'd like to understand your use-case. For my use-case, I disagree with your statement. Qt 5.14 could memcpy and cast a QByteArray into a QJsonDocument (untested, but by inspection of the Woboq code and the documentation saying binary JSON was "fast for mmap"). Qt 5.15 can't, although the fromBinaryJson API is still supported. Under the covers (see https://codereview.qt-project.org/c/qt/qtbase/+/265312/37/src/corelib/serialization/qjsondocument.cpp#362), after the memcpy operations: Qt 5.14 -> return QJsonDocument(d); Qt 5.15 -> return d->toJsonDocument(); -> toJsonDocument() calls toJsonObject() (or toJsonArray()) -> toJsonObject() recurses over all of the elements (See here https://codereview.qt-project.org/c/qt/qtbase/+/265312/37/src/corelib/serialization/qbinaryjson.cpp#204) The result is that fromBinaryData() is significantly slower than parsing the JSON text directly, not significantly faster. This format is different, and I'm not sure what use-case it is better for... Thanks, Brett ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Nominating Andrei Golubev as Approver
Congratulations to Andrei, approver status has been granted. -- Alex From: Edward Welbourne Sent: Friday, 9 April 2021 14:07 To: Alex Blasche Subject: Re: Nominating Andrei Golubev as Approver On Thursday, 18 March 2021 15:00 I wrote: > I'd like to nominate Andrei Golubev as an Approver. A fortnight - and more - has passed. Eddy. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
