[Interest] Faster QXmlStreamWriter?

I'm looking at optimizing an applications save performance - it is too slow right now and causes too much of a freeze. We're using QXmlStreamWriter, here's the implementation . Profiling suggests that Qt's write element / attribu

Re: [Interest] Faster QXmlStreamWriter?

Il 22/04/2018 16:30, Vadim Peretokin ha scritto: A lot of time seems to be spent resizing the buffer, and unfortunately there doesn't seem to be a way to pre-set the QBuffer size to something reasonable to begin with. Anyone have tips? Didn't check the actual code, but you can always resize

Re: [Interest] Faster QXmlStreamWriter?

Thanks! That helped remove the resize operation from the profiling data, but QBuffer::writeData is still there, and the overall time didn't change much. Is there anything else I can look at, or is this a dead end then - requires a different serialization solution altogether (a lot of work)? On Su

Re: [Interest] Faster QXmlStreamWriter?

On Sunday, 22 April 2018 22:15:48 PDT Vadim Peretokin wrote: > Thanks! That helped remove the resize operation from the profiling data, > but QBuffer::writeData is still there, and the overall time didn't change > much. > > Is there anything else I can look at, or is this a dead end then - require

Re: [Interest] Faster QXmlStreamWriter?

On Mon, Apr 23, 2018 at 8:32 AM Thiago Macieira wrote: > On Sunday, 22 April 2018 22:15:48 PDT Vadim Peretokin wrote: > > Thanks! That helped remove the resize operation from the profiling data, > > but QBuffer::writeData is still there, and the overall time didn't change > > much. > > > > Is the

Re: [Interest] Faster QXmlStreamWriter?

On Monday, 23 April 2018 22:02:20 PDT Vadim Peretokin wrote: > > Correct, QBuffer is not optimised for speed. I ran into that while > > developing > > QCborStreamWriter. The solution was to bypass it for speed. > > OK, that's a pity. I'll look into using pugixml instead, and if anyone else > knows

Re: [Interest] Faster QXmlStreamWriter?

Am Tue, 24 Apr 2018 05:02:20 + schrieb Vadim Peretokin : > On Mon, Apr 23, 2018 at 8:32 AM Thiago Macieira > wrote: > > > On Sunday, 22 April 2018 22:15:48 PDT Vadim Peretokin wrote: > > > Thanks! That helped remove the resize operation from the profiling data, > > > but QBuffer::writeData

Re: [Interest] Faster QXmlStreamWriter?

On Tue, Apr 24, 2018 at 8:14 AM Thiago Macieira wrote: > On Monday, 23 April 2018 22:02:20 PDT Vadim Peretokin wrote: > > > Correct, QBuffer is not optimised for speed. I ran into that while > > > developing > > > QCborStreamWriter. The solution was to bypass it for speed. > > > > OK, that's a pi

Re: [Interest] Faster QXmlStreamWriter?

Google's protobuf come highly recommended. https://github.com/google/protobuf Sent: Tuesday, April 24, 2018 at 4:39 AM From: "Vadim Peretokin" To: "Thiago Macieira" Cc: interest@qt-project.org Subject: Re: [Interest] Faster QXmlStreamWriter?     On Tue, Apr 2

Re: [Interest] Faster QXmlStreamWriter?

24.04.2018, 11:40, "Vadim Peretokin" : > On Tue, Apr 24, 2018 at 8:14 AM Thiago Macieira > wrote: >> On Monday, 23 April 2018 22:02:20 PDT Vadim Peretokin wrote: >>> > Correct, QBuffer is not optimised for speed. I ran into that while >>> > developing >>> > QCborStreamWriter. The solution was t

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: > 24.04.2018, 11:40, "Vadim Peretokin" : > > On Tue, Apr 24, 2018 at 8:14 AM Thiago Macieira wrote: > >> On Monday, 23 April 2018 22:02:20 PDT Vadim Peretokin wrote: > >>> > Correct, QBuffer is not optimised for speed. I ran into th

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 08:52:53 PDT Thiago Macieira wrote: > > If your serialized data is intended to be read by Qt applications only, > > QDataStream may be a good choice > > Also slow. > > Binary QJsonDocument is the fastest, followed closely by QCborValue (new in > 5.12). Followed by regul

Re: [Interest] Faster QXmlStreamWriter?

2018-04-24 17:52 GMT+02:00 Thiago Macieira : > On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: >> If your serialized data is intended to be read by Qt applications only, >> QDataStream may be a good choice > > Also slow. > > Binary QJsonDocument is the fastest, followed closely by

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 09:39:37 PDT Julien Cugnière wrote: > 2018-04-24 17:52 GMT+02:00 Thiago Macieira : > > On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: > >> If your serialized data is intended to be read by Qt applications only, > >> QDataStream may be a good choice > > >

Re: [Interest] Faster QXmlStreamWriter?

> On Tuesday, 24 April 2018 09:39:37 PDT Julien Cugnière wrote: > > 2018-04-24 17:52 GMT+02:00 Thiago Macieira : > > > On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: > > >> If your serialized data is intended to be read by Qt applications > > >> only, QDataStream may be a good cho

Re: [Interest] Faster QXmlStreamWriter?

Am Tue, 24 Apr 2018 08:52:53 -0700 schrieb Thiago Macieira : > On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: > > 24.04.2018, 11:40, "Vadim Peretokin" : > > > On Tue, Apr 24, 2018 at 8:14 AM Thiago Macieira > wrote: > > >> On Monday, 23 April 2018 22:02:20 PDT Vadim Peretok

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 22:46:43 PDT Maurice Kalinowski wrote: > > On Tuesday, 24 April 2018 09:39:37 PDT Julien Cugnière wrote: > > > 2018-04-24 17:52 GMT+02:00 Thiago Macieira : > > > > On Tuesday, 24 April 2018 07:39:08 PDT Konstantin Tokarev wrote: > > > >> If your serialized data is intended

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 22:48:03 PDT alexander golks wrote: > when using QJsonDocument you may/should pay attention to QtQTBUG-47629: > QJsonObject size maximum length 128MB > https://bugreports.qt.io/browse/QTBUG-47629 That's one of the reasons I'm adding CBOR support. The new implementation s

Re: [Interest] Faster QXmlStreamWriter?

On Tuesday, 24 April 2018 23:36:15 PDT Thiago Macieira wrote: > 3) use the examples/corelib/serialization/convert tool[1] to > convert to other formats > 4) use the same tool with the "null" output to benchmark reading[2] > 5) use the tool with binary JSON as source to benchmark writing Attached

Re: [Interest] Faster QXmlStreamWriter?

25.04.2018, 09:43, "Thiago Macieira" : > So Qt 6 QJsonDocument & family will have the 128 MB limit > removed, at the expense of the binary JSON format requiring parsing. That's sad, as we lose the single solution inside Qt for serialization without parsing > > -- > Thiago Macieira - thiago.maci

Re: [Interest] Faster QXmlStreamWriter?

On Wednesday, 25 April 2018 07:13:18 PDT Konstantin Tokarev wrote: > 25.04.2018, 09:43, "Thiago Macieira" : > > So Qt 6 QJsonDocument & family will have the 128 MB limit > > removed, at the expense of the binary JSON format requiring parsing. > > That's sad, as we lose the single solution inside Q

Re: [Interest] Faster QXmlStreamWriter?

On Wednesday, 25 April 2018 08:45:41 PDT Thiago Macieira wrote: > On Wednesday, 25 April 2018 07:13:18 PDT Konstantin Tokarev wrote: > > 25.04.2018, 09:43, "Thiago Macieira" : > > > So Qt 6 QJsonDocument & family will have the 128 MB limit > > > removed, at the expense of the binary JSON format req

Re: [Interest] Faster QXmlStreamWriter?

25.04.2018, 19:08, "Thiago Macieira" : > On Wednesday, 25 April 2018 08:45:41 PDT Thiago Macieira wrote: >>  On Wednesday, 25 April 2018 07:13:18 PDT Konstantin Tokarev wrote: >>  > 25.04.2018, 09:43, "Thiago Macieira" : >>  > > So Qt 6 QJsonDocument & family will have the 128 MB limit >>  > > re

Re: [Interest] Faster QXmlStreamWriter?

On 25/04/18 01:52, Thiago Macieira wrote: Binary QJsonDocument is the fastest, followed closely by QCborValue (new in 5.12). Is compatibility guaranteed on binary QJsonDocument between Qt versions? Can I safely exchange data both ways between current Qt 5.8 and future 5.x versions? Hamis

Re: [Interest] Faster QXmlStreamWriter?

On Friday, 27 April 2018 20:21:07 PDT Hamish Moffatt wrote: > On 25/04/18 01:52, Thiago Macieira wrote: > > Binary QJsonDocument is the fastest, followed closely by QCborValue > > (new in > > 5.12). > > Is compatibility guaranteed on binary QJsonDocument between Qt versions? Yes. > Can I safely

Re: [Interest] Faster QXmlStreamWriter?

Hi Thiago PS: YMMV, especially if you don't use CPU-optimised UTF-8 methods like I do. You need to compile your own Qt to get those. Could you please give more information about this? I thought sse2 code for utf8 is enabled by default at least in 64 bit build. What do I miss here? Regar

Re: [Interest] Faster QXmlStreamWriter?

I went with pugixml (https://pugixml.org) for anyone interested and the speed savings are worth it. It is also fairly easy to transition from qml to pugixml and as a bonus, it does not choke on non-printable characters. On Sun, Apr 29, 2018 at 10:58 PM Gunnar Roth wrote: > >Hi Thiago > > > > >PS