Given the network-centric world we live in, I usually think about networks, not
disks, because as you point out, there are a variety of faster-than-rotational
media - be RAM FS, flash FS or some other kind of mixed (SSD). Still the CPU is
orders of magnitude faster than anything but a ramdisk.
Still the whole-file-in memory issue I think is controlling because it limits
your concurrency, else you'll be thrashing through virtual memory (still
painful on a flash FS). Given that parallelism scales, it would seem benficial
to work with just a few kB at at time, that way you can scale across more
processors, disks or whole systems, while increasing fault tolerance.
________________________________
From: Atlant Schmidt <aschm...@dekaresearch.com>
To: 'Jason H' <scorp...@yahoo.com>; Tony Rietwyk <t...@rightsoft.com.au>;
"interest@qt-project.org" <interest@qt-project.org>
Sent: Tuesday, August 21, 2012 11:36 AM
Subject: RE: [Interest] memory fragmentation?
All:
> It can be very easily justified that many to Utf* calls are justified because
> of latency in the I/O system means it happens "for free" while reducing the
> overall latency.
In the “modern world”, be a little cautious when you decide what activities can
overlap with other activities. For example, for many of us in the Linux world
who are using Flash-based file systems (e.g., UBIfs), there’s really no such
thing as overlapping disk I/O with other activities; all “disk” I/O to the
Flash
file system requires the CPU and will block ordinary interactive priority
user-level activities that also need the CPU.
It’s not like in the old days when you queued an I/O request to a rotating
magnetic disk and between seek latencies and DMA data transfers, you
could slip-in a lot of computing. Nowadays, that “disk” I/O may still require
the CPU.
Atlant
From:interest-bounces+aschmidt=dekaresearch....@qt-project.org
[mailto:interest-bounces+aschmidt=dekaresearch....@qt-project.org] On Behalf Of
Jason H
Sent: Tuesday, August 21, 2012 10:17 AM
To: Tony Rietwyk; interest@qt-project.org
Subject: Re: [Interest] memory fragmentation?
Not true for really large XML. With the all-in-one approach, you have to have
the entire document completed. This will create a long pause, and increase
latency, and increase memory usage. It can be very easily justified that many
to Utf* calls are justified because of latency in the I/O system means it
happens "for free" while reducing the overall latency.
Would you prefer an image transcoding server that waited for an image to be
completely uploaded and completed, or start receiving the conversion as soon as
final blocks or scan lines were available?
________________________________
From:Tony Rietwyk <t...@rightsoft.com.au>
To: interest@qt-project.org
Sent: Monday, August 20, 2012 9:52 PM
Subject: Re: [Interest] memory fragmentation?
...
In the end I chose Qt 4.8.1 QXmlStreamWriter(QString *). The QByteArray
version seemed the logical choice - but since the API only uses QString, each
call resulted in temporaries being allocated and disposed. It seems QString is
crying out for an appendLatin1(const char *, int len) that re-allocates the
unicode array if necessary, and then does the latin1 conversion without
creating a temporary. Doing a single toUtf8 at the end is also better than
calling it a million times.
Click here to report this email as spam.
________________________________
This e-mail and the information, including any attachments, it contains are
intended to be a confidential communication only to the person or entity to
whom it is addressed and may contain information that is privileged. If the
reader of this message is not the intended recipient, you are hereby notified
that any dissemination, distribution or copying of this communication is
strictly prohibited. If you have received this communication in error, please
immediately notify the sender and destroy the original message.
Thank you.
Please consider the environment before printing this email.
_______________________________________________
Interest mailing list
Interest@qt-project.org
http://lists.qt-project.org/mailman/listinfo/interest
