On Tue, Mar 16, 2010 at 9:43 PM, dreamcat four <dreamc...@gmail.com> wrote: > And remember, > > Its not just the number of times its send to ICU for conversion. Its > also the number of times your UTF-16 string has to be converted back > into utf-8 afterwards. This is why Apple makes its utf-16 strings > immutable. So they are read-only, and the utf-8 representation can be > cached afterward. > > Think of it this way: > > 1. Load a utf-8 string from DB or file > 2. Convert it to utf-16 > 3. Perform ICU conv 3-5 times > 4. Page gets hit by memcache > 5. utf-16 is converted back to utf-8 > 6. Something changes > ? String was cached ? > 7. need to spit out another utf-8 version of the string again > > And a persistent web application can be held for many hours in memory. > Are we converting back to utf-8 every time? Then it might be better to > wrap the string conversions just around ICU. > > I'd suggest selecting a real (but still as easy-to-work with as can be > found) unicode php app. One that has been written to use a unicode php > module. Then getting a single, representative page from it. By that I > mean the kind of page that gets accessed the most. So for imdb that > would be a movie's page, etc. The smalled 'slice' of the app, not the > whole thing. Dummy-out the other stuff. > > Then convert that part (for rendering one page) into the current php6 > unicode scheme. And can see what's what. > I would choose mediawiki software for this kind of test, it works in a really internationalized environment, plus I did see posting/contributing the main developer of the mediawiki/wikipedia application on the mailing list.
But that's just my two cents. Tyrael > > > On Tue, Mar 16, 2010 at 8:04 PM, Ferenc Kovacs <tyr...@gmail.com> wrote: >> On Tue, Mar 16, 2010 at 8:05 PM, Stanislav Malyshev <s...@zend.com> wrote: >>> Hi! >>> >>>> On disk storage should probably be UTF-8 without any question? Windows >>>> use of widestrings for some files simple doubles up the on disk storage >>> >>> As file content, it's OK (an it'd be easy to add option to specify content >>> transformation if we wanted), but prescribing filenames as UTF-8 would >>> probably be not workable, since different systems (and maybe even different >>> filesystems inside same OS?) can have different opinions on that. >>> >>>> '3' is not a very processor friendly number, so working with 4 even >>>> though wasteful on memory, does make perfect sense. How long is it since >>> >>> I'm not sure it does. Most of PHP strings are short, so memory loss would be >>> very significant. Also, take into account that CPU caches aren't as big as >>> the main memory, and not fitting your data into the cache is expensive. >>> >>>> we had a 640k limit on working memory? SERVERS should have a good amount >>> >>> It doesn't matter how much memory you have, in numbers. Until we find an >>> unlimited source of computer memory left by the aliens in Himalayas, memory >>> costs money. It doesn't matter how much memory do you have - however many >>> gigs you have, you'll be able to run 3 times less PHP processes in new >>> version on the same hardware than in old version, which means new PHP would >>> cost you more to run. "Memory is cheap" is a very misunderstood expression - >>> it's only cheap if you always have much more than you need. >>> >>>> Probably 90% of the time a string will come in and go out without >>>> requiring any processing at all, so leave it as UTF-8 ? The only time we >>> >>> It might be great if we could do that. The problem might be that right now >>> AFAIK we don't have a good library to work with utf-8 strings (please >>> correct me if I'm wrong here). >> http://source.icu-project.org/repos/icu/icuhtml/trunk/design/strings/icu_utf8.html >> from ICU 3.6 changelog => The UTF-8 transformation functions and >> macros are faster. >> from 4.2 => UTF-8 friendly internal data structure for Unicode data lookup >> so it's seems that guys at ICU tries to close the gap between the >> UTF-16 and UTF-8 performance, so maybe it would be a good idea, to >> check out the current situation. >> >> Tyrael >>> -- >>> Stanislav Malyshev, Zend Software Architect >>> s...@zend.com http://www.zend.com/ >>> (408)253-8829 MSN: s...@zend.com >>> >>> -- >>> PHP Internals - PHP Runtime Development Mailing List >>> To unsubscribe, visit: http://www.php.net/unsub.php >>> >>> >> >> -- >> PHP Internals - PHP Runtime Development Mailing List >> To unsubscribe, visit: http://www.php.net/unsub.php >> >> > -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
