Re: UCD in XML or in CSV? (was: Re: Unicode Digest, Vol 56, Issue 20)
On 31/08/18 08:25 Marius Spix via Unicode wrote: > > A good compromise between human readability, machine processability and > filesize would be using YAML. > > Unlike JSON, YAML supports comments, anchors and references, multiple > documents in a file and several other features. Thanks for advice. Already I do use YAML syntaxic highlighting to display XCompose files, that use the colon as a separator, too. Did you figure out how YAML would fit UCD data? It appears to heavily rely on line breaks, that may get lost as data turns around across environments. XML indentation is only a readability feature and irrelevant to content. The structure is independent of invisible characters and is stable if only graphics are not corrupted (while it may happen that they are). Linebreaks are odd in that they are inconsistent across OSes, because Unicode was denied the right to impose a unique standard in that matter. The result is mashed-up files, and I fear YAML might not hold out. Like XML, YAML needs to repeat attribute names in every instance. That is precisely what CSV gets around of, at the expense of readability in plain text. Personally I could use YAML as I do use XML for lookup in the text editor, but I’m afraid that there is no advantage over CSV with respect to file size. Regards, Marcel > > Regards, > > Marius Spix > > > On Fri, 31 Aug 2018 06:58:37 +0200 (CEST) Marcel Schneider via Unicode > wrote: > […]
Re: UCD in XML or in CSV?
On 31/08/18 19:59 Ken Whistler via Unicode wrote: […] > Second, one of the main obligations of a standards organization is > *stability*. People may well object to the ad hoc nature of the UCD data > files that have been added over the years -- but it is a *stable* > ad-hockery. The worst thing the UTC could do, IMO, would be to keep > tweaking formats of data files to meet complaints about one particular > parsing inconvenience or another. That would create multiple points of > discontinuity between versions -- worse than just having to deal with > the ongoing growth in the number of assigned characters and the > occasional addition of new data files and properties to the UCD. I did not want to make trouble asking for moving conventions back and forth. I liked to learn why UnicodeData.txt was released as a draft without a header and nothing, given Unicode knew well in advance that the scheme adopted at first release would be kept stable for decades or forever. Then I’d like to learn how Unicode came to not devise a consistent scheme for all the UCD files if any such could be devised, so that people could get able to assess whether complaints about inconsistencies are well-founded or not. It is not enough for me that a given adhockery is stable; IMO it should also be well-designed, in responsiveness facing history from a standards body. That is not what one is telling about UnicodeData.txt, although it is the only effectively formatted file in UCD for streamlined processing. Was there not enough time to think about a header line and a file header? With the header line it would be flexible, and all the problems would be solved if specifying that parsers should start with counting the field number prior to creating storage arrays. We are lacking a real history of Unicode, explaining why everybody was in a hurry. “Authors falling like flies” is the only hint that comes to mind. And given Unicode appear to have missed the hit, to discuss whether it would be time to add a more accomplished file for better usability. > > Keep in mind that there is more to processing the UCD than just > "latest". People who just focus on grabbing the very latest version of > the UCD and updating whatever application they have are missing half the > problem. There are multiple tools out there that parse and use multiple > *versions* of the UCD. That includes the tooling that is used to > maintain the UCD (which parses *all* versions), and the tooling that > creates UCD in XML, which also parses all versions. Then there is > tooling like unibook, to produce code charts, which also has to adapt to > multiple versions, and bidi reference code, which also reads multiple > versions of UCD data files. Those are just examples I know off the top > of my head. I am sure there are many other instances out there that fit > this profile. And none of the applications already built to handle > multiple versions would welcome having to permanently build in tracking > particular format anomalies between specific versions of the UCD. That point is clear to me, and even when suggesting to make changes to BidiMirrored.txt, I had alternatives with a stable existing file and a new enhanced file. But what is totally unclear to me is what are old versions doing in compiling latest data. Delta is OK, research on particular topic in old data is OK, but what does it mean to need to parse *all* versions to get newest products? > > Third, please remember that folks who come here complaining about the > complications of parsing the UCD are a very small percentage of a very > small percentage of a very small percentage of interested parties. > Nearly everybody who needs UCD data should be consuming it as a > secondary source (e.g. for reference via codepoints.net), or as a > tertiary source (behind specialized API's, regex, etc.), or as an end > user (just getting behavior they expect for characters in applications). > Programmers who actually *need* to consume the raw UCD data files and > write parsers for them directly should actually be able to deal with the > format complexity -- and, if anything, slowing them down to make them > think about the reasons for the format complexity might be a good thing, > as it tends to put the lie to the easy initial assumption that the UCD > is nothing more than a bunch of simple attributes for all the code points. That makes no sense to me. UCD raw data is and remains a primary source, I see no way to consume it as a secondary source or as a tertiary source. Do you mean to consume it via secondary or tertiary sources? Then we actually appear to consume those sources instead of UCD raw data. These sources are fine for the purpose of getting information about some particular code points, but most of these tools I remember don’t allow to filter values and compute overviews, nor to add data, as we can do it in spreadsheet software. Honestly are we so few people usi
Re: Private Use areas
Hi I have now found the following document. http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf William Overington Friday 31 August 2018 Original message >From : wjgo_10...@btinternet.com Date : 2018/08/31 - 21:43 (GMTDT) To : m...@kli.org, unicode@unicode.org Subject : Re: Private Use areas Hi Thank you for your posts from earlier today. Actually I learned about JSON yesterday and I am thinking that using JSON could well be a good idea. I found a helpful page with diagrams. http://www.json.org/ Although I hope that a format of recording information about the properties of particular uses of Private Use Area characters will become implemented as a practicality, and that that format can be applied in practice where desired, and indeed I would be happy to participate in a group project, I do not know enough about Unicode properties to play a major role or to lead such a project. William Overington Friday 31 August 2018
Re: Private Use areas
Hi Thank you for your posts from earlier today. Actually I learned about JSON yesterday and I am thinking that using JSON could well be a good idea. I found a helpful page with diagrams. http://www.json.org/ Although I hope that a format of recording information about the properties of particular uses of Private Use Area characters will become implemented as a practicality, and that that format can be applied in practice where desired, and indeed I would be happy to participate in a group project, I do not know enough about Unicode properties to play a major role or to lead such a project. William Overington Friday 31 August 2018
Re: Private Use areas
On 08/28/2018 04:26 AM, William_J_G Overington via Unicode wrote: Hi Mark E. Shoulson wrote: I'm not sure what the advantage is of using circled characters instead of plain old ascii. My thinking is that "plain old ascii" might be used in the text encoded in the file. Sometimes a file containing Private Use Area characters is a mix of regular Unicode Latin characters with just a few Private Use Area characters mixed in with them. So my suggestion of using circled characters is for disambiguation purposes. The circled characters in the PUAINFO sequence would not be displayed if a special software program were being used to read in the text file, then act upon the information that is encoded using the circled characters. What if circled characters are used in the text encoded in the file? They're characters too, people use them and all. Whenever you designate some characters to be used in a way outside their normal meaning, you have the problem of how to use them *with* their normal meaning. So there are various escaping schemes and all. So in XML, all characters have their normal meanings—except <, >, and &, which mean something special and change the interpretations of other nearby characters (so "bold" is a word in English that appears in the text, but "" is part of an instruction to the renderer that doesn't appear in the text.) And the price is that those three characters have to be expressed differently (< > &). I don't really see what you gain by branding some large swath of unicode ("circled characters") as "special" and not meaning their usual selves, and for that matter making these hard-to-type characters *necessary* for using your scheme, when you could do something like what XML does, and say "everything between < and > is to be interpreted specially, and there, these characters have the following meanings" and then have some other way of expressing those two reserved characters. (not saying you need to do it XML's way, but something like that: reserve a small number of characters that have to be escaped, not some huge chunk.) My thinking is that using this method just adds some encoded information at the start of the text file and does not require the whole document to become designated as a file conformant to a particular markup format. That's another way of saying that this is a markup format which accepts a large variety of plain texts. Because you ARE talking about making a "particular markup format," just a different and new one. I guess there's not even any reason for me to argue the point, though, since it is up to you how to design your markup language, and you can take advice (or not) from anyone you like. Draw up some design, find some interested people, start a discussion, and work it out. (but not here; this list is for discussing Unicode.) ~mark
Re: Private Use areas
On 08/28/2018 11:58 AM, William_J_G Overington via Unicode wrote: Asmus Freytag wrote: There are situations where an ad-hoc markup language seems to fulfill a need that is not well served by the existing full-fledged markup languages. You find them in internet "bulletin boards" or services like GitHub, where pure plain text is too restrictive but the required text styles purposefully limited - which makes the syntactic overhead of a full-featured mark-up language burdensome. I am thinking of such an ad-hoc special purpose markup language. I am thinking of something like a special purpose version of the FORTH computer language being used but with no user definitions, no comparison operations and no loops and no compiler. Just a straight run through as if someone were typing commands into FORTH in interactive mode at a keyboard. Maybe no need for spaces between commands. For example, circled R might mean use Right-to-left text display. That starts to sound no longer "ad-hoc", but that is not a well-defined term anyway. You're essentially describing a special-purpose markup language or protocol, or perhaps even programming language. Which is quite reasonable; you should (find some other interested people and) work out some of the details and start writing up parsers and such I am thinking that there could be three stacks, one for code points and one for numbers and one for external reference strings such as for accessing a web page or a PDF (Portable Document Format) document or listing an International Standard Book Number and so on. Code points could be entered by circled H followed by circled hexadecimal characters followed by a circled character to indicate Push onto the code point stack. Numbers could be entered in base 10, followed by a circled character to mean Push onto the number stack. A later circled character could mean to take a certain number of code points (maybe just 1, or maybe 0) from the character stack and a certain number of numbers (maybe just 1, or maybe just 0) from the number stack and use them to set some property. It could all be very lightweight software-wise, just reading the characters of the sequence of circled characters and obeying them one by one just one time only on a single run through, with just a few, such as the circled digits, each having its meaning dependent upon a state variable such as, for a circled digit, whether data entry is currently hexadecimal or base 10. I still don't see why you're fixated on using circled characters. You're already dealing with a markup-language type setup, why not do what other markup schemes do? You reserve three or four characters and use them to designate when other characters are not being used in their normal sense but are being used as markup. In XML, when characters are inside '<>' tags, they are not "plain text" of the document, but they mean other things—perhaps things like "right-to-left" or "reference this web page" and so forth, which are exactly the kinds of things you're talking about here. If you don't want to use plain ascii characters because then you couldn't express plain ascii in your text, you're left with exactly the same problem with circled characters: you can't express circled characters in your text. While that is a smaller problem, it can be eliminated altogether by various schemes used by XML or RTF or lightweight markup languages. Reserve a few special characters to give meanings to the others, and arrange for ways to escape your handful of reserved characters so you can express them. More straightforward to say "you have to escape <, >, and & characters" than to say "you have to escape all circled characters." Anyway, this is clearly a whole new high-level protocol you need (or want) to work out, which would *use* Unicode (just like XML and JSON do), but doesn't really affect or involve it (Unicode is all about the "plain text". Kind of getting off-topic, but get some people interested and start a mailing list to discuss it. Good luck! ~mark
Re: UCD in XML or in CSV?
On 8/31/2018 1:36 AM, Manuel Strehl via Unicode wrote: For codepoints.net I use that data to stuff everything in a MySQL database. Well, for some sense of "everything", anyway. ;-) People having this discussion should keep in mind a few significant points. First, the UCD proper isn't "everything", extensive as it is. There are also other significant sets of data that the UTC maintains about characters in other formats, as well, including the data files associated with UTS #46 (IDNA-related), UTS #39 (confusables mapping, etc.), UTS #10 (collation), UTR #25 (a set of math-related property values), and UTS #51 (emoji-related). The emoji-related data has now strayed into the CLDR space, so a significant amount of the information about emoji characters is now carried as CLDR tags. And then there is various other information about individual characters (or small sets of characters) scattered in the core spec -- some in tables, some not, as well as mappings to dozens of external standards. There is no actual definition anywhere of what "everything" actually is. Further, it is a mistake to assume that every character property just associates a simple attribute with a code point. There are multiple types of mappings, complex relational and set properties, and so forth. The UTC attempts to keep a fairly clear line around what constitutes the "UCD proper" (including Unihan.zip), in part so that it is actually possible to run the tools that create the XML version of the UCD, for folks who want to consume a more consistent, single-file format version of the data. But be aware that that isn't everything -- nor would there be much sense in trying to keep expanding the UCD proper to actually represent "everything" in one giant DTD. Second, one of the main obligations of a standards organization is *stability*. People may well object to the ad hoc nature of the UCD data files that have been added over the years -- but it is a *stable* ad-hockery. The worst thing the UTC could do, IMO, would be to keep tweaking formats of data files to meet complaints about one particular parsing inconvenience or another. That would create multiple points of discontinuity between versions -- worse than just having to deal with the ongoing growth in the number of assigned characters and the occasional addition of new data files and properties to the UCD. Keep in mind that there is more to processing the UCD than just "latest". People who just focus on grabbing the very latest version of the UCD and updating whatever application they have are missing half the problem. There are multiple tools out there that parse and use multiple *versions* of the UCD. That includes the tooling that is used to maintain the UCD (which parses *all* versions), and the tooling that creates UCD in XML, which also parses all versions. Then there is tooling like unibook, to produce code charts, which also has to adapt to multiple versions, and bidi reference code, which also reads multiple versions of UCD data files. Those are just examples I know off the top of my head. I am sure there are many other instances out there that fit this profile. And none of the applications already built to handle multiple versions would welcome having to permanently build in tracking particular format anomalies between specific versions of the UCD. Third, please remember that folks who come here complaining about the complications of parsing the UCD are a very small percentage of a very small percentage of a very small percentage of interested parties. Nearly everybody who needs UCD data should be consuming it as a secondary source (e.g. for reference via codepoints.net), or as a tertiary source (behind specialized API's, regex, etc.), or as an end user (just getting behavior they expect for characters in applications). Programmers who actually *need* to consume the raw UCD data files and write parsers for them directly should actually be able to deal with the format complexity -- and, if anything, slowing them down to make them think about the reasons for the format complexity might be a good thing, as it tends to put the lie to the easy initial assumption that the UCD is nothing more than a bunch of simple attributes for all the code points. --Ken
Re: CLDR (was: Private Use areas)
On 31/08/18 07:27 Janusz S. Bień via Unicode wrote: […] > > Given NamesList.txt / Code Charts comments are kept minimal by design, > > one couldn’t simply pop them into XML or whatever, as the result would be > > disappointing and call for completion in the aftermath. Yet another task > > competing with CLDR survey. > > Please elaborate. It's not clear for me what do you mean. These comments are designed for the Code Charts and as such must not be disproportionate in exhaustivity. Eg we have lists of related languages ending in an ellipsis. Once this is popped into XML, ie extracted from NamesList.txt to be fed in an extensible and unconstrained format (without any constraint as of available space, number and length of comments, and so on), any lack is felt as a discriminating neglect, and there will be a huge rush adding data. Yet Unicode hasn’t set up products where that data could be published, ie not in the Code Charts (for the abovementioned reason), not in ICU so far as the additional information involved does not match a known demand on user side (localizing software does not mean providing scholarly exhaustive information about supported characters). The use will be in character pickers providing every available information about a given character. That is why Unicode is to prioritize CLDR for CLDR users, rather than extra information for the web. > > > Reviewing CLDR data is IMO top priority. > > There are many flaws to be fixed in many languages including in English. > > A lot of useful digest charts are extracted from XML there, > > Which XML? where? More precisely it is LDML, the CLDR-specific XML. What I called “digest charts” are the charts found here: http://www.unicode.org/cldr/charts/34/ The access is via this page: http://cldr.unicode.org/index/downloads where the charts are in the Charts column, while the raw data is under SVN Tag. > > > and we really > > need to go through the data and correct the many many errors, please. > > Some time ago I tried to have a close look at the Polish locale and > found the CLDR site prohibitively confusing. I experienced some trouble too, mainly because "SVN Tag" is counter-intuitive for the access to the XML data (except when knowing about SubVersioN). Polish data is found here: https://www.unicode.org/cldr/charts/34/summary/pl.html The access is via the top of the "Summary" index page (showing root data): https://www.unicode.org/cldr/charts/34/summary/root.html You may wish to particularly check the By-Type charts: https://www.unicode.org/cldr/charts/34/by_type/index.html Here I’d suggest to first focus on alphabetic information and on punctuation. https://www.unicode.org/cldr/charts/34/by_type/core_data.alphabetic_information.punctuation.html Under Latin (table caption, without anchor) we find out what punctuation Polish has compared to other locales using the same script. The exact character appears when hovering the header row. Eg U+2011 NON-BREAKING HYPHEN is systematically missing, which is an error in almost every locale using hyphen. TC is about to correct that. Further you will see that while Polish is using apostrophe https://slowodnia.tumblr.com/post/136492530255/the-use-of-apostrophe-in-polish CLDR does not have the correct apostrophe for Polish, as opposed eg to French. You may wish to note that from now on, both U+0027 APOSTROPHE and U+0022 QUOTATION MARK are ruled out in almost all locales, given the preferred characters in publishing are U+2019 and, for Polish, the U+201E and U+201D that are already found in CLDR pl. Note however that according to the information provided by English Wikipedia: https://en.wikipedia.org/wiki/Quotation_mark#Polish Polish also uses single quotes, that by contrast are still missing in CLDR. Now you might understand what I meant when pointing that there are still many errors in many languages in CLDR, including in English. Best regards, Marcel > > Best regards > > Janusz > > -- > , > Janusz S. Bien > emeryt (emeritus) > https://sites.google.com/view/jsbien > >
Re: UCD in XML or in CSV? (was: Re: Unicode Digest, Vol 56, Issue 20)
To handle the UCD XML file a streaming parser like Expat is necessary. For codepoints.net I use that data to stuff everything in a MySQL database. If anyone is interested, the code for that is Open Source: https://github.com/Codepoints/unicode2mysql/ The example for handling the large XML file can be found here: https://github.com/Codepoints/unicode2mysql/blob/master/bin/ucd_to_sql.py For me it's currently much easier to have all the data in a single place, e.g. a large XML file, than spread over a multitude of files _with different ad-hoc syntaxes_. The situation would possibly be different, though, if the UCD data would be split in several files of the same format. (Be it JSON, CSV, YAML, XML, TOML, whatever. Just be consistent.) Nota bene: That is also true for the emoji data, which consists as of now of five plain text files with similar but not identical formats. Cheers, Manuel Am Fr., 31. Aug. 2018 um 08:19 Uhr schrieb Marius Spix via Unicode : > > A good compromise between human readability, machine processability and > filesize would be using YAML. > > Unlike JSON, YAML supports comments, anchors and references, multiple > documents in a file and several other features. > > Regards, > > Marius Spix > > > On Fri, 31 Aug 2018 06:58:37 +0200 (CEST) Marcel Schneider via Unicode > wrote: > > > On 30/08/18 23:34 Philippe Verdy via Unicode wrote: > > > > > > Welel an alternative to XML is JSON which is more compact and > > > faster/simpler to process; > > > > Thanks for pointing the problem and the solution alike. Indeed the > > main drawback of the XML format of UCD is that it results in an > > “insane” filesize. “Insane” was applied to the number of semicolons > > in UnicodeData.txt, but that is irrelevant. What is really insane is > > the filesize of the XML versions of the UCD. Even without Unihan, it > > may take up to a minute or so to load in a text editor. > > > > > however JSON has no explicit schema, unless the schema is being > > > made part of the data itself, complicating its structure (with many > > > levels of arrays of arrays, in which case it becomes less easy to > > > read by humans, but more adapted to automated processes for fast > > > processing). > > > > > > I'd say that the XML alone is enough to generate any JSON-derived > > > dataset that will conform to the schema an application expects to > > > process fast (and with just the data it can process, excluding > > > various extensions still not implemetned). But the fastest > > > implementations are also based on data tables encoded in code (such > > > as DLL or Java classes), or custom database formats (such as > > > Berkeley dB) generated also automatically from the XML, without the > > > processing cost of decompression schemes and parsers. > > > > > > Still today, even if XML is not the usual format used by > > > applications, it is still the most interoperable format that allows > > > building all sorts of applications in all sorts of languages: the > > > cost of parsing is left to an application builder/compiler. > > > > I’ve tried an online tool to get ucd.nounihan.flat.xml converted to > > CSV. The tool is great and offers a lot of options, but given the > > “insane” file size, my browser was up for over two hours of trouble > > until I shut down the computer manually. From what I could see in the > > result field, there are many bogus values, meaning that their > > presence is useless in the tags of most characters. And while many > > attributes have cryptic names in order to keep the file size minimal, > > some attributes have overlong values, ie the design is inconsistent. > > Eg in every character we read: jg="No_Joining_Group" That is bogus. > > One would need to take them off the tags of most characters, and even > > in the characters where they are relevant, the value would be simply > > "No". What’s the use of abbreviating "Joining Group" to "jg" in the > > atribute name if in the value it is written out? And I’m quoting from > > U+. Further many values are set to a crosshatch, instead of > > simply being removed from the characters where they are empty. Then > > the many instances of "undetermined script" resulting in *two* > > attribues with "Zyyy" value. Then in almost each character we’re told > > that it is not a whitespace, not a dash, not a hyphen, and not a > > quotation mark: Dash="N" WSpace="N" Hyphen="N" QMark="N" One couldn’t > > tell that UCD does actually benefit from the flexibility of XML, > > given that many attributes are systematically present even where they > > are useless. Perhaps ucd-*.xml would be two thirds, half, or one > > third their actual size if they were properly designed. > > > > > Some apps embed the compilers themselves and use a stored cache for > > > faster processing: this approach allows easy updates by detecting > > > changes in the XML source, and then downloading them. > > > > > > But in CLDR such updates are generally not automated : the general > > > scheme evol
Re: CLDR (was: Private Use areas)
The XML files in these folders: https://unicode.org/repos/cldr/tags/latest/common/ But I agree. I spent an extreme amount of time to get somewhat used to cldr.unicode.org and and the data repo, and still I have no clue, where to find a concrete piece of information without digging into the site. Am Fr., 31. Aug. 2018 um 07:22 Uhr schrieb Janusz S. Bień via Unicode : > > On Thu, Aug 30 2018 at 2:27 +0200, unicode@unicode.org writes: > > [...] > > > Given NamesList.txt / Code Charts comments are kept minimal by design, > > one couldn’t simply pop them into XML or whatever, as the result would be > > disappointing and call for completion in the aftermath. Yet another task > > competing with CLDR survey. > > Please elaborate. It's not clear for me what do you mean. > > > Reviewing CLDR data is IMO top priority. > > There are many flaws to be fixed in many languages including in English. > > A lot of useful digest charts are extracted from XML there, > > Which XML? where? > > > and we really > > need to go through the data and correct the many many errors, please. > > Some time ago I tried to have a close look at the Polish locale and > found the CLDR site prohibitively confusing. > > Best regards > > Janusz > > -- > , > Janusz S. Bien > emeryt (emeritus) > https://sites.google.com/view/jsbien >