Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Wed, 13 Jun 2012 17:30:41 -0700, Charles Mills wrote:

>FWIW, z/OS Unicode Services does indicate that at least one SUB character
>was output. It's not an error (RC still = 0) but it is a documented output
>status bit flag.
> 
What avail is this if SUB is a valid character in the input code page?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

FWIW, z/OS Unicode Services does indicate that at least one SUB character
was output. It's not an error (RC still = 0) but it is a documented output
status bit flag.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Paul Gilmartin
Sent: Wednesday, June 13, 2012 5:20 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On Wed, 13 Jun 2012 13:50:17 -0500, McKown, John wrote:

>I __think__ I understand what IBM is saying. You must differentiate between
a "hex value" (0x00..0xFF) and a "code point" (a subset of "hex values").
Not every "hex value" is a "code point" in every CCSID. I.e. a single byte
CCSID may have less than 256 "code points", or a double byte CCSID may have
less than 65336 "code points"...
> 
In that case, a conscientious utility _ought_ to indicate an error on
encountering an octet which is not a "code point" rather than mapping it to
some arbitrary value.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Wed, 13 Jun 2012 08:31:12 -0400, Shmuel Metz (Seymour J.) wrote:

>In
>... a Unicode code page ...
> 
Oxymoron?  Unicode is unicode.  There's only one code point
for any given character.  Unicode was designed to avoid the
babel of code pages.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Wed, 13 Jun 2012 13:50:17 -0500, McKown, John wrote:

>I __think__ I understand what IBM is saying. You must differentiate between a 
>"hex value" (0x00..0xFF) and a "code point" (a subset of "hex values"). Not 
>every "hex value" is a "code point" in every CCSID. I.e. a single byte CCSID 
>may have less than 256 "code points", or a double byte CCSID may have less 
>than 65336 "code points"...
> 
In that case, a conscientious utility _ought_ to indicate an error on 
encountering
an octet which is not a "code point" rather than mapping it to some arbitrary
value.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Tony Harminc]

On 13 June 2012 15:02, Mike Schwab  wrote:

> Roundtrip example:  Every defined character in 1027, excluding values
> that do not have a character defined, exist in 1208, is successfully
> translated from 1027 to 1208 and back to 1027.  All codepoints that do
> not have a character defined will be translated to (one?) non-valid
> value.
>
> Non-roundtrip example:  Some defined characters in 1208, and all
> codepoints that do not have a character defined, do not exist in 1027.
>  If you translate text from 1208 to 1027, the characters not defined
> in 1027, and all undefined codepoints will be translated to (one?)
> non-valid codepoint.
>
> In any translation, a codepoint that does not encode a character will
> be translated to (one?) non-valid codepoint.

Not in *any* translation. IBM does define this kind of translation,
but it isn't "round trippable" - it's "enforced subset match".

>From the CDRA book: "The enforced subset match criterion guarantees
the preservation of the subset of characters that are common to both
the input and output character sets. Any character not in this common
subset will be replaced with a unique character that indicates that a
substitution has occurred." They go on to recommend use of X'1A' in
ASCII-ish environments, and X'3F' in EBCDIC.

> You missed the word defined.  If a codepoint does not have a defined
> character, it is translated to (one?) invalid value.

I can't make it parse that way.

"CDRA defines only the graphic character data conversion part of the
overall data conversion process. A limited number of control
characters are addressed as part of handling different string types
(see "Types of Strings") and as part of control character mappings
(see "Pairings of Code Points"). Other control characters are treated
as bytes, and are dealt with according to mismatch management
criteria."

and under "Criteria for Character Set Mismatch Management", describing
"Pairing of Code Points Using Round Trip":

 - An input graphic code point outside the common set is mapped to an
output graphic code point outside the common set
 - An input control code point is mapped to an output control code
point outside the mnemonic-based common set
 - If the graphic encoding space of the source is larger than the
graphic encoding space of the target, some graphic code points will be
mapped to control code points, and vice versa.

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

I did not "miss" the word defined. It's not in IBM's definition.

I hear everyone who is saying "the term 'code point' *really* means a bit
value with a glyph assigned to it," but that's not what the definitions out
there say. Wikipedia:

In character encoding terminology, a code point or code position is any of
the numerical values that make up the code space (or code page).[1] For
example, ASCII comprises 128 code points in the range 0hex to 7Fhex,
Extended ASCII comprises 256 code points in the range 0hex to FFhex, and
Unicode comprises 1,114,112 code points in the range 0hex to 10hex. The
Unicode code space is divided into seventeen planes (the basic multilingual
plane, and 16 supplementary planes), each with 65,536 (= 216) code points.
Thus the total size of the Unicode code space is 17 × 65,536 = 1,114,112.

"ASCII comprises 128 code points." Not all 128 of those have glyphs.

"Unicode comprises 1,114,112 code points." Not all of those million-plus
code points have glyphs.

"every code point in the source CCSID maps to a unique code point in the
target CCSID" (note no "defined")

That says every one of the 128 ASCII code points maps to a unique bit
combination. But it ain't so.

If one is going to define "round trip conversion" as applying only to
corresponding glyphs then the definition loses any meaning. Any rational
translation is round trip with regard to corresponding glyphs.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Mike Schwab
Sent: Wednesday, June 13, 2012 12:02 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On Wed, Jun 13, 2012 at 12:59 PM, Charles Mills  wrote:
> I got a response to the PMR. Taking the liberty of paraphrasing a long 
> reply, the essence of it seemed to be that -- per the CCSID pair lists 
> in the manual -- they support round trip conversion from 1027 to 1208 
> but not from 1208 to 1027. Here is what I wrote back:
>
What this means is:
Roundtrip example:  Every defined character in 1027, excluding values that
do not have a character defined, exist in 1208, is successfully translated
from 1027 to 1208 and back to 1027.  All codepoints that do not have a
character defined will be translated to (one?) non-valid value.

Non-roundtrip example:  Some defined characters in 1208, and all codepoints
that do not have a character defined, do not exist in 1027.
 If you translate text from 1208 to 1027, the characters not defined in
1027, and all undefined codepoints will be translated to (one?) non-valid
codepoint.

In any translation, a codepoint that does not encode a character will be
translated to (one?) non-valid codepoint.

If, while the text is in 1208 is changed to add a character not in 1027,
upon translation that value will be changed to an invalid value in 1027.

> It sounds like you are saying (for the CCSIDs in question) "we support 
> round trip, but in one direction only." It would be like if I bought a 
> round trip ticket on Delta between San Francisco and Atlanta, and 
> after I got to Atlanta, they explained that it was a round trip ticket 
> only in one direction.
>
> I would kind of question also whether what you are doing conforms to 
> your definition of round trip in the Unicode manual glossary: Round 
> trip. Encoding that occurs when every code point in the source CCSID 
> maps to a unique code point in the target CCSID.

You missed the word defined.  If a codepoint does not have a defined
character, it is translated to (one?) invalid value.

> Using round trip
> tables ensure the capability of reversing the conversion, and 
> recovering the complete original source datastream.
>
> I would question "every code point in the source CCSID maps to a 
> unique code point in the target CCSID" when both 3F and 41 map to the 
> same code point, and I wonder how I would recover the original source 
> datastream.
>
Again, you missed the word defined.  If a codepoint does not have a defined
character, it is translated to (one?) invalid value.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Shmuel Metz (Seymour J.)]

In
<93891f43642f3c419a7d75acc2b1db6f3c68cc9...@exchangemb2.dhs.state.ia.us>,
on 06/12/2012
   at 03:38 PM, "Roberts, John J"  said:

>Because "bijective" is not so well understood by anyone born before
>1952 or so. 

I was born before 1952 and understand "bijective" just fine. OTOH,
"round trip" is something I specify when I am buying travel tickets.

OTOH, a mapping from an EBCDIC code page to a Unicode code page can't
be bijective, only injective, and there is no round trip for, e.g.,
Unicode->EBCDIC->Unicode, regardless of which EBCDIC code page and
Unicode transform you use. Contrast "left inverse" with "right
inverse".
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Mike Schwab]

On Wed, Jun 13, 2012 at 12:59 PM, Charles Mills  wrote:
> I got a response to the PMR. Taking the liberty of paraphrasing a long
> reply, the essence of it seemed to be that -- per the CCSID pair lists in
> the manual -- they support round trip conversion from 1027 to 1208 but not
> from 1208 to 1027. Here is what I wrote back:
>
What this means is:
Roundtrip example:  Every defined character in 1027, excluding values
that do not have a character defined, exist in 1208, is successfully
translated from 1027 to 1208 and back to 1027.  All codepoints that do
not have a character defined will be translated to (one?) non-valid
value.

Non-roundtrip example:  Some defined characters in 1208, and all
codepoints that do not have a character defined, do not exist in 1027.
 If you translate text from 1208 to 1027, the characters not defined
in 1027, and all undefined codepoints will be translated to (one?)
non-valid codepoint.

In any translation, a codepoint that does not encode a character will
be translated to (one?) non-valid codepoint.

If, while the text is in 1208 is changed to add a character not in
1027, upon translation that value will be changed to an invalid value
in 1027.

> It sounds like you are saying (for the CCSIDs in question) "we support
> round trip, but in one direction only." It would be like if I bought a
> round trip ticket on Delta between San Francisco and Atlanta, and after
> I got to Atlanta, they explained that it was a round trip ticket only
> in one direction.
>
> I would kind of question also whether what you are doing conforms to
> your definition of round trip in the Unicode manual glossary: Round
> trip. Encoding that occurs when every code point in the source CCSID
> maps to a unique code point in the target CCSID.

You missed the word defined.  If a codepoint does not have a defined
character, it is translated to (one?) invalid value.

> Using round trip
> tables ensure the capability of reversing the conversion, and
> recovering the complete original source datastream.
>
> I would question "every code point in the source CCSID maps to a unique
> code point in the target CCSID" when both 3F and 41 map to the same
> code point, and I wonder how I would recover the original source
> datastream.
>
Again, you missed the word defined.  If a codepoint does not have a
defined character, it is translated to (one?) invalid value.

> Charles
>

-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[McKown, John]

I __think__ I understand what IBM is saying. You must differentiate between a 
"hex value" (0x00..0xFF) and a "code point" (a subset of "hex values"). Not 
every "hex value" is a "code point" in every CCSID. I.e. a single byte CCSID 
may have less than 256 "code points", or a double byte CCSID may have less than 
65336 "code points". A "round trip" conversion from one CCSID to a different 
CCSID says that every possible "code point" in CCSID#1 will map to a unique 
"code point" in CCSID#1. I.e. the "trip" is on a two way street and you can 
"turn around" to go back to where you started. However, there may be a "one 
way" street from CCSID#2 to CCSID#1. So when you start in CCSID#2, you cannot 
just "turn around" when you get do CCSID#1 and return to your starting 
location. 

Think of the "code points" in a CCSID as streets in a city. If the street is 
"two way" (round trip), then you can go back the way you came. If the street is 
"one way", then you can't. So CCSID#2 may have a "one way" street to CCSID#1. 
You can make the trip, but if you go back, you must use a different street. So 
when you go back to CCSID#2, you end up somewhere else in CCSID#2 from where 
you started. 

A "round trip" route from CCSID#1 to CCSID#2 means every street from CCSID#1 to 
CCSID#2 is "two way". But that does not mean that every street from CCSID#2 to 
CCSID#1 must be a "two way" street. There may be a "one way" trip from #2 to 
#1. So you cannot say that #2 is "round trip" to #1. That is, every "code 
point" in CCSID#2 must go to a "code point" in CCSID#1, but if more than one 
does, then the conversion is not classified "round trip".

And it means that, so far as these conversions are concerned, if you use a hex 
value not in the set of "code points", nothing at all is guarnateed (you went 
"off road" so to speak). So "round trip" does not say that every "hex value" 
maps to a unique destination. In fact a "hex value" which is not a "code point" 
in a source CCSID could actually translate to a "hex value" which is a "code 
point" in the destination CCSID.

I now have a headache and am going to have a lay down.

-- 
John McKown 
Systems Engineer IV
IT

Administrative Services Group

HealthMarkets(r)

9151 Boulevard 26 * N. Richland Hills * TX 76010
(817) 255-3225 phone * 
john.mck...@healthmarkets.com * www.HealthMarkets.com

Confidentiality Notice: This e-mail message may contain confidential or 
proprietary information. If you are not the intended recipient, please contact 
the sender by reply e-mail and destroy all copies of the original message. 
HealthMarkets(r) is the brand name for products underwritten and issued by the 
insurance subsidiaries of HealthMarkets, Inc. -The Chesapeake Life Insurance 
Company(r), Mid-West National Life Insurance Company of TennesseeSM and The 
MEGA Life and Health Insurance Company.SM


> -Original Message-
> From: IBM Mainframe Discussion List 
> [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Charles Mills
> Sent: Wednesday, June 13, 2012 1:00 PM
> To: IBM-MAIN@bama.ua.edu
> Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion
> 
> I got a response to the PMR. Taking the liberty of paraphrasing a long
> reply, the essence of it seemed to be that -- per the CCSID 
> pair lists in
> the manual -- they support round trip conversion from 1027 to 
> 1208 but not
> from 1208 to 1027. Here is what I wrote back:
> 
> It sounds like you are saying (for the CCSIDs in question) 
> "we support  
> round trip, but in one direction only." It would be like if I 
> bought a  
> round trip ticket on Delta between San Francisco and Atlanta, 
> and after 
> I got to Atlanta, they explained that it was a round trip 
> ticket only   
> in one direction. 
>   
>   
>   
> I would kind of question also whether what you are doing 
> conforms to
> your definition of round trip in the Unicode manual glossary: 
> Round 
> trip. Encoding that occurs when every code point in the 
> source CCSID
> maps to a unique code point in the target CCSID. Using round 
> trip   
> tables ensure the capability of reversing the conversion, and 
>   
> recovering the complete original source datastream.   
>   
>   
>   
> I would question "every code point in the source CCSID maps 
> to a unique

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

I got a response to the PMR. Taking the liberty of paraphrasing a long
reply, the essence of it seemed to be that -- per the CCSID pair lists in
the manual -- they support round trip conversion from 1027 to 1208 but not
from 1208 to 1027. Here is what I wrote back:

It sounds like you are saying (for the CCSIDs in question) "we support  
round trip, but in one direction only." It would be like if I bought a  
round trip ticket on Delta between San Francisco and Atlanta, and after 
I got to Atlanta, they explained that it was a round trip ticket only   
in one direction.   

I would kind of question also whether what you are doing conforms to
your definition of round trip in the Unicode manual glossary: Round 
trip. Encoding that occurs when every code point in the source CCSID
maps to a unique code point in the target CCSID. Using round trip   
tables ensure the capability of reversing the conversion, and   
recovering the complete original source datastream. 

I would question "every code point in the source CCSID maps to a unique 
code point in the target CCSID" when both 3F and 41 map to the same 
code point, and I wonder how I would recover the original source
datastream.

Charles

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Bill Godfrey]

The same glossary says this about code point: "A unique bit pattern defined in 
a code. Depending on the code, a code point can be 7-bit, 8-bit, 16-bit, or 
other. Code points are assigned to a graphic character in a code page."

Note the last sentence of that.

Also, you earlier quoted a section of a DB2 website about "round trip 
conversion", including this:

"Example: In ASCII CCSID 1252, the trademark symbol T is code point X'99'. In 
EBCDIC CCSID 37, this code point does not exist."

So a code point is not just any possible combination of bits, it's a 
combination of bits that is assigned to a character. It is possible to say that 
a code point "does not exist" in a CCSID. Therefore the description of "round 
trip" that you quoted:

"Round trip. Encoding that occurs when *every* code point in the source CCSID 
maps to a unique code point in the target CCSID"

can be read to apply only to certain bit combinations, those to which a 
character is assigned, and not to all bit combinations.

The glossary in the DB2 website that you referenced earlier has this for "code 
point":

"In CDRA, a unique bit pattern that represents a character in a code page."

Bill

On Tue, 12 Jun 2012 21:07:51 -0700, Charles Mills wrote:

>1027 was a test case and the data is arbitrary. I think the point of round
>trip is supposed to be that *every* character is "recoverable."
>
>A good write-up in of all places  the Glossary of the Z Unicode manual.
>Emphasis is mine.
>
>Round trip. Encoding that occurs when *every* code
>point in the source CCSID maps to a unique code point
>in the target CCSID. Using round trip tables ensure the
>capability of reversing the conversion, and recovering
>the complete original source datastream.
>
>I'm going to see what they say on the PMR.
>
>Charles
>
>-Original Message-
>From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
>Of Bill Godfrey
>Sent: Tuesday, June 12, 2012 7:12 PM
>To: IBM-MAIN@bama.ua.edu
>Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion
>
>Maybe it will turn out that "round trip" only applies to the characters that
>are defined in the Source CCSID (and I'm not sure if "defined" is the right
>word). When I look at a chart of CCSID 1027 here:
>
>ftp://ftp.software.ibm.com/software/globalization/gcoc/attachments/CP01027.p
>df
>
>I see that all of the characters that you see being being converted to X'1A'
>are either outside the chart (which starts at X'40') or have no identified
>value in the chart. If your data contains these unidentified characters,
>what is their purpose, and is it really appropriate to consider the data to
>be associated with CCSID 1027, or just something related to but not the same
>as CCSID 1027?
>
>Bill
>
>On Tue, 12 Jun 2012 16:42:02 -0700, Charles Mills wrote:
>
>>Not sure what you mean. Here's the PMR:
>>
>>Problem Details
>>.
>>Product or Service: Support for Unicode Component ID: 5752SCUNI .
>>Operating System: z/OS
>>.
>>Problem title
>>Round trip conversion not working as expected .
>>Problem description
>>My understanding of "round trip" conversion is that every code point in
>>the Source CCSID should convert to a unique code point in the Target
>>CCSID (so that conversion in the reverse direction will restore the
>>original code points). However, when I use CUNLCNV to convert between
>>CCSIDs 1027 and 1208 I see more than one character converted to x'1A'
>>(UTF8 SUB). Is round trip conversion working correctly? (Both the
>>documentation and CUNLINFO report that 'R' is a supported conversion
>>technique for these two CCSIDs.)
>>
>>Characters converted to X'1A' include 3F, 41, 6A, 80, 90, etc.
>>
>>Thank you for your consideration.
>>.
>>Business impact ( BusImpact )
>>Negatively impacting product design decisions.
>>
>>Charles
>>-Original Message-
>>From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On
>>Behalf Of Tony Harminc
>>Sent: Tuesday, June 12, 2012 4:15 PM
>>To: IBM-MAIN@bama.ua.edu
>>Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion
>>
>>On 12 June 2012 18:55, Charles Mills  wrote:
>>> I have opened a PMR.
>>
>>For the doc, or the behaviour of the service? Or did you choose the
>>"let us decide for you" option...?
>>
>>Tony H.
>>
>
>--
>For IBM-MAIN subscribe / signoff / archive access instructions, send email
>to lists...@bama.ua.edu with the message: INFO IBM-MAIN
>
>--
>For IBM-MAIN subscribe / signoff / archive access instructions,
>send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Shmuel Metz (Seymour J.)]

In <00f101cd48c5$195bdba0$4c1392e0$@mcn.org>, on 06/12/2012
   at 10:59 AM, Charles Mills  said:

>Fair enough. Unicode services reports however that it supports
>roundtrip conversion in many of these cases, including for example,
>37 to 850 (pretty basic ASCII).

850 is not ASCII; it includes code points from 128 to 244.
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Peter Sylvester]

On 06/12/2012 10:56 PM, John Gilmore wrote:

The term 'bijective' is a fairly old one, the earliest reference I
found in a Mathematical Reviews index was for 1939..  Anyone who has
had a college course in  mathematical logic or, yes, set theory is
likely to know or have forgotten its meaning.

It does, however, have a bad reputation because of its use to
intimidate non-mathematicians.  The example in Quine's Quiddities
under the topic mathematosis illustrates why.

Still, it is perhaps better than 'roundtrip', which in ordinary use
does not really imply bijective.  My wife and I both found that we
were measurably heavier after a recent Boston-Roma-Boston trip.
Additional efforts were required to restore our status quo ante
weights.


you have two mappings between unicode and ebcdic,
you need an injection from ebcdic to unicode,
and a surjection from the image of that injection
which must be the inverse of the injection.
thus both mappings are bijections between their
field and image and inverse to each other.

in fact no one needs mappings, codes, computers at all,
you just take the universal library, i.e. the bit.

field and image of you two mappings of your roundtrip
are different, still, at least almost, applying a regime
may create a projection that preserve your integrities.
you don't become your wife for example or have your
head cut off.

a mapping that exchanges your with your with is
perfectly bijective and even using itself as inverse is so,
but still, you may have a problem when + is interpreted
as - and vice versa, so you also want preservation of
the somewhat implied semantics of a character/person.

have fun

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

1027 was a test case and the data is arbitrary. I think the point of round
trip is supposed to be that *every* character is "recoverable."

A good write-up in of all places  the Glossary of the Z Unicode manual.
Emphasis is mine.

Round trip. Encoding that occurs when *every* code
point in the source CCSID maps to a unique code point
in the target CCSID. Using round trip tables ensure the
capability of reversing the conversion, and recovering
the complete original source datastream.

I'm going to see what they say on the PMR.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Bill Godfrey
Sent: Tuesday, June 12, 2012 7:12 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Maybe it will turn out that "round trip" only applies to the characters that
are defined in the Source CCSID (and I'm not sure if "defined" is the right
word). When I look at a chart of CCSID 1027 here:

ftp://ftp.software.ibm.com/software/globalization/gcoc/attachments/CP01027.p
df

I see that all of the characters that you see being being converted to X'1A'
are either outside the chart (which starts at X'40') or have no identified
value in the chart. If your data contains these unidentified characters,
what is their purpose, and is it really appropriate to consider the data to
be associated with CCSID 1027, or just something related to but not the same
as CCSID 1027?

Bill 

On Tue, 12 Jun 2012 16:42:02 -0700, Charles Mills wrote:

>Not sure what you mean. Here's the PMR:
>
>Problem Details
>.
>Product or Service: Support for Unicode Component ID: 5752SCUNI .
>Operating System: z/OS
>.
>Problem title
>Round trip conversion not working as expected .
>Problem description
>My understanding of "round trip" conversion is that every code point in 
>the Source CCSID should convert to a unique code point in the Target 
>CCSID (so that conversion in the reverse direction will restore the 
>original code points). However, when I use CUNLCNV to convert between 
>CCSIDs 1027 and 1208 I see more than one character converted to x'1A'
>(UTF8 SUB). Is round trip conversion working correctly? (Both the 
>documentation and CUNLINFO report that 'R' is a supported conversion 
>technique for these two CCSIDs.)
>
>Characters converted to X'1A' include 3F, 41, 6A, 80, 90, etc.
>
>Thank you for your consideration.
>.
>Business impact ( BusImpact )
>Negatively impacting product design decisions.
>
>Charles
>-Original Message-
>From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On 
>Behalf Of Tony Harminc
>Sent: Tuesday, June 12, 2012 4:15 PM
>To: IBM-MAIN@bama.ua.edu
>Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion
>
>On 12 June 2012 18:55, Charles Mills  wrote:
>> I have opened a PMR.
>
>For the doc, or the behaviour of the service? Or did you choose the 
>"let us decide for you" option...?
>
>Tony H.
>

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@bama.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Bill Godfrey]

Maybe it will turn out that "round trip" only applies to the characters that 
are defined in the Source CCSID (and I'm not sure if "defined" is the right 
word). When I look at a chart of CCSID 1027 here:

ftp://ftp.software.ibm.com/software/globalization/gcoc/attachments/CP01027.pdf

I see that all of the characters that you see being being converted to X'1A' 
are either outside the chart (which starts at X'40') or have no identified 
value in the chart. If your data contains these unidentified characters, what 
is their purpose, and is it really appropriate to consider the data to be 
associated with CCSID 1027, or just something related to but not the same as 
CCSID 1027?

Bill 

On Tue, 12 Jun 2012 16:42:02 -0700, Charles Mills wrote:

>Not sure what you mean. Here's the PMR:
>
>Problem Details
>.
>Product or Service: Support for Unicode
>Component ID: 5752SCUNI
>.
>Operating System: z/OS
>.
>Problem title
>Round trip conversion not working as expected
>.
>Problem description
>My understanding of "round trip" conversion is that every code point in
>the Source CCSID should convert to a unique code point in the Target
>CCSID (so that conversion in the reverse direction will restore the
>original code points). However, when I use CUNLCNV to convert between
>CCSIDs 1027 and 1208 I see more than one character converted to x'1A'
>(UTF8 SUB). Is round trip conversion working correctly? (Both the
>documentation and CUNLINFO report that 'R' is a supported conversion
>technique for these two CCSIDs.)
>
>Characters converted to X'1A' include 3F, 41, 6A, 80, 90, etc.
>
>Thank you for your consideration.
>.
>Business impact ( BusImpact )
>Negatively impacting product design decisions.
>
>Charles
>-Original Message-----
>From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
>Of Tony Harminc
>Sent: Tuesday, June 12, 2012 4:15 PM
>To: IBM-MAIN@bama.ua.edu
>Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion
>
>On 12 June 2012 18:55, Charles Mills  wrote:
>> I have opened a PMR.
>
>For the doc, or the behaviour of the service? Or did you choose the "let us
>decide for you" option...?
>
>Tony H.
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Not sure what you mean. Here's the PMR:

Problem Details 
.   
Product or Service: Support for Unicode 
Component ID: 5752SCUNI 
.   
Operating System: z/OS  
.   
Problem title   
Round trip conversion not working as expected   
.   
Problem description 
My understanding of "round trip" conversion is that every code point in 
the Source CCSID should convert to a unique code point in the Target
CCSID (so that conversion in the reverse direction will restore the 
original code points). However, when I use CUNLCNV to convert between   
CCSIDs 1027 and 1208 I see more than one character converted to x'1A'   
(UTF8 SUB). Is round trip conversion working correctly? (Both the   
documentation and CUNLINFO report that 'R' is a supported conversion
technique for these two CCSIDs.)

Characters converted to X'1A' include 3F, 41, 6A, 80, 90, etc.  

Thank you for your consideration.   
.   
Business impact ( BusImpact )   
Negatively impacting product design decisions.  

Charles
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Tony Harminc
Sent: Tuesday, June 12, 2012 4:15 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On 12 June 2012 18:55, Charles Mills  wrote:
> I have opened a PMR.

For the doc, or the behaviour of the service? Or did you choose the "let us
decide for you" option...?

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@bama.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Tony Harminc]

On 12 June 2012 18:55, Charles Mills  wrote:
> I have opened a PMR.

For the doc, or the behaviour of the service? Or did you choose the
"let us decide for you" option...?

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

I have opened a PMR.

Charles
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 12:50 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Thanks. I think either your and my understanding of roundtrip is flawed, or
Unicode Services understands it differently, or I am missing something.

Charles

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

They are saying that if you go from Linux to z/OS to Linux then '%' might
become '?' and then back to '%'. (Those are just example characters -- don't
take them literally.)

But if you go from z/OS to Linux to Java then '?' might become '%' and then
'!'. 

They are saying that round-trip does not mean you can just translate it
twice and be back to the original character. You have to go "round trip" --
back to where (the CCSID?) you came from.

It's not the clearest example. 

Charles
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Paul Gilmartin
Sent: Tuesday, June 12, 2012 1:45 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On Tue, 12 Jun 2012 12:59:13 -0700, Charles Mills wrote:
>
>" A round-trip conversion works only in a two-tier homogenous 
>environment where the data makes the complete round trip. For example, 
>if you pass data from DB2 for Linux, UNIX, and Windows to DB2 for z/OS 
>and then back to DB2 for Linux, UNIX, and Windows with a round-trip
conversion, no data is lost.
>The data was converted back to its original format. However, if you 
>have a more complicated environment, a round-trip conversion does not 
>necessarily preserve data integrity. For example, if you pass data from 
>DB2 for z/OS to
>DB2 for Linux, UNIX, and Windows and then to Linux on a Java client, 
>two conversions have potentially occurred. Because the data was not 
>converted back to its original format before the second conversion, 
>data might have been lost even if round-trip conversions are used.
> 
???  Errr...

The composition of two (or more) bijective function is a bijective function.
This can't violate data integrity.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[John Gilmore]

The term 'bijective' is a fairly old one, the earliest reference I
found in a Mathematical Reviews index was for 1939..  Anyone who has
had a college course in  mathematical logic or, yes, set theory is
likely to know or have forgotten its meaning.

It does, however, have a bad reputation because of its use to
intimidate non-mathematicians.  The example in Quine's Quiddities
under the topic mathematosis illustrates why.

Still, it is perhaps better than 'roundtrip', which in ordinary use
does not really imply bijective.  My wife and I both found that we
were measurably heavier after a recent Boston-Roma-Boston trip.
Additional efforts were required to restore our status quo ante
weights.


--jg

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Kirk Wolf]

I would say that bijection (a one-one correspondence) is not exactly the
same as "roundtripping".

For example:

IBM-1047 (single byte EBCDIC)  ->  UTF-8

- you can round-trip this, since you can take any character in the source
code page, and get a UTF-8 character.   If you take any of *those* 256
UTF-8 characters, they will map uniquely back to where they started.

But, you cannot have a round-trip from UTF-8 -> IBM-1047  (since the source
set is MUCH bigger).

But, I still don't understand the results that Charles is seeing; doesn't
look like round-tripping  :-(

Kirk Wolf
Dovetailed Technologies
http://dovetail.com

On Tue, Jun 12, 2012 at 3:38 PM, Roberts, John J
wrote:

> >Why does IBM have this compulsion to be different from everyone else
> and invent its own terminology rather than using a conventional,
> well-understood word such as "bijective"?
>
> Because "bijective" is not so well understood by anyone born before 1952
> or so.  The term relates to SET Theory, which was a big part of the "New
> Math".  My younger brother, born in 1953 was exposed to this, but I was
> not.  Actually, I only learned of the term (and the noun "bijection") on
> this forum, this past year.
>
> But, of course we have digressed far from Charles original question.
>  Sorry.
>
> John
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Tue, 12 Jun 2012 12:59:13 -0700, Charles Mills wrote:
>
>" A round-trip conversion works only in a two-tier homogenous environment
>where the data makes the complete round trip. For example, if you pass data
>from DB2 for Linux, UNIX, and Windows to DB2 for z/OS and then back to DB2
>for Linux, UNIX, and Windows with a round-trip conversion, no data is lost.
>The data was converted back to its original format. However, if you have a
>more complicated environment, a round-trip conversion does not necessarily
>preserve data integrity. For example, if you pass data from DB2 for z/OS to
>DB2 for Linux, UNIX, and Windows and then to Linux on a Java client, two
>conversions have potentially occurred. Because the data was not converted
>back to its original format before the second conversion, data might have
>been lost even if round-trip conversions are used.
> 
???  Errr...

The composition of two (or more) bijective function is a bijective
function.  This can't violate data integrity.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[zMan]

On Tue, Jun 12, 2012 at 4:03 PM, Paul Gilmartin wrote:

> Why does IBM have this compulsion to be different from everyone else
> and invent its own terminology rather than using a conventional,
> well-understood word such as "bijective"?
>

Harumph. It may be an established and well-defined term, and your point is
valid, but "conventional" and "well-understood" are arguable. Sure, the
math PhDs use that term, but reg'lar folks don't.

However, it would have behooved IBM to define and use it. Or maybe the
people who wrote that don't know it...
-- 
zMan -- "I've got a mainframe and I'm not afraid to use it"

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Roberts, John J]

>Why does IBM have this compulsion to be different from everyone else
and invent its own terminology rather than using a conventional,
well-understood word such as "bijective"?

Because "bijective" is not so well understood by anyone born before 1952 or so. 
 The term relates to SET Theory, which was a big part of the "New Math".  My 
younger brother, born in 1953 was exposed to this, but I was not.  Actually, I 
only learned of the term (and the noun "bijection") on this forum, this past 
year.

But, of course we have digressed far from Charles original question.  Sorry.

John

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Tue, 12 Jun 2012 15:02:17 -0400, Tony Harminc wrote:
>
>"The objective of this criterion is to send data from one system to
>another one that has different representations of character data, and
>retrieve it without loss. Often the "do not convert" choice is not
>available. For example, data stored in a System/370 database is
>configured to have all its graphic character data in one CCSID. If it
>acts as a remote repository for data from a PC application, or from an
>application in another System/370 using a different CCSID, the data
>must be converted to the configured CCSID. The data is intended to be
>retrieved by the same application without loss when it is converted
>back for use in its original CCSID."
>
>There's more, but I think the notion of Round Trip Integrity is clear enough.
>
Why does IBM have this compulsion to be different from everyone else
and invent its own terminology rather than using a conventional,
well-understood word such as "bijective"?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

FYI:

" A round-trip conversion works only in a two-tier homogenous environment
where the data makes the complete round trip. For example, if you pass data
from DB2 for Linux, UNIX, and Windows to DB2 for z/OS and then back to DB2
for Linux, UNIX, and Windows with a round-trip conversion, no data is lost.
The data was converted back to its original format. However, if you have a
more complicated environment, a round-trip conversion does not necessarily
preserve data integrity. For example, if you pass data from DB2 for z/OS to
DB2 for Linux, UNIX, and Windows and then to Linux on a Java client, two
conversions have potentially occurred. Because the data was not converted
back to its original format before the second conversion, data might have
been lost even if round-trip conversions are used.

Example: In ASCII CCSID 1252, the trademark symbol T is code point X'99'. In
EBCDIC CCSID 37, this code point does not exist. During character conversion
from ASCII CCSID 1252 to EBCDIC CCSID 37, the trademark symbol is converted
to a control character, X'39'. If you use SPUFI to select the data, the data
displays as the unprintable character that you specified in your emulator,
which is generally a quotation mark (") or a period (.). If you issue a
SELECT HEX(column) statement, the data displays as X'39', which is the DEL
control character. When the client uses a round-trip conversion to convert
this character back to ASCII CCSID 1252, the trademark symbol is preserved
as code point X'99'. Notice that this conversion is not lossless unless it
is converted back to the original format."

--
http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=%2Fco
m.ibm.db2z9.doc.char%2Fsrc%2Ftpc%2Fdb2z_roundtripconversion.htm 

Charles

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Thanks. I think either your and my understanding of roundtrip is flawed, or
Unicode Services understands it differently, or I am missing something.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Tony Harminc
Sent: Tuesday, June 12, 2012 12:02 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On 12 June 2012 13:59, Charles Mills  wrote:

> Fair enough. Unicode services reports however that it supports roundtrip
> conversion in many of these cases, including for example, 37 to 850
(pretty
> basic ASCII). What does roundtrip conversion mean? That is my fundamental
> question.

The CDRA Reference and Registry book (SC09-2190-00 - admittedly rather
old...) says this about Round Trip Integrity (in a larger section on
Criteria for Character Set Mismatch Management):

"The objective of this criterion is to send data from one system to
another one that has different representations of character data, and
retrieve it without loss. Often the "do not convert" choice is not
available. For example, data stored in a System/370 database is
configured to have all its graphic character data in one CCSID. If it
acts as a remote repository for data from a PC application, or from an
application in another System/370 using a different CCSID, the data
must be converted to the configured CCSID. The data is intended to be
retrieved by the same application without loss when it is converted
back for use in its original CCSID."

There's more, but I think the notion of Round Trip Integrity is clear
enough.

And having typed that in with the recollection that this book was not
available online, I see that there is an updated version at

http://www-01.ibm.com/software/globalization/cdra/

The bit I quoted is in chapter 6.

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Tony Harminc]

On 12 June 2012 13:59, Charles Mills  wrote:

> Fair enough. Unicode services reports however that it supports roundtrip
> conversion in many of these cases, including for example, 37 to 850 (pretty
> basic ASCII). What does roundtrip conversion mean? That is my fundamental
> question.

The CDRA Reference and Registry book (SC09-2190-00 - admittedly rather
old...) says this about Round Trip Integrity (in a larger section on
Criteria for Character Set Mismatch Management):

"The objective of this criterion is to send data from one system to
another one that has different representations of character data, and
retrieve it without loss. Often the "do not convert" choice is not
available. For example, data stored in a System/370 database is
configured to have all its graphic character data in one CCSID. If it
acts as a remote repository for data from a PC application, or from an
application in another System/370 using a different CCSID, the data
must be converted to the configured CCSID. The data is intended to be
retrieved by the same application without loss when it is converted
back for use in its original CCSID."

There's more, but I think the notion of Round Trip Integrity is clear enough.

And having typed that in with the recollection that this book was not
available online, I see that there is an updated version at

http://www-01.ibm.com/software/globalization/cdra/

The bit I quoted is in chapter 6.

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Right. I don't have a functional problem. There is no "fix" that I am
looking for. I have an understanding and documentation (mine!) problem.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 11:21 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I will have to leave the answer to your fundamental question for wiser and
more experienced heads to answer.  I would think you could file a PMR on
this and see what IBM has to say about it.  You might only get a doc change
out of it, but at least then there would be doc on what is and is not
supported for a guaranteed "round trip".

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

> I've lost the original post, what is the mapping you are trying to
achieve?

The original example was somewhat extreme.

CCSID 01027: Encoding scheme 1100 - EBCDIC, SBCS; Name JAPAN LATIN

CCSID 01208: Encoding scheme 7807 - UTF-8, UCS-2 transform; Name UTF-8 WITH
IBM 

Both the Unicode manual for z/OS V1R13 and the returned information from
CUNLINFO indicate that roundtrip conversion is supported for this CCSID
pair.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Steve Comstock
Sent: Tuesday, June 12, 2012 11:06 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

On 6/12/2012 11:59 AM, Charles Mills wrote:
> Peter, thanks. Believe me, I have done a *lot* of CCSID research.
>
> CCSID 1208 is Encoding scheme 7807 - UTF-8, UCS-2 transform; Name 
> UTF-8 WITH IBM PUA. It is *the* basic UTF-8 CCSID.
>
>> there is no way for EBCDIC 41 to "round trip" through classic 7-bit 
>> ASCII
>
> Fair enough. Unicode services reports however that it supports 
> roundtrip conversion in many of these cases, including for example, 37 
> to 850 (pretty basic ASCII). What does roundtrip conversion mean? That 
> is my fundamental question.

Roundtrip conversion means what you think: translate string A to string B
then back from string B to string A' then A = A'

Clearly you are not experiencing that.

Just as clearly, there are many mappings for which roundtrip conversion is
not possible. I've lost the original post, what is the mapping you are
trying to achieve?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Farley, Peter x23353]

I will have to leave the answer to your fundamental question for wiser and more 
experienced heads to answer.  I would think you could file a PMR on this and 
see what IBM has to say about it.  You might only get a doc change out of it, 
but at least then there would be doc on what is and is not supported for a 
guaranteed "round trip".

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of 
Charles Mills
Sent: Tuesday, June 12, 2012 1:59 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Peter, thanks. Believe me, I have done a *lot* of CCSID research. 

CCSID 1208 is Encoding scheme 7807 - UTF-8, UCS-2 transform; Name UTF-8 WITH
IBM PUA. It is *the* basic UTF-8 CCSID.

> there is no way for EBCDIC 41 to "round trip" through classic 7-bit ASCII

Fair enough. Unicode services reports however that it supports roundtrip
conversion in many of these cases, including for example, 37 to 850 (pretty
basic ASCII). What does roundtrip conversion mean? That is my fundamental
question.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 10:48 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, I don't have my "green card" with me at the moment, but IIRC 3F is the
EBCDIC SUB character, which means mapping it to ASCII 1A is correct.  EBCDIC
41 I think is "hard space" or something like that, and classic 7-bit ASCII
does not support such a character, so there is no way for EBCDIC 41 to
"round trip" through classic 7-bit ASCII (which is what I believe CCSID 1208
is, no?).  Now if you are transferring through ISO8879 (Latin-1) or UTF-8
instead, there may be a code point in one of those CCSID's that maps to
"hard space".  I don't have time for the necessary code point research at
the moment, but you may want to look into those CCSID's as an alternative to
CCSID 1208.

HTH

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 1:10 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Thanks, Peter. 

Please understand I am not criticizing or faulting z/OS Unicode Services. I
am just trying to explain to a customer the output they may expect to see
when they use our product that in turn uses Unicode Services (USS? -- LOL --
never mind).

Let's leave PCs out of it. If I were going z/OS to z/OS to z/OS, how could
both 3F and 41 make the round trip successfully?

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 9:04 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I believe that slide refers to "round trips" within the z/OS world only.
There is no statement that CCSID conversion by a system other than z/OS
(such as the PC ftp client in your example) will be covered in the 'round
trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible
iconv() instance to be able to make such a guarantee.  It's remotely
possible that transmission to a linux ftp client with an iconv() that is
compatible with the z/OS iconv() *might* support such a guarantee (e.g.,
z/Linux), but I wouldn't bet on it.  And Win systems are almost guaranteed
NOT to support such a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?
--

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibi

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Steve Comstock]

On 6/12/2012 11:59 AM, Charles Mills wrote:

Peter, thanks. Believe me, I have done a *lot* of CCSID research.

CCSID 1208 is Encoding scheme 7807 - UTF-8, UCS-2 transform; Name UTF-8 WITH
IBM PUA. It is *the* basic UTF-8 CCSID.


there is no way for EBCDIC 41 to "round trip" through classic 7-bit ASCII


Fair enough. Unicode services reports however that it supports roundtrip
conversion in many of these cases, including for example, 37 to 850 (pretty
basic ASCII). What does roundtrip conversion mean? That is my fundamental
question.


Roundtrip conversion means what you think: translate string A to string B then
back from string B to string A' then A = A'

Clearly you are not experiencing that.

Just as clearly, there are many mappings for which roundtrip
conversion is not possible. I've lost the original post, what
is the mapping you are trying to achieve?




Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 10:48 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, I don't have my "green card" with me at the moment, but IIRC 3F is the
EBCDIC SUB character, which means mapping it to ASCII 1A is correct.  EBCDIC
41 I think is "hard space" or something like that, and classic 7-bit ASCII
does not support such a character, so there is no way for EBCDIC 41 to
"round trip" through classic 7-bit ASCII (which is what I believe CCSID 1208
is, no?).  Now if you are transferring through ISO8879 (Latin-1) or UTF-8
instead, there may be a code point in one of those CCSID's that maps to
"hard space".  I don't have time for the necessary code point research at
the moment, but you may want to look into those CCSID's as an alternative to
CCSID 1208.

HTH

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 1:10 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Thanks, Peter.

Please understand I am not criticizing or faulting z/OS Unicode Services. I
am just trying to explain to a customer the output they may expect to see
when they use our product that in turn uses Unicode Services (USS? -- LOL --
never mind).

Let's leave PCs out of it. If I were going z/OS to z/OS to z/OS, how could
both 3F and 41 make the round trip successfully?

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 9:04 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I believe that slide refers to "round trips" within the z/OS world only.
There is no statement that CCSID conversion by a system other than z/OS
(such as the PC ftp client in your example) will be covered in the 'round
trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible
iconv() instance to be able to make such a guarantee.  It's remotely
possible that transmission to a linux ftp client with an iconv() that is
compatible with the z/OS iconv() *might* support such a guarantee (e.g.,
z/Linux), but I wouldn't bet on it.  And Win systems are almost guaranteed
NOT to support such a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@bama.ua.edu with the message: INFO IBM-MAIN

This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential.
If the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you h

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Peter, thanks. Believe me, I have done a *lot* of CCSID research. 

CCSID 1208 is Encoding scheme 7807 - UTF-8, UCS-2 transform; Name UTF-8 WITH
IBM PUA. It is *the* basic UTF-8 CCSID.

> there is no way for EBCDIC 41 to "round trip" through classic 7-bit ASCII

Fair enough. Unicode services reports however that it supports roundtrip
conversion in many of these cases, including for example, 37 to 850 (pretty
basic ASCII). What does roundtrip conversion mean? That is my fundamental
question.

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 10:48 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, I don't have my "green card" with me at the moment, but IIRC 3F is the
EBCDIC SUB character, which means mapping it to ASCII 1A is correct.  EBCDIC
41 I think is "hard space" or something like that, and classic 7-bit ASCII
does not support such a character, so there is no way for EBCDIC 41 to
"round trip" through classic 7-bit ASCII (which is what I believe CCSID 1208
is, no?).  Now if you are transferring through ISO8879 (Latin-1) or UTF-8
instead, there may be a code point in one of those CCSID's that maps to
"hard space".  I don't have time for the necessary code point research at
the moment, but you may want to look into those CCSID's as an alternative to
CCSID 1208.

HTH

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 1:10 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Thanks, Peter. 

Please understand I am not criticizing or faulting z/OS Unicode Services. I
am just trying to explain to a customer the output they may expect to see
when they use our product that in turn uses Unicode Services (USS? -- LOL --
never mind).

Let's leave PCs out of it. If I were going z/OS to z/OS to z/OS, how could
both 3F and 41 make the round trip successfully?

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 9:04 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I believe that slide refers to "round trips" within the z/OS world only.
There is no statement that CCSID conversion by a system other than z/OS
(such as the PC ftp client in your example) will be covered in the 'round
trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible
iconv() instance to be able to make such a guarantee.  It's remotely
possible that transmission to a linux ftp client with an iconv() that is
compatible with the z/OS iconv() *might* support such a guarantee (e.g.,
z/Linux), but I wouldn't bet on it.  And Win systems are almost guaranteed
NOT to support such a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@bama.ua.edu with the message: INFO IBM-MAIN

This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential.
If the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by e-mail
and delete the message and any attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to lists...@bama.ua.edu with the message: INFO IBM-MAIN

--

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Farley, Peter x23353]

Well, I don't have my "green card" with me at the moment, but IIRC 3F is the 
EBCDIC SUB character, which means mapping it to ASCII 1A is correct.  EBCDIC 41 
I think is "hard space" or something like that, and classic 7-bit ASCII does 
not support such a character, so there is no way for EBCDIC 41 to "round trip" 
through classic 7-bit ASCII (which is what I believe CCSID 1208 is, no?).  Now 
if you are transferring through ISO8879 (Latin-1) or UTF-8 instead, there may 
be a code point in one of those CCSID's that maps to "hard space".  I don't 
have time for the necessary code point research at the moment, but you may want 
to look into those CCSID's as an alternative to CCSID 1208.

HTH

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of 
Charles Mills
Sent: Tuesday, June 12, 2012 1:10 PM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Thanks, Peter. 

Please understand I am not criticizing or faulting z/OS Unicode Services. I
am just trying to explain to a customer the output they may expect to see
when they use our product that in turn uses Unicode Services (USS? -- LOL --
never mind).

Let's leave PCs out of it. If I were going z/OS to z/OS to z/OS, how could
both 3F and 41 make the round trip successfully?

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 9:04 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I believe that slide refers to "round trips" within the z/OS world only.
There is no statement that CCSID conversion by a system other than z/OS
(such as the PC ftp client in your example) will be covered in the 'round
trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible
iconv() instance to be able to make such a guarantee.  It's remotely
possible that transmission to a linux ftp client with an iconv() that is
compatible with the z/OS iconv() *might* support such a guarantee (e.g.,
z/Linux), but I wouldn't bet on it.  And Win systems are almost guaranteed
NOT to support such a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Thanks, Peter. 

Please understand I am not criticizing or faulting z/OS Unicode Services. I
am just trying to explain to a customer the output they may expect to see
when they use our product that in turn uses Unicode Services (USS? -- LOL --
never mind).

Let's leave PCs out of it. If I were going z/OS to z/OS to z/OS, how could
both 3F and 41 make the round trip successfully?

Charles

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 9:04 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

I believe that slide refers to "round trips" within the z/OS world only.
There is no statement that CCSID conversion by a system other than z/OS
(such as the PC ftp client in your example) will be covered in the 'round
trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible
iconv() instance to be able to make such a guarantee.  It's remotely
possible that transmission to a linux ftp client with an iconv() that is
compatible with the z/OS iconv() *might* support such a guarantee (e.g.,
z/Linux), but I wouldn't bet on it.  And Win systems are almost guaranteed
NOT to support such a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Paul Gilmartin]

On Tue, 12 Jun 2012 12:03:55 -0400, Farley, Peter x23353 wrote:

>I believe that slide refers to "round trips" within the z/OS world only.  
>There is no statement that CCSID conversion by a system other than z/OS (such 
>as the PC ftp client in your example) will be covered in the 'round trip" 
>guarantee.
>
>You have to be transmitting and receiving with the same or a very compatible 
>iconv() instance to be able to make such a guarantee.  It's remotely possible 
>that transmission to a linux ftp client with an iconv() that is compatible 
>with the z/OS iconv() *might* support such a guarantee (e.g., z/Linux), but I 
>wouldn't bet on it.  And Win systems are almost guaranteed NOT to support such 
>a guarantee.
> 
(More at:

http://publib.boulder.ibm.com/infocenter/zos/v1r11/topic/com.ibm.zos.r11.halz001/mbdata.htm
)
Which apparently implies that FTP doesn't support conversion between single-byte
code pages and multi-byte code pages.  Why not!?

Otherwise, I'd be inclined to try from a desktop client:

BINARY
QUOTE SITE ASCII
QUOTE SITE MBDATACONN(IBM-1047,UTF-8)

or whatever.  Let z/OS perform all the conversion.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Farley, Peter x23353]

I believe that slide refers to "round trips" within the z/OS world only.  There 
is no statement that CCSID conversion by a system other than z/OS (such as the 
PC ftp client in your example) will be covered in the 'round trip" guarantee.

You have to be transmitting and receiving with the same or a very compatible 
iconv() instance to be able to make such a guarantee.  It's remotely possible 
that transmission to a linux ftp client with an iconv() that is compatible with 
the z/OS iconv() *might* support such a guarantee (e.g., z/Linux), but I 
wouldn't bet on it.  And Win systems are almost guaranteed NOT to support such 
a guarantee.

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of 
Charles Mills
Sent: Tuesday, June 12, 2012 11:49 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

I certainly understand that not every character in a given CCSID maps
meaningfully to another CCSID. That's why I said "possibly meaningless" in
my original question.

Charles
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 8:41 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

This is a false assumption: "... every code point in the from CCSID
translates to a unique (possibly "meaningless") code point in the to CCSID
...".

There is no guarantee that all code points in a given CCSID map to a
"unique" code point in any other CCSID.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

Well, Peter, that's certainly consistent with what I see.

I'm looking, however, at slide 11 of
http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/index.jsp?topic=/c
om.ibm.iea.zos/zos/1.9/IntegratingNewAppOnzOS/zOSV1R9_Integrating_newAppl_LE
UnicodeServices/player.html . (You may have to unfold that URL.)

It says "R - Roundtrip conversion. Roundtrip conversions between two CCSIDs
assure that all characters making the roundtrip arrive as they were
originally." How is that going to be accomplished if both 3F and 41
translate to 1A? How will they make the round trip back to what they were?

What does technique R mean?

I certainly understand that not every character in a given CCSID maps
meaningfully to another CCSID. That's why I said "possibly meaningless" in
my original question.

Charles
-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf
Of Farley, Peter x23353
Sent: Tuesday, June 12, 2012 8:41 AM
To: IBM-MAIN@bama.ua.edu
Subject: Re: Anyone a Unicode Services expert? -- roundtrip conversion

This is a false assumption: "... every code point in the from CCSID
translates to a unique (possibly "meaningless") code point in the to CCSID
...".

There is no guarantee that all code points in a given CCSID map to a
"unique" code point in any other CCSID.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Re: Anyone a Unicode Services expert? -- roundtrip conversion

[Farley, Peter x23353]

This is a false assumption: "... every code point in the from CCSID translates 
to a unique (possibly "meaningless") code point in the to CCSID ...".

There is no guarantee that all code points in a given CCSID map to a "unique" 
code point in any other CCSID.

Peter

-Original Message-
From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of 
Charles Mills
Sent: Tuesday, June 12, 2012 11:32 AM
To: IBM-MAIN@bama.ua.edu
Subject: Anyone a Unicode Services expert? -- roundtrip conversion

My understanding of "roundtrip conversion" is that every code point in the
from CCSID translates to a unique (possibly "meaningless") code point in the
to CCSID so that if for example a customer is so foolish as to transmit, for
example, an object deck from z/OS to a PC in "text" format, and then
transfer it back to z/OS the same way, he will have a good object deck. 

I am running Kirk Wolf's excellent showtrtab with the command showtrtab -s
1027 -t 1208 -q R

The results I see include the following:

3F:  1A 
40:  20 
41:  1A

Several other EBCDIC code points also translate to ASCII sub, 0x1a. So the
question is if multiple EBCDIC code points all translate to 0x1a, how the
heck is this a round trip translation?

(I'm not alleging a bug in showtrtab. I am getting the same results with my
code and ran showtrtab as a reality check and a clearer example.)

Where am I going wrong?

Charles 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN

Anyone a Unicode Services expert? -- roundtrip conversion

[Charles Mills]

My understanding of "roundtrip conversion" is that every code point in the
from CCSID translates to a unique (possibly "meaningless") code point in the
to CCSID so that if for example a customer is so foolish as to transmit, for
example, an object deck from z/OS to a PC in "text" format, and then
transfer it back to z/OS the same way, he will have a good object deck. 

I am running Kirk Wolf's excellent showtrtab with the command showtrtab -s
1027 -t 1208 -q R

The results I see include the following:

3F:  1A 
40:  20 
41:  1A

Several other EBCDIC code points also translate to ASCII sub, 0x1a. So the
question is if multiple EBCDIC code points all translate to 0x1a, how the
heck is this a round trip translation?

(I'm not alleging a bug in showtrtab. I am getting the same results with my
code and ran showtrtab as a reality check and a clearer example.)

Where am I going wrong?

Charles 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN