RE: ICU library version - was: IBM DFDL is upgrading ICU level

[Interrante, John A (GE Research, US)]

The fix to ICU's Java code changes only a single line and adds a new test case. 
 But that single line was causing 400 test cases in IBM DFDL to fail because 
it'd made ICU's Java code behave differently than ICU's C code (ouch).  The 
difference seems to be mostly rounding of real numbers, e.g., df.format(1235.5) 
with pattern "##00" was producing "1236.0" instead of the expected "1236".  
Ideally the PR will be merged, ICU Java will produce "1236" like ICU C does, 
and IBM's 400 test cases will pass again even though some of Daffodil's test 
cases may or may not even notice this difference in behavior (it'll depend on 
whether any test were using this kind of pattern on real numbers).

-Original Message-
From: Beckerle, Mike  
Sent: Wednesday, May 19, 2021 8:54 AM
To: dev@daffodil.apache.org
Subject: EXT: ICU library version - was: IBM DFDL is upgrading ICU level

DFDL implementors at IBM have noticed some issues with the ICU library worth 
noting.
There is an ICU pull request to fix, targeted at ICU version 70.1.


From: 
Subject: IBM DFDL is upgrading ICU level

Hi Mike

We are moving up the level of ICU that IBM DFDL is built with, as it is still 
on 51 which is out-of-support. We are trying 68.x.
In the process we found several of our regression tests failed due to behaviour 
changes in lax decimal/calendar processing.
If you recall we deliberately changed the DFDL 1.0 spec to make lax 
implementation-dependent/defined (I forget which).
We've analysed the differences and most are to do with bug fixes or other 
changes that are acceptable or benign or we don't think any of our customer 
will hit.
However, we got 400 failures in our Java version which didn't appear in our C 
version.
This looks to be have been caused by a regression somewhere, we think back in 
62 - see https://unicode-org.atlassian.net/browse/ICU-20425.
ICU have accepted there is a problem and the fix is in PR 
https://github.com/unicode-org/icu/pull/1726 which is targeted at 70.1.
Letting you know as moving to 70.1 and higher might therefore cause a Daffodil 
behaviour change.

Regards

Steve Hanson

IBM Hybrid Integration, Hursley, UK
Architect, IBM 
DFDL<http://www.ibm.com/developerworks/library/se-dfdl/index.html>
Co-Chair, OGF DFDL Working Group<http://www.ogf.org/dfdl/>

ICU library version - was: IBM DFDL is upgrading ICU level

[Beckerle, Mike]

DFDL implementors at IBM have noticed some issues with the ICU library worth 
noting.
There is an ICU pull request to fix, targeted at ICU version 70.1.


From: 
Subject: IBM DFDL is upgrading ICU level

Hi Mike

We are moving up the level of ICU that IBM DFDL is built with, as it is still 
on 51 which is out-of-support. We are trying 68.x.
In the process we found several of our regression tests failed due to behaviour 
changes in lax decimal/calendar processing.
If you recall we deliberately changed the DFDL 1.0 spec to make lax 
implementation-dependent/defined (I forget which).
We've analysed the differences and most are to do with bug fixes or other 
changes that are acceptable or benign or we don't think any of our customer 
will hit.
However, we got 400 failures in our Java version which didn't appear in our C 
version.
This looks to be have been caused by a regression somewhere, we think back in 
62 - see https://unicode-org.atlassian.net/browse/ICU-20425.
ICU have accepted there is a problem and the fix is in PR 
https://github.com/unicode-org/icu/pull/1726 which is targeted at 70.1.
Letting you know as moving to 70.1 and higher might therefore cause a Daffodil 
behaviour change.

Regards

Steve Hanson

IBM Hybrid Integration, Hursley, UK
Architect, IBM 
DFDL
Co-Chair, OGF DFDL Working Group