On 10/10/11 13:31, Allen Egerton wrote: > On 10/10/2011 8:12 AM, Israel, John R. wrote: >> I am coming late to this party, but wanted to throw out a bug that was just >> reported to Rocket. We found that PERCISION was causing a calculation to >> round in some cases and truncate in others. I am not sure where this stands >> because someone else on my team is driving this ticket, but it was pretty >> disturbing that the same calculation worked differently. >> >> John Israel > > John, > > Here's the essence of an open ticket we've got with Rocket for Universe > 10.3.7 on a windows server: > > >> CT ADE.BP TT > > TT > 0001 FOR II = 1 TO 18 > 0002 CRT FMT(II, "2R"): " ": FMT(10^II + 1, "35R,") > 0003 NEXT II > 0004 STOP >> RUN ADE.BP TT > 1 11 > 2 101 > 3 1,001 > 4 10,001 > 5 100,001 > 6 1,000,001 > 7 10,000,001 > 8 100,000,001 > 9 1,000,000,001 > 10 10,000,000,001 > 11 100,000,000,001 > 12 1,000,000,000,001 > 13 10,000,000,000,001 > 14 100,000,000,000,001 > 15 1,000,000,000,000,000 > 16 10,000,000,000,000,000 > 17 100,000,000,000,000,000 > 18 1,000,000,000,000,000,000 > > See anything wrong with the least significant column? > I notice that it starts going wrong beyond 14... Which I understood was the maximum possible value of PRECISION. Maybe you can set it higher, but it doesn't *work* any higher, because, iirc, a DOUBLE PRECISION mantissa only has 14 decimal digits of precision.
Drifting slightly, Brett's comment that PRECISION only affected display output is interesting. I always thought it actually controlled the store (ie A = B * C, the calculation would be accurate precision, but A would be stored to PRECISION places). HOWEVER this is what I understood from Pr1me days. And, iirc, Pr1me had BCD microcode which INFORMATION used, so it makes sense that Pr1me behaviour would be noticeably different (and that Pr1me behaviour would also be far more "sensible" when viewed through our decimal eyes :-) Cheers, Wol _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
