Eldon, The formula that you are using to calculate the standard deviation (with N-1 in the denominator) is the proper function when the number of elements (N) used in the calculation represent a sample of the total population.
DB2 assumes that the result set it is using represents the entire population and therefore uses the proper definition for caluclating STDDEV for an entire population (which has N in the denominator, not N-1). Good luck, Ian ----- Original Message ----- From: "Eldon B Tucker" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Wednesday, November 14, 2001 2:19 PM Subject: DB2EUG: db2 computation of standard deviation > > I was looking at the formula to compute a standard > deviation using sum and sum of squares (without having > to know the mean in advance.) > > Using udb 7.1.0, I tried the formula, comparing it > to what the stddev function returned, and noticed > a different answer. I varied the formula to do a > final divide by (n) rather than by (n - 1) and then > got the same answer as udb. Could this be a problem > with the stddev function? > > My query ran: > > with raw ( x ) as ( select dec(week_seq,17,5) from hc.t_week ) > select sqrt((sum(x*x) - ((sum(x)*sum(x))/count(*)))/( count(*) - 1 )) as > "right" > , sqrt((sum(x*x) - ((sum(x)*sum(x))/count(*)))/ count(*) ) as > "wrong" > , stddev(x) as "db2" > from raw ; > > And the results were: > > right wrong db2 > ------------------------ ------------------------ ------------------------ > +1.28027340829996E+002 +1.27882758806651E+002 +1.27882758806651E+002 > > -- Eldon Tucker > > > > ===== > To unsubscribe, send 'unsubscribe' to [EMAIL PROTECTED] > For other info (and scripts), see http://people.mn.mediaone.net/scottrmcleod > ===== To unsubscribe, send 'unsubscribe' to [EMAIL PROTECTED] For other info (and scripts), see http://people.mn.mediaone.net/scottrmcleod
