Nick Andrew wrote:
On Thu, Apr 01, 2010 at 05:47:37PM +1100, Jeff Waugh wrote:
<quote who="Nick Andrew">
On Thu, Apr 01, 2010 at 03:27:23PM +1100, Jeff Waugh wrote:
Not sure what Linux has to do with this -- there's far more going on
(with dates and times especially) in a complex stack of software than
just the OS. Consider the amount of legacy software and multi-system
integration involved in a bank's computing environment.
I see it more like "software superstition". Bad things might happen - we
don't know, we won't (or can't) test it, and we won't (or can't) fix it.
Sorry dudes, but this just sounds like Open Source snootiness from the
small end of town.
I want my bank to run on logic, not voodoo.
... and you say this with broad knowledge of the many and varied systems in
place? There may just be an entirely sensible reason why one or more pieces
of the system, at this point in its evolution, requires hand-holding or no
external access during a DST changeover.
The bank either knows that their system won't work during the DST changeover,
or they suspect that it won't work. I suspect it's the latter, but either
situation is a worry.
Its called unknown unknowns.
The bank may well be pretty sure that nothing will go wrong but given
the cost/benefit ratio its prudent not to take the chance that there is
one line of code somewhere or another in the many tens of millions they
have that will freak out when the clock goes backwards.
If it zeros out everybodys account balances due to an incorrect interest
calculation wrapping or something then even if they fix it by 10:00AM
they are going to be swamped by a storm of hate the likes of which you
can only dream.
DST changeover is predictable. Well, it's predictable that it will happen
at some time, but the changeover date itself varies according to the whim
of politicians. The bank should have expected DST, and built their systems
to cope when it changes.
I presume they have, they handle it by turning things off for an hour in
the middle of the night, no great loss.
Also the only information we have is on netbank which is perhaps the
crappiest section of the system.
Its links to the legacy systems are fragile at best and having people
stick transactions in during the time transition could well cause weirdness.
On the other hand, if they don't know that something will break and just
suspect it, that's a worry because the bank should understand very deeply
how their systems work, to achieve maximum reliability.
I presume they do, but again its called unknown unknowns.
They don't know what they don't know, and given the cost of failure its
cheap not to risk it.
On the third hand, hearing about how they can't manage a simple DNS change,
getting DST right is probably the least of their worries.
It wasn't a DNS change, My understanding is coming 3rd hand.
A data centre went down.
They have some kind of geoIP+ load balancer + user stickyness system in
place that is meant to keep users generally accessing sites close to
them and also sticking to them.
Initially after the failure their own DNS servers were sending users to
the dead data centre.
Once they had fixed that they had problems with ISP's caching the
incorrect DNS and spreading it over the rest of their users or something
to that effect.
Also they had a problem that the failover systems they had in place sent
all the traffic to one other centre and caused it to become overloaded
leading to 10 minute pageloads.
I understand that the problem was brought about by some contractors
cutting cables in a cable duct that was running in a lift shaft or
something along those lines.
It was a proper cut of a phat cable at that.
"Whee, Linux!" is not an answer if
it's an application problem - and that's being polite. "Whee, Linux!" might
not be a useful answer for plenty of other reasons.
Yep, and I never said it was.
Nick.
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
