[GNC-dev] [MAINT] Unplanned network outage
Hi, My ISP had an unplanned network outage starting around 2am this morning that took all connectivity offline -- their router ran out of memory and took out all BGP routes. Their router has since been rebooted and packets are now flowing again. Sorry for any inconvenience this may have caused. -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] [MAINT] Unplanned network outage for code.gnucash.org
Thanks for all this work Derek... Not a whole lot of fun :( Jean On 6/7/22 4:09 PM, Derek Atkins wrote: Hi All, tl;dr: Double-whammy of UPS failing and then AT Router failing kept me offline for two days, including code.gnucash.org. Finally received the new router around 5pm so the network is back up, but the UPS is still failing so equipment currently is up but not protected. Long version: Around 10:30pm on Sunday night by TV displayed "Network Cable Unplugged". I didn't think anything of it until the show I was streaming stopped a few minutes later. I went down to the basement to discover that everything plugged into the UPS was powered off. After fighting with the UPS (the 'on' button didn't do anything) I figured out how to reset it by unplugging it and the plugging it in again. This got everything up and running, but the AT router would not authenticate to the network so I couldn't get online. I thought it might've been a result of my ER-4 so I removed that from the equation, but it didn't help. After an hour on the phone with a nice AT tech in Slovenia I had a new modem on order and I finally got to bed at 12:30 knowing there wasn't anything else I could do. The tech said I should have the modem by Tuesday, but it was possible I could get it Monday. Monday morning I called AT to try to get tracking info, but my modem order was still processing. I called back again Monday afternoon and got FedEx tracking for the box, which was being sent to me from California. The modem arrived today around 5pm, and I quickly installed it, only to find that no, it did not have my static IP config in there. So I called back to AT and got another tech who was able to get me configured. I verified it was working while he was on the phone, and then worked with him on a few additional configuration changes, and verified my billing (which of course AT screwed up, too, but he claims to have fixed it -- I'll check that tomorrow). However, during all this, the UPS powered-down my equipment several times over the past couple days, including at least four times in a 3-hour spread from 3:20-6pm. I finally decided that the UPS wasn't helping so I unplugged all the equipment from the UPS and into my mainline (unprotected) circuit, while I work on figuring out why the UPS keeps turning off. So for now: 1) network is up, but I might be subject to NAT tables filling up until I can return to using my ER-4 gateway. 2) systems are up, but are not UPS protected 3) code is back up and running. I'm going to have several maintenance windows going forward to return the ER-4 into service, and, eventually, once I get the UPS repaired (whatever that means). Any network outage due to the former should be relatively short. However I will have to take all my hardware down to change where it's plugged in. -derek PS: Yes, I really should look into a backup to my main fiber network, although that wouldn't have helped with the UPS turning off its outputs! ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage for code.gnucash.org
Hi All, tl;dr: Double-whammy of UPS failing and then AT Router failing kept me offline for two days, including code.gnucash.org. Finally received the new router around 5pm so the network is back up, but the UPS is still failing so equipment currently is up but not protected. Long version: Around 10:30pm on Sunday night by TV displayed "Network Cable Unplugged". I didn't think anything of it until the show I was streaming stopped a few minutes later. I went down to the basement to discover that everything plugged into the UPS was powered off. After fighting with the UPS (the 'on' button didn't do anything) I figured out how to reset it by unplugging it and the plugging it in again. This got everything up and running, but the AT router would not authenticate to the network so I couldn't get online. I thought it might've been a result of my ER-4 so I removed that from the equation, but it didn't help. After an hour on the phone with a nice AT tech in Slovenia I had a new modem on order and I finally got to bed at 12:30 knowing there wasn't anything else I could do. The tech said I should have the modem by Tuesday, but it was possible I could get it Monday. Monday morning I called AT to try to get tracking info, but my modem order was still processing. I called back again Monday afternoon and got FedEx tracking for the box, which was being sent to me from California. The modem arrived today around 5pm, and I quickly installed it, only to find that no, it did not have my static IP config in there. So I called back to AT and got another tech who was able to get me configured. I verified it was working while he was on the phone, and then worked with him on a few additional configuration changes, and verified my billing (which of course AT screwed up, too, but he claims to have fixed it -- I'll check that tomorrow). However, during all this, the UPS powered-down my equipment several times over the past couple days, including at least four times in a 3-hour spread from 3:20-6pm. I finally decided that the UPS wasn't helping so I unplugged all the equipment from the UPS and into my mainline (unprotected) circuit, while I work on figuring out why the UPS keeps turning off. So for now: 1) network is up, but I might be subject to NAT tables filling up until I can return to using my ER-4 gateway. 2) systems are up, but are not UPS protected 3) code is back up and running. I'm going to have several maintenance windows going forward to return the ER-4 into service, and, eventually, once I get the UPS repaired (whatever that means). Any network outage due to the former should be relatively short. However I will have to take all my hardware down to change where it's plugged in. -derek PS: Yes, I really should look into a backup to my main fiber network, although that wouldn't have helped with the UPS turning off its outputs! -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage. yet again. (Upstream ISP failure)
Hi again, You may have noticed that code went offline yesterday -- there was a coolant failure at my ISP which caused the colo room temperature to raise to 122F (50C). Obvoiusly computers don't like to be at that temp. Water was restored earlier today and the ISP services were restored. The equipment at my premises were online the whole time, and indeed I was even able to reach my gateway through another address. Sorry for any inconvenience. Alas, there is very little I could do about *this* particular outage. -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage. again.
Hi All. TL;DR: You may have noticed a 22-hour network outage of gnucash email/wiki/git. It's all back now and should be stable going forward. Long Version: Yesterday around 1pm an Amazon truck decided to flay my (temporary) fiber line. You may recall that my network went out a couple weeks ago. The way my ISP performs the repair is a 3-part journey: 1) A tech comes to run a temporary line to get me up and running 2) A crew comes to deploy and bury a new service line 3) Another tech comes to connect the new service line When #1 happened they ran the line across the driveway. This isn't too surprising. Then #2 happened a week or so ago; it was actually perfect timing with some landscaping we were doing. Then I was waiting for #3 to happen. It was, apparently, scheduled for this Wednesday (May 12), but of course I didn't know that until yesterday when I was enjoying some music and the Amazon truck drove up my driveway, then back down, and the music stopped. OOPS! I called yesterday but they only made a note. So I called again today, got a really nice call-center agent in Eastern Europe somewhere, and she got me set up with an appointment from 4-8pm today. Then about an hour ago they called me to ask if they could come "now", and I said "YES". A short time later they arrived, connected the new service line, and then all my stuff came up on its own. So we're back, and should be pretty stable until the next significant power hit or a tree decides to take down something well off my property!! Happy Gnucashing. And sorry for the inconvenience. -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage
HI all, We had a short service outage today. My power went out (literally the power blew a fuse --- at the street corner) around 12:35pm. It was restored around 5pm. Unfortunately my UPS battery died around 4:45. Once the power came back, the systems all rebooted and came back on their own (around 5:15 or so). Sorry for the inconvenience. -derek PS: At 12:35 I heard a loud BANG and the power went out! I notified people on IRC immediately and went into power-saving mode. They were originally supposed to restore power by 3:30, which meant there would not have been any outage. Then they pushed it another hour and a half, which was just a LITTLE too much for the UPS. Alas. Again, sorry for any inconvenience. -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage
HI, Around 1pm EDT yesterday my network broke. I suspect it was a fiber "jerk" (not quite a fiber cut), but enough force to cause the fiber to stop sending light through. I wasn't home when it happened, but the Tech yelled at me about it. A new fiber has been run and bits are flowing as of 11:15am today. Sorry for the inconvenience (again). In about a week or two there will be an unplanned 20-minute outage when they bury a new line and swap it over. Thanks, -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage
Good morning GnuCashers, Just wanted to report on a (relatively short) unplanned network outage overnight. Around 12:30am US/ET my network went away. It came back around 1:15am. This was my upstream ISP doing some maintenance work. This is the same kind of maintenance that took me completely offline for hours in the past. The GOOD news is that my systems scripts handled the changes just fine and didn't get wedged! As a result of this outage I know the flatpak nightly build failed. I suspect the windows nightly build failed as well. I have not manually restarted them; new builds will happen tonight. Thanks, and have a great Thursday. Happy Earth Day! -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] [MAINT] Unplanned network outage
Derek, If you figure out LTE bulk data service please PM me the info. I'd like to switch away from my 800kbs upload with CenturyCantLink. --Steve On 3/10/21 9:39 AM, Derek Atkins wrote: Hi everyone, I apologize for the 23-hour outage. Some landscape installers did some work unannounced and sliced my network line in the process. As you can tell, it's back now! Yay rapid response. Even though this is the last operation that I expect to have, I am still seriously considering an LTE backup, and an insurance policy. I can get the hardware relatively inexpensively -- it's the data service that I'm trying to find. Again, sorry for any inconvenience. Just me, it wasn't easy here, either! LOL. Happy Gnucashing! -derek -- Stephen M Butler, PMP, PSM stephen.m.butle...@gmail.com kg...@arrl.net 253-350-0166 --- GnuPG Fingerprint: 8A25 9726 D439 758D D846 E5D4 282A 5477 0385 81D8 ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage
Hi everyone, I apologize for the 23-hour outage. Some landscape installers did some work unannounced and sliced my network line in the process. As you can tell, it's back now! Yay rapid response. Even though this is the last operation that I expect to have, I am still seriously considering an LTE backup, and an insurance policy. I can get the hardware relatively inexpensively -- it's the data service that I'm trying to find. Again, sorry for any inconvenience. Just me, it wasn't easy here, either! LOL. Happy Gnucashing! -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage
Hi, Some of you may have noticed that code went offline around 2:30am US/ET this morning. My ISP made a change which caused my "magic box" (a device I use to work around a NAT table limit) to stop working. I manually corrected the issue at 7:15am and have written some code to attempt to do that automagically next time. Of course I can't test that until the next time $ISP decides to do some network hardware work. I apologize for any inconvenience. -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage for GnuCash Server
Hi All, TL;DR: There was an unplanned network outage for code.gnucash.org for the past 16 hours. I was able to route around the damage and packets are flowing again as of about 18h45 US/EDT. Long version: Sometime around 3:30am US/EDT my network stopped talking to my ISP. Unfortunately I was away on holiday for a quick 2-day lake getaway and the only debugging I could do remotely was that I determined I could not reach any of my machines on my raw ISP network. Once I got home this afternoon I was able to debug it further and determine that a bridge I had put in place between my ONT, Modem, and local devices was no longer properly working. I removed that box and reconnected without it an everything came back up. I do want to place that box back into service, because it allows me to bypass the modem NAT table limits, but for now I will keep the network running without it until I can find the time to track down what changes AT made to the configuration that caused it to break. But until I can find the time to do that (which will entail taking the network down), I will keep it up and running as a direct-connect. Sorry for any inconvenience this may have caused. Trust me, it wasn't pleasant for me, either, waking up from 100 miles away to see that the network was down! Have a great morning/evening/rest-of-your-day and happy GnuCashing! -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] [MAINT] Unplanned network outage for GnuCash Server
Hi, TL;DR: My ISP's upstream router failed causing my network to go away for the last 36 hours. IPv4 service is back as of 9am US/EDT today. IPv6 service should be back shortly (and and I'm writing this, IPv6 is back). Long version: Around 8pm US/EDT on Sunday, July 12th I noticed that my network had failed. I traced it down to the fact that I couldn't reach my upstream ISP router so I sent email asking for assistance. There was nothing else I could do from my end until I heard back from them. Yesterday I reached out again and was told that the router crashed hard and they were working on replacing it and trying to find a backup configuration as well. That was as of 3am Monday morning. Yesterday evening I noticed that the router was reachable again, but it wasn't routing my network to me, so this morning I reached out yet again, and they were able to re-install my configuration, but they had a bug in the IPv6 config. Still, I noticed email started flowing shortly after 9am. Once they corrected the IPv6 config (which happened while I was typing this in), everything came back up and now appears to be reachable. Because it was only down for 36 hours, no email should be lost, only delayed, and no other local data was lost on the GnuCash server. I'm sorry for any inconvenience this may have caused. Alas, there's not much I can do about an upstream router blowing up. :( Happy GnuCashing y'all. :-D -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
