Finally got us back online this evening after being dead since very early Tuesday morning.
On August 8th a very bad thunderstorm in the area tried to zap equipment. Battery backup kicked in, but a repeating brownout situation for the next 4 or 5 minutes really wore out the battery as it kept switching from AC power to battery backup (5 times in as many minutes.) By the time the worst was over, the UPS was sounding its alarm that battery needed replacing, because it would only charge to between 3% and 5%. This is enough to keep equipment running on AC power, but they'd go out immediately if another power outage occurred. I don't keep spare batteries on hand, so ordered one and it arrived on the 14th.
And that's where things went really wrong, from me being lazy about replacing it. After all, equipment had been running fine for a few days and I don't like having to move everything aside and unplug everything things to get to the UPS. So I was kind of sitting on it until I was good and ready to spend the time replacing it as the risk was low.
Then sometime Tuesday morning the 20th, the battery finally gave out and dropped below the threshold to safely ensure a delivery of clean AC power and the UPS immediately shut off secondary outlets (like the one to the router) in order to keep server running as long as possible so it could safely shutdown (it didn't have time, but no harm was done upon power up later.)
So Tuesday I was forced to take the time and replace the battery, and got everything up and running again, except the websites didn't work. I reset router several times, rebooted servers over and over, checked settings on both, you name it. I was racking my brain trying to figure out why nothing was working. I even went through the process of getting a new IP three times and updating the GoDaddy DNS records each time in an effort to get it to work. For the last several days I'd got through this exercise trying to tweak things to get them to work.
Finally chanced upon a diagnostics screen in the router configuration that was showing current leased IP as something that wasn't correct. Like most providers, I get an IPv4 address assigned via DHCP, but its practically mine until I release it or go more than 4 hours without traffic so it very rarely needs changing. Of course, I'd already accounted for the IP change that would have occurred Tuesday morning. But the router diagnostics said it was holding a different IP than assigned, which I could see on sites like whatismyip.com. This explained why I could ping my network and get response, but trying to sent anything over TCP came back unresponsive as the router tried to reply with an IP address that didn't match request. So I had to force it to release and renew the DHCP lease from this diagnostics window instead of the main configuration screen, go through the GoDaddy DNS updates again and that's what's finally got it working. When it lost power Tuesday morning it must have got it stuck in its flash memory that it was holding an IP address it no longer owned, and this effectively made all outbound traffic look like I was spoofing another address.
But now everything is running again and all is back to the way it should be. Sorry for being gone a few days.
