Platform Outage

6 September - The infrastructure provider, Interserver, had a major incident earlier today causing one of their virtual node servers to go down. The TTN database is hosted on one of these VPS's which caused all database queries to fail and therefore bring the platforms down. TTN Social, TTN Tube and TTN Shop (currently under development) were all affected.

I noticed the outage at 6:50am UK time and reported it to the supplier at 7:08am after my initial investigations. They informed me that they are failing over the node to a different host and that it should take but a few minutes. I contacted them every 30 minutes for an update from the time of logging the ticket. I was assured that they had raised the incident to "Critical" from a "Standard" and was dealing with it.

At 10:23 am UK, I was informed that the server was online at which point I posted the Announcement on Social.

I do make weekly backups of all data, but didn't want to use that as the past week's data would have been "lost" when the database came back online. I therefore decided to give the tech'ies a bit of time to rather resolve the issue that lose the week's data.

I will be putting adding database replication into the solution architecture to ensure that I can more speedily failover to a different database in the event of such an occurrence.