OUTAGE - Database Server Down

[Gobberwart]

As you're probably already aware, the game is currently down. Unfortunately I'm at work and have absolutely NO access to anything, so unfortunately there is NOTHING I can do directly. Theoretically the service provider has scripts that can detect this kind of thing and restart the server, but I've not yet seen these work and manual intervention is likely to be required.

I have logged a critical support ticket with the provider and hope they will attend to it quickly.

[Gobberwart]

Nearly 3 hours later, the game's back up.

Response from HostV:

"Mysql service was restarted and is working fine now."

I've also asked them to look into why the built-in script that is supposed to identify problems and restart the service are failing. They appear to identify the problem and attempt to restart, but end up just getting stuck and manual intervention is required.

[Gobberwart]

Response received:

"This time a plain mysql restart did work. Sometimes, the stop part of the restart process wont be able to stop and end some mysql processes on the server. Thats the issue. Thats why you maybe have to kill processes manually. Some of these mysql processes get jammed and stuck and will have to be killed manually. Its being said that this happens due to improper mysql queries, but as far as I know, this happens once a while in every server.

kill -9 $(prgep mysql)

Now this would kill off all mysql processes thats there. A mysql restart after this would just make it running. I understand doing this at times is really disturbing. If you want I can set up a script that just does kill all mysql processes and restart it whenever its down. The time interval to check if its down is only constraint that has to be decided in this case."

Good response and, while I could probably write such a script myself in about ten minutes, I'm happy to let HostV Harry do it instead.