Server Down - Network Issues

Stuff we think you need to know.

Moderator: Public Forum Moderators

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Server Down - Network Issues

Unread post by Gobberwart » Thu Apr 16, 2009 4:58 am

As of about 30 minutes ago, the server on which Paradox is hosted became inaccessible. I reported the problem to our hosting provider and received the following response:

Hi,

We are experiencing an unexpected network issue at our Hardware nodes , we are working with DC to resolve the issue as early as possible,We will update you through this post regarding the progress.

For further updates , please follow <link to forum post that's password protected for some reason>


Thank you
--------------------------------------------------------
Regards,
CirtexHosting - Joseph
System Administrator / Level 3 Tech Support
--------------------------------------------------------


Sorry guys, there's nothing I can do. Hopefully it'll come back fairly soon. Even more hopefully, there'll be no data corruption issues.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Thu Apr 16, 2009 5:13 am

Latest update from HostV:

Hi,

The node2 is back to online and all the VPS inside that node will be running fine now.We are checking with the other nodes and will keep you posted

We appreciate your patience and thanks for understanding


What a shame we're not on Node 2 :(
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Thu Apr 16, 2009 5:22 am

Aaand, we appear to be back online. Checking for any obvious errors.

If you have reason to believe there's any data corruption as a direct result of this, please submit a bug report with as much information as you can possibly include, and I'll do my best to fix it asap.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Thu Apr 16, 2009 5:29 am

No errors that I can see, but as per previous post, please submit a bug report if you have reason to believe there's a problem.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Tue Apr 21, 2009 1:10 am

The provider is having network issues again. The game's up but is currently very, very slow. I've logged a fault and they say it'll be fixed "as soon as possible".
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Tue Apr 21, 2009 1:12 am

I've just discovered that intermission didn't happen as planned so I'm going to have to force it to happen now. Bear with me.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Tue Apr 21, 2009 1:36 am

OK, Intermission's done and the site's back up and working at what looks like normal speed. We now return you to your regular broadcast.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Tue Apr 21, 2009 11:41 am

Apparently everything went down again just after I went to bed. Typical. All looks ok now, but the provider really worries me with comments like this:

We got update from DC that the servers are in same power strip which overloaded the power circuit .Now the main servers are powered on again

Power strip???? Surely they don't mean what it sounds like they mean. No, surely not.

Anyway, this came from HostV around 7 hours ago:

FINAL UPDATE:

The downtime was caused by an issue with our power circuit and few VPS Nodes were affected. We have currently resolved the issue and we will need to replace the power circuit fully later on this week with scheduled + minimal downtime so this does not happen again, with a brand new power circuit.

We will keep everyone updated and notified before the downtime. Sorry for the inconvenience caused again.


It was in bold, too, so it must be real. Of course, the fact that they haven't replaced the "power circuit" yet (or power strip *shudder*) means the site could go down again at any time. And they probably won't tell me WHEN the "scheduled+minimal downtime" will occur.

Sorry guys, this is the best we can afford.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Server Down - Network Issues

Unread post by Gobberwart » Wed Apr 29, 2009 12:00 am

As many of you know, the game went offline for a while yesterday. Without wanting to point fingers the outage was ENTIRELY the fault of HostV who did some "scheduled maintenance" without telling any of their customers. Ok, they posted something on a forum nobody EVER reads until it's too late, but honestly they knew they were going to do it before hand, they have all their customers' email addresses and they just totally failed to communicate with us. As usual. SIGH.

ANYWAY, the point is that I didn't know anything about it until after the site had already come back up, by which time it was too late to do anything about it. Not that I could have in this case, but I could at least have put an update on twitter or the forums for those who keep up with those things.

Right now, I'm trying to look into setting up something that will send a message to my cellphone if the site is down, and into teaching other devs about using Linux so that, if I'm not available, maybe someone else can do something about it. If anyone has any thoughts on this, please let me know.

In the meantime, my apologies for HostV's stupidity. Rest assured they have been berated severely. Again.
Image

Post Reply