Continued Server Load Problems

Stuff we think you need to know.

Moderator: Public Forum Moderators

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Continued Server Load Problems

Unread post by Gobberwart » Sun Aug 16, 2009 3:59 pm

As per my earlier in-game update, I have already logged a support ticket with our hosting company regarding the load problems that have caused lag for the past 7 hours or so. I logged the ticket as "Critical" but apparently that doesn't matter, because it's been ignored.

Now, HostV has a "Live Chat" facility, but only for Sales and Billing. Since I have no other way to get in direct contact with anyone, I thought I'd contact the sales department in the hope they'd at least escalate it to the appropriate department. Here's the chat transcript. You be the judge:

Cirtex Daniel: Hi, Thank You for choosing CirtexHosting.How may I help you today?
Stuart Travers: I'd like to know when the critical support ticket I logged four hours ago will be looked at
Stuart Travers: ticket id THI-579966
Stuart Travers: I'm yet to receive a response
Cirtex Daniel: please hold on let me check
Cirtex Daniel: is that a vps?
Cirtex Daniel: or a dedicated server?
Stuart Travers: vps
Stuart Travers: server loads have been huge for 7 hours now. I reported it 4 hours ago and nothing's been done
Stuart Travers: I'm not a happy customer at the moment
Cirtex Daniel: The issue that you have seems to be a VPS issue. You have reached the technical support department of shared support and unfortunately I am not in a position to help you. To get more information, please post a ticket to support@hostv.com. They are the right persons who have access to VPS data.
Stuart Travers: I already HAVE posted a ticket. I need something done about it now and nobody's dealing with the ticket
Stuart Travers: If you can't help me, please put me in touch with someone who can, don't tell me to go and log another ticket
Cirtex Daniel: , I can understand your situation right now. I wish I could help you with this. Unfortunately I too do not have access to their support system. If I were in your situation I will have to wait for them to respond to that ticket. I would suggest you to post another ticket with high priority to support@hostv.com, they best we can do at this point of time.
Stuart Travers: Since there is no "live chat" for VPS support, this is my only option to actually speak to someone
Stuart Travers: that's terrible. the ticket's already posted as "critical"
Stuart Travers: Are you seriously telling me you can't escalate it to someone who can get it dealt with?
Cirtex Daniel: We have no authorisation
Cirtex Daniel: to escalate it
Stuart Travers: and there's nobody you can speak to who does?
Cirtex Daniel: Stuart please undersatnd us . This is shares sepport team
Stuart Travers: I understand that you are part of the Cirtex/HostV organisation, and I would therefore expect you to be able to at least pass it on to the support team that there is an outstanding ticket that needs attention.
Stuart Travers: As a customer, it's of no concern to me how your organisation is set up. I want my issue dealt with and I have every right to expect it to be handled professionally. Saying "oh that's not my department" does not reflect well on your company.
Stuart Travers: If you care to spend a couple of minutes thinking about it, you will be able to find some way of communicating this problem to the correct part of the organisation, and I would appreciate it if you would do so.
Cirtex Daniel: ok well then i will convey it to my higher officials well and also covey your regrets
Cirtex Daniel: I apologize for any inconvenience this may have caused to you.

Cirtex Daniel has left the chat conversation
Click Here to Leave a Message

Stuart Travers: I'll pass that on to all my users who can't access the system


Yes, that's right, he "hung up" on me before I was finished typing. Great service, huh?

Oh well, I will continue to harrass the support department through any means possible. No doubt it'll be fixed at some point. Sorry about that, everyone. Hopefully some day we'll be able to afford a dedicated server (which runs around $400 a month), but until then, this is the best we can get :(
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun Aug 16, 2009 4:08 pm

UPDATE: I've just emailed the owner of the company. Hopefully he's working late tonight and will kick some ass. I've also removed the "Powered by HostV" logo from the login page, because seriously, I'm not giving them free advertising for being shite.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun Aug 16, 2009 4:20 pm

UPDATE: for those who care about these kind of things, here's our current load averages:

01:39:01 up 110 days, 19:01, 1 user, load average: 1.33, 1.66, 1.45
01:40:01 up 110 days, 19:02, 1 user, load average: 2.83, 1.97, 1.56
01:41:01 up 110 days, 19:03, 1 user, load average: 2.42, 1.99, 1.59
01:42:01 up 110 days, 19:04, 1 user, load average: 2.04, 1.98, 1.62
01:43:01 up 110 days, 19:05, 1 user, load average: 1.65, 1.88, 1.60
01:44:01 up 110 days, 19:06, 1 user, load average: 1.13, 1.68, 1.55
01:45:01 up 110 days, 19:07, 1 user, load average: 1.45, 1.63, 1.54
01:46:01 up 110 days, 19:08, 1 user, load average: 2.10, 1.84, 1.61
01:47:01 up 110 days, 19:09, 1 user, load average: 2.12, 1.88, 1.64
01:48:01 up 110 days, 19:10, 1 user, load average: 1.95, 1.85, 1.64

And here's what it normally looks like:

15:16:01 up 110 days, 8:38, 0 users, load average: 0.40, 0.36, 0.44
15:17:01 up 110 days, 8:39, 0 users, load average: 0.33, 0.35, 0.43
15:18:01 up 110 days, 8:40, 0 users, load average: 0.38, 0.36, 0.43
15:19:01 up 110 days, 8:41, 0 users, load average: 0.63, 0.42, 0.44
15:20:01 up 110 days, 8:42, 0 users, load average: 0.30, 0.36, 0.42
15:21:01 up 110 days, 8:43, 0 users, load average: 0.50, 0.43, 0.44
15:22:01 up 110 days, 8:44, 0 users, load average: 0.18, 0.35, 0.41
15:23:01 up 110 days, 8:45, 0 users, load average: 0.11, 0.30, 0.39
15:24:01 up 110 days, 8:46, 0 users, load average: 0.14, 0.27, 0.37

I've restarted all services on the Paradox VPS, and it hasn't made any difference. HostV has to fix this, and they have to do it soon or there's going to be a cranky Gobber. More cranky than normal, I mean.

Sure, there's little difference between 0.40 and 2.5 in real numbers, but when you consider that that represents the percentage of the Paradox VPS CPU being utilised (eg. 0.40 = 40% and 2.5 = 250%) it's a HUGE deal. If we were using it, I'd fix it, but the fact is that someone ELSE is using it and that pisses me off.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun Aug 16, 2009 7:34 pm

Ok, it looks like there's no more lag issues for now, but Cirtex/HostV FINALLY got around to looking at the problem around 30 minutes after it went away on its own. So that means they can't see what was causing it or do anything about it until next time it happens. Sigh. Will keep you informed.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun Aug 16, 2009 9:40 pm

Yeah, the lag is back. I'm TRYING to get them to do something.

06:46:03 up 111 days, 8 min, 1 user, load average: 2.40, 1.75, 1.09
06:47:01 up 111 days, 9 min, 1 user, load average: 2.38, 1.87, 1.16
06:48:05 up 111 days, 10 min, 1 user, load average: 1.93, 1.83, 1.20
06:49:04 up 111 days, 11 min, 1 user, load average: 1.99, 1.87, 1.25
06:50:10 up 111 days, 12 min, 1 user, load average: 2.09, 1.93, 1.31
06:51:02 up 111 days, 13 min, 1 user, load average: 1.97, 1.93, 1.34
06:52:03 up 111 days, 14 min, 1 user, load average: 2.01, 1.94, 1.38
06:53:01 up 111 days, 15 min, 1 user, load average: 1.79, 1.89, 1.39
06:54:02 up 111 days, 16 min, 1 user, load average: 2.30, 2.01, 1.46
06:55:01 up 111 days, 17 min, 1 user, load average: 2.04, 1.99, 1.49
06:56:01 up 111 days, 18 min, 1 user, load average: 1.78, 1.96, 1.51
06:57:01 up 111 days, 19 min, 1 user, load average: 1.61, 1.87, 1.50
06:58:03 up 111 days, 20 min, 1 user, load average: 2.24, 2.05, 1.59
06:59:02 up 111 days, 21 min, 1 user, load average: 2.26, 2.09, 1.63
07:00:26 up 111 days, 22 min, 1 user, load average: 2.82, 2.25, 1.72
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Mon Aug 17, 2009 11:57 am

After several more rants at HostV over the last 24 hours (the usual me:"it's lagged"... 2 hour delay.. them: "I just checked, it looks fine now"... me:"Yeah, it looks fine NOW but it hasn't been. monitor it and look harder!"... them:"oh ok"... 2 hours... me:"it's lagged again!" etc.) I finally got the following response from "Cirtex Jeff"...

"We were monitoring your server for the last few hours and found that the load increased upto 1.35. I have terminated/disabled some processes on the main node that caused this load problem. We are still monitoring your server."

I've checked the logs and for the past 8 hours or so, the load averages look pretty good. Of course, maybe what Jeff's done is change it so I can't look at load averages properly, but we'll see.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Mon Aug 17, 2009 8:33 pm

So, I put together a script that checks server load every minute and sends me an email if it's been excessive for 5 minutes or more. The idea being that I could take the email and forward it to HostV, but then, just as a bit of a dig I suggested to them that maybe I should modify the script to email them as well. Their response:

"OKay, that might really help. Thank you"

Well, who am I to argue? The script has been modified and will email them every time the 5-minute load average exceeds 1.0. They're going to love it if they get another 8-hour disaster like yesterday.

Latest update:

"We will be checking and hopefully everything can be fixed within 24 to 48 hours."

Stay tuned.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Wed Aug 19, 2009 11:49 am

Apart from one report (unconfirmed) there's been no sign of significant load/lag for around 36 hours now. Hopefully they really HAVE done something to fix it. I'm going to leave my monitoring scripts running for a while though, just in case.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Thu Aug 20, 2009 4:26 pm

OK, after a couple of half-hour instances of load, where I made sure HostV support was bombarded with alerts every five minutes (ain't automation GRAND?), I finally got this response today:

Hello Stuart,

We have found the one or two vps's causing the issue. Just have to be in those vps's at the right time. Will keep you updated.

Thank you


Now, while that's nice... It still doesn't explain WHY a VPS is able to steal our paid-for processor time and is no guarantee that it won't happen again. Sigh.

Anybody have a few thousand dollars lying around spare so we can afford dedicated hosting? I know I don't.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Thu Aug 20, 2009 4:33 pm

Here's my response, CC'd to HostV's owner, for those playing at home:

OK, thanks, but why does this happen and why do I need to set up special monitoring and harrass you for a week to get anything done? I understand that HostV allows VPS users to "burst" beyond their paid-for, allocated resources for a short time (although even that's questionable), but to allow a single VPS to monopolise the resources of the entire node for any more than a couple of minutes is... well, it's ridiculous and incredibly irritating.

What has HostV done to make sure this doesn't happen again? Or do we just wait until the NEXT inconsiderate VPS administrator comes along and does it again, then repeat this whole dance?


I'm sure most of you aren't interested, but for those that are, this is how I'm trying to protect your valuable time from stupidity.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Fri Aug 21, 2009 12:06 pm

Latest update from HostV:

Hi Stuart,

We have found one account that caused the load issues on the server. We have disabled that account. Your server load wont hike again from now. Please set the same cron that sends mail to our support queue (support@hostv.com), so that we can find if anything else causes that load problem.
--------------------------------------------------------
Regards,
CirtexHosting - Jeff
System Administrator / Level 3 Tech Support
--------------------------------------------------------


Ok, so someone got disabled. Awww. Too bad for them, great for us. This time. I'm going to continue harrassing them, though, to explain WHY one account is able to ruin it for everyone else and continue to strongly suggest that they do something to stop it happening, rather than just disabling accounts when it happens. Seriously, if I have a PHYSICAL server, I can't use more CPU than it physically has, why should a VIRTUAL server user be able to use more than they're allocated? Will keep you updated.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Fri Sep 18, 2009 12:26 am

Been wondering why the game's been lagged ridiculously the last few hours? So was I, so I logged yet another support call. Here's HostV's response:

We are taking a 100GB of data from the main node and moving a big account .This is the reason for the load spike for the last couple of hours .The process will be completed within another 6-7 hour.

There is a customer , who accidentally lost all the data .So now we are trying our level best to help him by taking backup and get his server back to online .Sorry to hear that it is causing slight load disturbance .Since the customer is badly affected , he is losing business.Seems he is accidentally deleted some important configuration files .So I am helping him to recover all the data . Once the process is completed , everything will be back to normal .


Ummm, right. So some customer is a complete moron and accidentally deleted something critical without backing it up, and now that's entirely OUR problem apparently. For another 6-7 hours.

We're due to renew our hosting in December, so if we can possibly manage to afford dedicated hosting (ie. a server all to ourselves) I'll do what I can to make that happen.

In the meantime, if you or anyone you know provides dedicated hosting and might be able to do us some kind of "mates rates" deal, please let me know.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Fri Sep 18, 2009 9:45 am

HostV has just informed me, in somewhat broken English, that they're going to move us (again) to a less-busy VPS node. I doubt they'll inform me before they do this, so... if the game suddenly goes offline for no apparent reason, that'll be why.

If they DO let me know, I'll give you a heads-up.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Wed Nov 25, 2009 1:48 pm

Just wanted to post an update to advise that this issue is STILL going on, and that we're currently experiencing irregular periods of significant load on the server, coming from elsewhere on the node, and that HostV are doing their usual spectacular attempts to fix it (ie. nothing visible at all beyond occasional spontaneous rebooting of the server).

Unfortunately, a site like Paradox has specific requirements that mean normal, cheap hosting is not an option. In this situation, I've tried to acquire the best possible hosting I can afford which is usually adequate but obviously not always. Paradox really needs a dedicated server, free from any possible interference from other systems, and that doesn't come cheap. Well, it does (if you call $120 per month cheap) but that's pretty low-end and likely to be susceptible to a whole different range of issues.

If we suddenly receive a large injection of funds or if someone can offer us a deal on a dedicated server or colocation setup, I'll jump at the chance to move, but in the meantime, please be patient and remember that we're in alpha and we don't have a ton of money to throw around.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Wed Nov 25, 2009 4:21 pm

Update: After weeks of them telling me there's nothing wrong or that I should check my code, I finally got this email a few minutes ago:

Hello Stuart,

We have identified the vps's on the main node causing the issue. We are trying to control their loads. If that's not possible we will move them or have them upgraded. We hope we can fix this real soon now. Will keep you updated.

Thank you


Stay tuned.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Wed Jan 27, 2010 11:11 am

Well, I had hoped that these load issues were resolved, but no dice apparently. It looks like there's been serious load on the server for several hours and right now, the server is in the process of rebooting, and seems to be stuck. I'm on it.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Wed Jan 27, 2010 11:16 am

Server has been rebooted but there's still excessive load, so I'm off to start the latest round of fighting with HostV. Sigh.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun Apr 25, 2010 11:45 am

We currently appear to have both periods of random excess load (resulting in a failed server reboot around 5 hours ago!) as well as someone who seems to be running a process-hogging script from around 10 minutes before intermission to around 15 minutes after. A couple of days ago, that resulted in a reboot in the middle of intermission and... yeah, that's not good.

I'm currently tearing my hair out trying to find any way I can possibly afford to move us to a dedicated server, but so far nothing.

Do we have anyone out there who has either an ability to provide us with affordable dedicated hosting (like REALLY affordable) or anyone with a stack of cash willing to help a struggling game startup? If you are in any way positioned to help us out, please contact me directly to discuss.

In the meantime, I'll see what HostV comes back to me with and continue to do everything within my power to prevent this continuing to happen.
Image


SportGuy
Forum Addict
Posts: 418
Joined: Wed Apr 16, 2008 6:55 pm
Location: Wisconsin

Re: Continued Server Load Problems

Unread post by SportGuy » Tue Apr 27, 2010 12:39 am

Just out of curiosity...is this VPS provider the only provider in town that charges prices similar to what you're paying now? I'm just wondering because even if dedicated hosting can't be afforded right now it might be at a minimum reasonable to change to a VPS that has some form of stability and customer support.


sodafett
Chat Moderator
Posts: 508
Joined: Mon Jul 20, 2009 3:10 pm

Re: Continued Server Load Problems

Unread post by sodafett » Tue Apr 27, 2010 7:27 am

have you thought about going with a colo instead of a dedicated server, and finding a hosting company?
Buy your own server, then have a hosting company host it? If so, I have a good friend here that works for a hosting company.
Image


SportGuy
Forum Addict
Posts: 418
Joined: Wed Apr 16, 2008 6:55 pm
Location: Wisconsin

Re: Continued Server Load Problems

Unread post by SportGuy » Tue Apr 27, 2010 7:36 pm

There is a thought...what do hosting companies charge just to host?

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Thu Apr 29, 2010 1:56 am

The current VPS company we're with is really no better or worse than any other and seriously I don't want to have to go through all the hassle of transferring to yet another provider (we're already on our second) just to prove that fact. The fact is that sharing physical resources with others is always going to put us at the mercy of idiots and that, to ensure any guaranteed level of continued service, a dedicated server is the only sensible option.

With regard to colo, that just opens a whole can of worms I'm not keen to look at right now. While a refurbished server can be purchased fairly cheaply (around $200), it's not new and therefore more likely to fail and have a rubbish warranty. It's also likely to be significantly older, and therefore replacement parts are almost certainly harder to come by. Unless we want to purchase refurbished parts, which are more likely to fail... and so on. A new server, on the other hand, with a decent warranty and current specs runs into the thousands and is just not affordable. And there's really no point purchasing an older server, since we won't be able to upgrade it cheaply or quickly when we need to.

Right now, I've found a reasonably priced dedicated server option which charges by the month, so I'm going to set it up for a little while and run some benchmarks just to see how things go. It's not high specced, but we don't really need a high-end server at this stage, and upgrading in the future when we DO need higher end hardware is likely to be relatively straightforward since we're only leasing the hardware. I'll let you know how the testing goes.
Image


sodafett
Chat Moderator
Posts: 508
Joined: Mon Jul 20, 2009 3:10 pm

Re: Continued Server Load Problems

Unread post by sodafett » Thu Apr 29, 2010 10:49 am

I am also still poking around...lol
Image

User avatar

Spiff
Wikinatrix
Posts: 560
Joined: Sun Dec 30, 2007 12:49 pm
Location: North Cackalacky :-P
Contact:

Re: Continued Server Load Problems

Unread post by Spiff » Sun May 09, 2010 11:17 am

okay so funny thing to tell you gobb... i was looking at this forum and i saw an ad for Intel server/web hosting... not sure on the specifics of our current server but thought youd like to see the page the ad took me to... https://interserver.net/custom-managed- ... nAod2Sz4GA
The Spiff

User avatar

commandermack
Chat Moderator
Posts: 1024
Joined: Thu Mar 19, 2009 5:40 pm

Re: Continued Server Load Problems

Unread post by commandermack » Sat May 15, 2010 10:34 am

The main server is pretty laggy right now, I have no idea if it is just me or everyone, but yeah... I'll be banking my turns till later to see if it is better.

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sun May 16, 2010 12:04 pm

Yeah, load on the node has been up and down all over the place. HostV are obviously not interested in fixing the problem, and we'll be moving somewhere else as soon as we find somewhere we can afford and where we can be sure the issues aren't going to occur again. I'm working on it.
Image

User avatar

Topic author
Gobberwart
Developer
Posts: 3396
Joined: Wed Jun 20, 2007 12:41 am
Location: Melbourne, Australia
Contact:

Re: Continued Server Load Problems

Unread post by Gobberwart » Sat May 22, 2010 12:08 am

Well, the game got migrated to a new server today and all is looking very very good. There's been no lag whatsoever for over 10 hours now, and intermission completed in record time. HostV can kiss our collective asses.
Image

Post Reply