AFF slow today ? - Page 2

**russellw** · 26-05-2014, 06:15 AM

Disk repair was completed at about 04:00. There is still an issue with mySQLd processes which appear to be what is consuming the CPU resources despite no changes having been made!

Cheers
Russ

**russellw** · 26-05-2014, 06:15 AM

Disk repair was completed at about 04:00. There is still an issue with mySQLd processes which appear to be what is consuming the CPU resources despite no changes having been made!

Cheers
Russ

**cs123** · 26-05-2014, 08:33 AM

Quote:

Originally Posted by russellw

There is a rogue apache process running so the only option will be a server restart which I am about to do.

Current Time: Sunday, 25-May-2014 21:13:03 EST
CPU Usage: u31649.9 s141.1 cu74.97 cs0 - 468% CPU load
39.8 requests/sec - 0.6 MB/second - 15.8 kB/request
300 requests currently being processed, 0 idle workers

Cheers
Russ

0 idle workers might explain it. You can try jacking the number of worker processes up but it's a slippery slope if the machine is heavily loaded. No sure how many CPUs the box has but you want to keep the utilisation manageable.

With this type of web application you can pretty much assume that it is database service times that are causing the slow web response times and that is what you have to optimise. Not sure what optimisations you have already put into vbulletin but if the avatars or attachments are stored in the DB move them onto the file system. After that you need to look at SQL optimisation and possibly memcache. Then look at upgrading to the latest version of mysql as they normally include performance optimisations in the newer releases.

After you have looked at all that then have a look at process priorities for mysql. You can jack the process priority of the database processes to give them priority over other processes. This helps performance when running mixed workload (web server and database server) and the run queues are long.

There are other things you can do like setting noatime on the volumes for the database and webserver files. This just stops the OS from updating the last access time on the files. It is redundant for a server like this and slows things down. You need to modify the /etc/fstab file with the noatime option and remount.

As a first step monitor the worker processes and if you often have 0 free then increase it a bit at a time and monitor performance. Also grab a snapshot of the machine using vmstat 5 5 when performance is slow.

XD 351 Ute · 26-05-2014, 09:50 AM

At first I thought it might be that Russ is still in 1975, going by his location, but then I see all the techo stuff you blokes are talking about. Either way, it's above my head, I'm only a fitter and turner so it is all a bit much for me.
Regardless, thanks to Russ and all that keep this place going, it helps me kill many hours!

Ed

**russellw** · 26-05-2014, 06:26 PM

Quote:

Originally Posted by cs123

0 idle workers might explain it. You can try jacking the number of worker processes up but it's a slippery slope if the machine is heavily loaded. No sure how many CPUs the box has but you want to keep the utilisation manageable.

With this type of web application you can pretty much assume that it is database service times that are causing the slow web response times and that is what you have to optimise. Not sure what optimisations you have already put into vbulletin but if the avatars or attachments are stored in the DB move them onto the file system. After that you need to look at SQL optimisation and possibly memcache. Then look at upgrading to the latest version of mysql as they normally include performance optimisations in the newer releases.

After you have looked at all that then have a look at process priorities for mysql. You can jack the process priority of the database processes to give them priority over other processes. This helps performance when running mixed workload (web server and database server) and the run queues are long.

There are other things you can do like setting noatime on the volumes for the database and webserver files. This just stops the OS from updating the last access time on the files. It is redundant for a server like this and slows things down. You need to modify the /etc/fstab file with the noatime option and remount.

As a first step monitor the worker processes and if you often have 0 free then increase it a bit at a time and monitor performance. Also grab a snapshot of the machine using vmstat 5 5 when performance is slow.

We have 16 cores in total. Avatars and attachments are already stored in the file system and we run APC as an opcode cache although I will check that it's running.

It's been running fine under this (and more) load for some time but I suspect that the last update to mysql replaced the tuned config file with a vanilla one. I am working on that now.

Apache is running out of workers as a result of mysql wait states and while there are no slow queries it is just the volume of queries that are creating short queues and keeping workers idle.

I've got the original conf files so they can go back and we can see what happens then.

There are 16 cores in the box and CPU utilisation has only been a problem this last 36 hours or so which suggests that something has changed and I have to nail down what.

Cheers
Russ

my_gxl · 26-05-2014, 11:30 PM

Its been telling me page can't be displayed

**supershifty** · 27-05-2014, 06:01 AM

As of 6am QLD time, the site is screaming fast once again. Been a tad slow on page loads lately.

Thanks for the work Russell, it's much appreciated.

Cheers!

Road Games · 27-05-2014, 06:23 AM

Hmmmmm approx 10 sec here for page to load as opposed to timing out completely yesterday

Cheers!

Might empty my cache and everything and start again

thanks Russ

Road Games · 27-05-2014, 06:24 AM

Still double post's :/

**russellw** · 27-05-2014, 06:29 AM

It is better this morning but I'm not overly convinced that we have it resolved as server loads are still much higher than they should be.

At the moment we are still debating with the ISP whether it is apache killing SQL (as they believe) or SQL killing apache (which I believe) but in the interim we are taking other preventative measures just in case.

Cheers
Russ

tweeked · 27-05-2014, 07:57 AM

Dont know if it is related but another vbulletin forum http://www.skyscrapercity.com/forumdisplay.php?f=65 has also been really slow or timing out for the last few days.

Road Games · 27-05-2014, 08:39 AM

Spot on Jon i noticed the time posted and the time actual there was 10mins difference recently ?

Road Games · 27-05-2014, 08:39 AM

Spot on Jon i noticed the time posted and the time actual there was 10mins difference recently ?

All times are GMT +10. The time now is 08:38 AM.

**MercurySilver** · 27-05-2014, 10:03 AM

yeah still a bit slow here

**supershifty** · 27-05-2014, 12:29 PM

I maybe jumped the gun on my previous post. 6am today was all good, just logged on again at 12pm and has slowed again. Nowhere near as slow as yesterday though.

Not complaining - I can still get my AFF fix for the day. It's better than no AFF.

Cheers!

needaXYGT · 27-05-2014, 01:08 PM

The site seems a lot better than it has been!

Trump · 27-05-2014, 01:22 PM

Seems to be back to race pace.

Good work Russ.

Vincenzo · 27-05-2014, 01:24 PM

In the last two to three days I have received one (1) email notification of a thread change. Do I need to resubscribe to the threads of interest and if so how (e.g., physically posting in each again?)

Cheers Vincenzo

**GasoLane** · 27-05-2014, 01:28 PM

Good stuff Russ, firing on all 8 now it seems

Vincenzo I think that might have been part of Russ's headache (and sleepless nights) all seems to be fine now though.

CRAIGAU3220 · 27-05-2014, 02:15 PM

Been hard to get on AFF for the last two/three days...............Now AFF seems to be running twin turbo

Vincenzo · 27-05-2014, 02:26 PM

Russell, on behalf of all those addicts addicted to AFF I would just like to say thank you for all your hard work of the last few days. I have had 20 email notifications in the last hour. Hope the head stops thumping soon.

Cheers Vincenzo

FPVS3 · 27-05-2014, 03:27 PM

FPVS3 - it's been answered many times before but one of the methods we use to keep spammers out is to IP block the ranges they use which funnily enough are mostly overseas. A simple email to admin@ will get the block removed or the range narrowed for you.

Cheers
Russ

Thanks Russ.

**supershifty** · 27-05-2014, 04:29 PM

Site is rocking hard now for me - seems faster than ever.

Thanks Russell.

burnz · 27-05-2014, 04:54 PM

russ does AFF run IP6 or IP4 or both?

**russellw** · 27-05-2014, 06:10 PM

Quote:

Originally Posted by burnz

russ does AFF run IP6 or IP4 or both?

IPv4 currently.

The issue (finally resolved middle of today) ended up being a script run by some hacker in Redmond DC that was basically acting like a googlebot and reading threads but at a much faster rate than they do. Discovered this (thanks Andy) after extracting a couple of thousand access log entries (about 5 minutes worth) where almost 60% of the entries came from that particular IP range - far more than any user could ever manage.

Wasn't caught by the DDoS software as it only looks at connections to the server such as you might find in an IP based DDoS attack but this effectively only used a single apache worker and then relied on the fact that it could then run up to 2,000 threads with that one worker. Multiply that by multiple workers and you have a recipe for server slowdown. Amazingly it didn't actually crash it (the server is well over the required specification) but we've closed that little loophole now.

Current Time: Tuesday, 27-May-2014 18:00:35 EST
Restart Time: Tuesday, 27-May-2014 08:19:48 EST
Parent Server Generation: 1
Server uptime: 9 hours 40 minutes 47 seconds
Total accesses: 1518568 - Total Traffic: 20.5 GB
CPU Usage: u33568.1 s181.62 cu91.05 cs0 - 11.1% CPU load
43.6 requests/sec - 0.6 MB/second - 14.1 kB/request
28 requests currently being processed, 30 idle workers

That's what it should look like.

Thanks to everyone for their patience and understanding.

Cheers
Russ

Kieron · 27-05-2014, 06:32 PM

Nice detecive work Russ.

By it's very meaning, DDOS attacks are Distributed Denial Of Service so will appear to come form multiple IP addresses. DOS attacks are simple, coming from one IP. Trouble is, these IP(s) could be spoofed so it looks like it's coming from an Australian IP, therefore blocking IP ranges becomes ineffective.

Does your DDOS software support testing the client for javascript Russ? thats a very handy way to block attacks as they typically can't run javascript with the attacking software (browsers can).

**MercurySilver** · 27-05-2014, 07:02 PM

great work russ and anyone else that helps

zipping along now

Geez Louise · 27-05-2014, 09:40 PM

Knew you would work it out Russ.

Geez! It is running like a finely tuned V8 Falcon now

Well done and I hope you get some rest!

Cheers
Col

AGH · 28-05-2014, 12:13 AM

AFF back to normal now......thx Russ and all those who work tirelessly in the background to keep this site up and running.

**russellw** · 29-05-2014, 05:56 PM

Just for the benefit of the curious, the apache status below is about what we usually see when everything is stable - not the CPU utilisation at 0.844 compared to the 300-500% we were seeing.

Current Time: Thursday, 29-May-2014 17:44:38 EST
Restart Time: Thursday, 29-May-2014 05:24:52 EST
Total accesses: 1531724 - Total Traffic: 17.7 GB
CPU Usage: u367.96 s5.68 cu1.17 cs0 - 0.844% CPU load
34.5 requests/sec - 417.2 kB/second - 12.1 kB/reques
t34 requests currently being processed, 12 idle workers