|
|
Site Support If something isn't working or you have a suggestion ( a nice one !! ) let us know here. |
|
Thread Tools | Display Modes |
26-05-2014, 06:15 AM | #31 | ||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
Disk repair was completed at about 04:00. There is still an issue with mySQLd processes which appear to be what is consuming the CPU resources despite no changes having been made!
Cheers Russ
__________________
Observatio Facta Rotae
|
||
26-05-2014, 06:15 AM | #32 | ||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
Disk repair was completed at about 04:00. There is still an issue with mySQLd processes which appear to be what is consuming the CPU resources despite no changes having been made!
Cheers Russ
__________________
Observatio Facta Rotae
|
||
26-05-2014, 08:33 AM | #33 | |||
Donating Member
Join Date: Jan 2005
Location: Morayfield
Posts: 28,091
|
Quote:
With this type of web application you can pretty much assume that it is database service times that are causing the slow web response times and that is what you have to optimise. Not sure what optimisations you have already put into vbulletin but if the avatars or attachments are stored in the DB move them onto the file system. After that you need to look at SQL optimisation and possibly memcache. Then look at upgrading to the latest version of mysql as they normally include performance optimisations in the newer releases. After you have looked at all that then have a look at process priorities for mysql. You can jack the process priority of the database processes to give them priority over other processes. This helps performance when running mixed workload (web server and database server) and the run queues are long. There are other things you can do like setting noatime on the volumes for the database and webserver files. This just stops the OS from updating the last access time on the files. It is redundant for a server like this and slows things down. You need to modify the /etc/fstab file with the noatime option and remount. As a first step monitor the worker processes and if you often have 0 free then increase it a bit at a time and monitor performance. Also grab a snapshot of the machine using vmstat 5 5 when performance is slow.
__________________
I love Holdens.... |
|||
26-05-2014, 09:50 AM | #34 | ||
Excessive Fuel Ingestion
Join Date: Dec 2004
Location: Central Queensland Coast
Posts: 1,586
|
At first I thought it might be that Russ is still in 1975, going by his location, but then I see all the techo stuff you blokes are talking about. Either way, it's above my head, I'm only a fitter and turner so it is all a bit much for me.
Regardless, thanks to Russ and all that keep this place going, it helps me kill many hours! Ed
__________________
Recommended Forum Traders: RSGerry, trimmaster, 51OAU, EB-92, adxr8, my67xr, RG, ZA-289, kruptor, gassa, Felony, RNXR, Rhino 351, Anchor, Smoke Pursuit, Mr. FPV (through E-Bay), |
||
26-05-2014, 06:26 PM | #35 | |||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
Quote:
It's been running fine under this (and more) load for some time but I suspect that the last update to mysql replaced the tuned config file with a vanilla one. I am working on that now. Apache is running out of workers as a result of mysql wait states and while there are no slow queries it is just the volume of queries that are creating short queues and keeping workers idle. I've got the original conf files so they can go back and we can see what happens then. There are 16 cores in the box and CPU utilisation has only been a problem this last 36 hours or so which suggests that something has changed and I have to nail down what. Cheers Russ
__________________
Observatio Facta Rotae
Last edited by russellw; 26-05-2014 at 06:32 PM. |
|||
26-05-2014, 11:30 PM | #36 | ||
Unintended Perfectionist
Join Date: Feb 2010
Location: Brissy North
Posts: 2,196
|
Its been telling me page can't be displayed
__________________
BA-FG parts for sale. http://fordforums.com.au/showthread.php?t=11411117 http://s1092.photobucket.com/user/my...?sort=3&page=1 The XR re-erection in the works http://www.fordforums.com.au/showthread.php?t=11386452 |
||
27-05-2014, 06:01 AM | #37 | ||
Missing a sock...
Join Date: Mar 2012
Location: Brisbane 4017
Posts: 8,250
|
As of 6am QLD time, the site is screaming fast once again. Been a tad slow on page loads lately.
Thanks for the work Russell, it's much appreciated. Cheers!
__________________
Real friends + great times = sheer bliss! Considering becoming an organ donor? Click here QLD Events, Cruises and Get Togethers: Click here Gain success instantly - lower your standards. It's not government funded - it's taxpayer funded.
|
||
27-05-2014, 06:23 AM | #38 | ||
Guest
Join Date: Feb 2007
Location: Gods Country
Posts: 16,258
|
Hmmmmm approx 10 sec here for page to load as opposed to timing out completely yesterday
Cheers! Might empty my cache and everything and start again thanks Russ |
||
27-05-2014, 06:24 AM | #39 | ||
Guest
Join Date: Feb 2007
Location: Gods Country
Posts: 16,258
|
Still double post's :/
|
||
27-05-2014, 06:29 AM | #40 | ||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
It is better this morning but I'm not overly convinced that we have it resolved as server loads are still much higher than they should be.
At the moment we are still debating with the ISP whether it is apache killing SQL (as they believe) or SQL killing apache (which I believe) but in the interim we are taking other preventative measures just in case. Cheers Russ
__________________
Observatio Facta Rotae
Last edited by russellw; 27-05-2014 at 06:36 AM. |
||
3 users like this post: |
27-05-2014, 07:57 AM | #41 | ||
N/A all the way
Join Date: Apr 2008
Location: Melbourne
Posts: 3,459
|
Dont know if it is related but another vbulletin forum http://www.skyscrapercity.com/forumdisplay.php?f=65 has also been really slow or timing out for the last few days.
__________________
BA GT 5.88 litres of Modular Boss Powered Muscle 300++ RWKW N/A on 98 octane on any dyno, happy or sad, on any day, with any operator you choose - 12.39@115.5 full weight |
||
27-05-2014, 08:39 AM | #42 | ||
Guest
Join Date: Feb 2007
Location: Gods Country
Posts: 16,258
|
Spot on Jon i noticed the time posted and the time actual there was 10mins difference recently ?
|
||
27-05-2014, 08:39 AM | #43 | ||
Guest
Join Date: Feb 2007
Location: Gods Country
Posts: 16,258
|
Spot on Jon i noticed the time posted and the time actual there was 10mins difference recently ?
All times are GMT +10. The time now is 08:38 AM. |
||
27-05-2014, 10:03 AM | #44 | ||
Isn't it obvious?
Join Date: Nov 2009
Location: in a world of idiots
Posts: 5,383
|
yeah still a bit slow here
__________________
08 Strike G6E T. 10 Ergo G6E Sept 75 XB Falcon in mushroom beige, 3 on the tree 200cid for sale, offers in the teens |
||
27-05-2014, 12:29 PM | #45 | ||
Missing a sock...
Join Date: Mar 2012
Location: Brisbane 4017
Posts: 8,250
|
I maybe jumped the gun on my previous post. 6am today was all good, just logged on again at 12pm and has slowed again. Nowhere near as slow as yesterday though.
Not complaining - I can still get my AFF fix for the day. It's better than no AFF. Cheers!
__________________
Real friends + great times = sheer bliss! Considering becoming an organ donor? Click here QLD Events, Cruises and Get Togethers: Click here Gain success instantly - lower your standards. It's not government funded - it's taxpayer funded.
|
||
27-05-2014, 01:08 PM | #46 | ||
AU II XR8
Join Date: Mar 2006
Location: melbourne
Posts: 978
|
The site seems a lot better than it has been!
__________________
AU XR8 II ute XD FAIRMONT GHIA 302 XA FAIRMONT 302 TOPLOADER |
||
27-05-2014, 01:22 PM | #47 | ||
bitch lasagne
Join Date: Aug 2012
Location: Sonova Beach
Posts: 15,110
|
Seems to be back to race pace.
Good work Russ.
__________________
|
||
This user likes this post: |
27-05-2014, 01:24 PM | #48 | ||
FF.Com.Au Hardcore
Join Date: Jul 2011
Location: Topend of Oz since 1980
Posts: 5,299
|
In the last two to three days I have received one (1) email notification of a thread change. Do I need to resubscribe to the threads of interest and if so how (e.g., physically posting in each again?)
Cheers Vincenzo
__________________
$DO || ! $DO : try try: command not found - (YODA the Geek) "WRC is for boys, Group B was for men" - Juha Kankkunen 4 times World Rally Champion Romanes Eunt Domus - 'people called Romanes they go the house' - You mean Romani Ite Domum [SIGPIC][/SIGPIC] |
||
27-05-2014, 01:28 PM | #49 | ||
Former BTIKD
Join Date: Mar 2005
Location: Sunny Downtown Wagga Wagga. NSW.
Posts: 53,197
|
Good stuff Russ, firing on all 8 now it seems
Vincenzo I think that might have been part of Russ's headache (and sleepless nights) all seems to be fine now though.
__________________
Dying at your job is natures way of saying that you're in the wrong line of work.
|
||
This user likes this post: |
27-05-2014, 02:15 PM | #50 | ||
Cat be Still !!!
Join Date: Oct 2007
Location: Perth. WA
Posts: 1,672
|
Been hard to get on AFF for the last two/three days...............Now AFF seems to be running twin turbo
__________________
2002 AUIII XR8 n/a: 13.65@101.8mph. With Raptor Assist:..12.73@111.5mph. With ET Streets:......12.44@110.7mph, a one time fluke.........12.54@110mph back to back. With 3.9 Gears:.......12.40@112.5mph, still trying! Tune by XFT |
||
27-05-2014, 02:26 PM | #51 | ||
FF.Com.Au Hardcore
Join Date: Jul 2011
Location: Topend of Oz since 1980
Posts: 5,299
|
Russell, on behalf of all those addicts addicted to AFF I would just like to say thank you for all your hard work of the last few days. I have had 20 email notifications in the last hour. Hope the head stops thumping soon.
Cheers Vincenzo
__________________
$DO || ! $DO : try try: command not found - (YODA the Geek) "WRC is for boys, Group B was for men" - Juha Kankkunen 4 times World Rally Champion Romanes Eunt Domus - 'people called Romanes they go the house' - You mean Romani Ite Domum [SIGPIC][/SIGPIC] |
||
27-05-2014, 03:27 PM | #52 | ||
FF.Com.Au Hardcore
Join Date: Jul 2010
Location: Queenstown
Posts: 626
|
FPVS3 - it's been answered many times before but one of the methods we use to keep spammers out is to IP block the ranges they use which funnily enough are mostly overseas. A simple email to admin@ will get the block removed or the range narrowed for you.
Cheers Russ Thanks Russ.
__________________
2011 FG GTP 335 S/C. Auto, tiptronic. iPV ceramic headers, stainless hi flow cats and big bore, BMC filter, X3 tune, DBA T3 rotors, Roadhouse Performance pads. From Herrod; 100lb injectors, hi flow pump, Aeroflow fuel regulator, air to water intercooler, hi boost pulley kit, JLT breather. Hub dynoed at 598fwkw, 498rwkw, 959Nm torque, boost 10.62psi at 3750rpm (warm day). All done by iPV! Other rides; '21 Jaguar F-Pace, '21 Ford Ranger FX4-MAX. |
||
27-05-2014, 04:29 PM | #53 | ||
Missing a sock...
Join Date: Mar 2012
Location: Brisbane 4017
Posts: 8,250
|
Site is rocking hard now for me - seems faster than ever.
Thanks Russell.
__________________
Real friends + great times = sheer bliss! Considering becoming an organ donor? Click here QLD Events, Cruises and Get Togethers: Click here Gain success instantly - lower your standards. It's not government funded - it's taxpayer funded.
|
||
27-05-2014, 04:54 PM | #54 | ||
VFII SS UTE
Join Date: Apr 2007
Location: Central Coast
Posts: 6,353
|
russ does AFF run IP6 or IP4 or both?
__________________
I don't often hear the sound of a screaming LSX. But when I do, So do the neighbours.. GO SOUTHS
|
||
27-05-2014, 06:10 PM | #55 | ||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
IPv4 currently.
The issue (finally resolved middle of today) ended up being a script run by some hacker in Redmond DC that was basically acting like a googlebot and reading threads but at a much faster rate than they do. Discovered this (thanks Andy) after extracting a couple of thousand access log entries (about 5 minutes worth) where almost 60% of the entries came from that particular IP range - far more than any user could ever manage. Wasn't caught by the DDoS software as it only looks at connections to the server such as you might find in an IP based DDoS attack but this effectively only used a single apache worker and then relied on the fact that it could then run up to 2,000 threads with that one worker. Multiply that by multiple workers and you have a recipe for server slowdown. Amazingly it didn't actually crash it (the server is well over the required specification) but we've closed that little loophole now. Current Time: Tuesday, 27-May-2014 18:00:35 EST Restart Time: Tuesday, 27-May-2014 08:19:48 EST Parent Server Generation: 1 Server uptime: 9 hours 40 minutes 47 seconds Total accesses: 1518568 - Total Traffic: 20.5 GB CPU Usage: u33568.1 s181.62 cu91.05 cs0 - 11.1% CPU load 43.6 requests/sec - 0.6 MB/second - 14.1 kB/request 28 requests currently being processed, 30 idle workers That's what it should look like. Thanks to everyone for their patience and understanding. Cheers Russ
__________________
Observatio Facta Rotae
Last edited by russellw; 28-05-2014 at 06:07 AM. |
||
27-05-2014, 06:32 PM | #56 | ||
FF.Com.Au Hardcore
Join Date: Jan 2005
Location: Perth WA
Posts: 1,204
|
Nice detecive work Russ.
By it's very meaning, DDOS attacks are Distributed Denial Of Service so will appear to come form multiple IP addresses. DOS attacks are simple, coming from one IP. Trouble is, these IP(s) could be spoofed so it looks like it's coming from an Australian IP, therefore blocking IP ranges becomes ineffective. Does your DDOS software support testing the client for javascript Russ? thats a very handy way to block attacks as they typically can't run javascript with the attacking software (browsers can). |
||
This user likes this post: |
27-05-2014, 07:02 PM | #57 | ||
Isn't it obvious?
Join Date: Nov 2009
Location: in a world of idiots
Posts: 5,383
|
great work russ and anyone else that helps
zipping along now
__________________
08 Strike G6E T. 10 Ergo G6E Sept 75 XB Falcon in mushroom beige, 3 on the tree 200cid for sale, offers in the teens |
||
27-05-2014, 09:40 PM | #58 | ||
Awesome
Join Date: Oct 2007
Location: In my own little world..Everyone here knows me :)
Posts: 9,401
|
Knew you would work it out Russ.
Geez! It is running like a finely tuned V8 Falcon now Well done and I hope you get some rest! Cheers Col
__________________
|
||
2 users like this post: |
28-05-2014, 12:13 AM | #59 | ||
FF.Com.Au Hardcore
Join Date: Feb 2012
Location: Wandering Oz atm.
Posts: 1,977
|
AFF back to normal now......thx Russ and all those who work tirelessly in the background to keep this site up and running.
|
||
29-05-2014, 05:56 PM | #60 | ||
Chairman & Administrator
Join Date: Dec 2004
Location: 1975
Posts: 107,236
|
Just for the benefit of the curious, the apache status below is about what we usually see when everything is stable - not the CPU utilisation at 0.844 compared to the 300-500% we were seeing.
Current Time: Thursday, 29-May-2014 17:44:38 EST Restart Time: Thursday, 29-May-2014 05:24:52 EST Total accesses: 1531724 - Total Traffic: 17.7 GB CPU Usage: u367.96 s5.68 cu1.17 cs0 - 0.844% CPU load 34.5 requests/sec - 417.2 kB/second - 12.1 kB/reques t34 requests currently being processed, 12 idle workers
__________________
Observatio Facta Rotae
|
||
This user likes this post: |