system loads and why

[incisor]

this is what i am battling...

have a look at the absolute unique visitors stat ... freakin me out... stats for the last month...

[p38arover]

Can you make a bigger pic? Even after clicking on the thumbnail, the enlarged one is hard to read.

Ron

[fraser130]

I'm far from an expert, but could it be "bots" hitting the site?

F

[waynep]

just let us know what we can do to help ..

[abaddonxi]

I'm far from an expert, but could it be "bots" hitting the site?

F


If you have a look here -
Australian Land Rover Owners - Who's Online (http://www.aulro.com/afvb/online.php)
- you can see who is online including bots.

Gives you an idea of the ratio of users, guests, bots.

Cheers
Simon

[Redback]

this is what i am battling...

have a look at the absolute unique visitors stat ... freakin me out... stats for the last month...


I guess Cooma will be a welcome distraction then;)

[incisor]

the second two stats dont include any search engine related hits, they are supposedly weeded out ...

the first includes all in the bandwidth figures going in and out....

[101RRS]

OK - and in plain language this means ?????

I assume something like the site is now so popular that it is getting overloaded.

Do you want us to stop recoomending it too people? Maybe we should stop Google having access.

[disco2hse]

Just having a quick sqizz at the Who's online page and there were at least two Google spiders and a large number of guests that could also be bots.

It seems to me that 11 seconds per site visit seems a little short (understatement intended) considering how long it takes for me to decipher some peoples' posts - not mentioning any names :D So leads me to suspect that you are getting pinged by Google et al fairly frequently, probably because of Joe Public's vague search criteria.

The unique visitors thing isn't that much of an issue. I would think that bots will be uniquely named so they don't cause integrity conflicts with home base. A larger issue is the volume of data number of requests you are putting out at peak times and whether your server can handle them. Obviously the data rate will be limited by your server's bus speed and the size of your pipe the number of requests has other limitations (cpu speed, RAM volume, disk R/W speeds, etc.). So if your current h/w is having problems with matching peak loads can you load balance with another server?

[disco2hse]

Wish I could edit my post to make sound sensible: hint, hint, hint

Anyway, last paragraph should read:

The unique visitors thing isn't that much of an issue. Bots are uniquely named/identified so they don't cause integrity conflicts with home base. A larger issue is the volume of data number and requests you are putting out at peak times and whether your server can handle them. Obviously the data rate will be limited by your server's bus speed and the size of your pipe but whether it can handle the number of requests is another issue and introduces other limitations (cpu speed, RAM volume, disk R/W speeds, etc.). So if your current h/w is having problems with matching peak loads can you load balance with another server or can you proxy large data files such as library images?

[RonMcGr]

Dave,

Is there a way to put a password in, even to open the front page?
i.e. NO visitors?

Ron

[Phoenix]

There is, and in the past it was like that, however I think that the bandwidth it used was minimal to have it open for the LR community to see.

[incisor]

Wish I could edit my post to make sound sensible: hint, hint, hint

Anyway, last paragraph should read:

The unique visitors thing isn't that much of an issue. Bots are uniquely named/identified so they don't cause integrity conflicts with home base. A larger issue is the volume of data number and requests you are putting out at peak times and whether your server can handle them. Obviously the data rate will be limited by your server's bus speed and the size of your pipe but whether it can handle the number of requests is another issue and introduces other limitations (cpu speed, RAM volume, disk R/W speeds, etc.). So if your current h/w is having problems with matching peak loads can you load balance with another server or can you proxy large data files such as library images?

the load balancing is the next step... but means another server and more rack space etc etc etc

the volume of data is a worry as well....

after cooma i will sit and think my way thru it, bit jaded at present....

[beforethevision]

If you need cash Inc, im sure people will be willing to support the site they frequent.

500 kbps average is some pretty serious traffic.

Cheers!

[disco2hse]

the load balancing is the next step... but means another server and more rack space etc etc etc

the volume of data is a worry as well....

after cooma i will sit and think my way thru it, bit jaded at present....

Fair enough. I am sure we are all content to work with things as they are in the meantime. :)

[dmdigital]

Inc, have you considered the following:

Limiting inline image sizes to a max of 800x800 and 200kB
Only displaying images and attachments if the user is registered & logged in

[Michael2]

Does this mean that you can sell advertising space at a premium?

I hope so.

[abaddonxi]

It's all the polls I tells ya!

They're sucking the life out of the server.

Ban em all.

:D

Cheers
Simon

[Ace]

i know lots of people like the gallery but with sites like photobucket is it needed? Wouldnt that save a heap of space and limit the bandwidth?

In a way its a good thing inc, all these people visiting means you are doing a top job and 40000 people a month think the same thing. Matt

[jik22]

the load balancing is the next step... but means another server and more rack space etc etc etc

the volume of data is a worry as well....

after cooma i will sit and think my way thru it, bit jaded at present....

A couple of shooting forums I'm on require logins - it's a minor PITA, but compared to no longer having them due to rising bandwidth needs and running costs, it's a small price to pay.

[Rob101]

<jibberish>
My advice is, IMHO. HEAPS of stuff, in no particular order.

depending where/what your bottlenecks are. I'm assuming you want to bring the volume of traffic down.

You seem to be running apache on Linux (or at least your hosting provider, voice valley is, depending on your set-up).
Install mod_gzip or mod_deflate, that will bring your text serving size down HEAPS for surprisingly little processing overhead. If you find you're losing too many CPU cycles, i'd suggest tuning the level of compression so that the response time is good but you also get reasonable compression.
grep your apache logs into a mysql table. Do a 'group by' query and accumulate the number of hits per URI multiplied by the byte size of the responses.
You'll find that the data is skewed to a few objects, which will probably be binary photographs. unlink the photographs.

depending on the distribution of bytes / response objects, a reverse proxy might also help serve common, static objects.

Check that your conditional requests are actually working. If clients are sending IMS requests and your server is sending back objects when it should be a positive cache acknowledgment.

If processing is your bottleneck, depending on the level of access you have to the machine i'd probably install zend optimizer, it is apparently compatible with suhosin.

If the DB is your bottleneck, you might need to look towards replication - which is probably too expensive.

The main page http://www.aulro.com/ is FAR too large, 119.7k. Pull the CSS out into a separate file, each time someone loads the main page, they keep downloading the static CSS, it should be seperate and locally cached. Most/all of the javascript should also be placed in seperate objects.
I'd pull all the whitespace from it too. It's just wasted.

100 characters of space are great, but if one hundred people load the page, thats 10kb.

think about removing dynamic content, like the time, from pages that are not otherwise changed. This will allow the content to be cached.

...

are you serving any content other than this forum?
</jibberish>

[Rob101]

Not linux, sorry. FreeBSD?

[incisor]

<jibberish>
My advice is, IMHO. HEAPS of stuff, in no particular order.

depending where/what your bottlenecks are. I'm assuming you want to bring the volume of traffic down.

You seem to be running apache on Linux (or at least your hosting provider, voice valley is, depending on your set-up).

welcome to the honey pot, all is not as it seems :P



Install mod_gzip or mod_deflate, that will bring your text serving size down HEAPS for surprisingly little processing overhead. If you find you're losing too many CPU cycles, i'd suggest tuning the level of compression so that the response time is good but you also get reasonable compression.

long since done


grep your apache logs into a mysql table. Do a 'group by' query and accumulate the number of hits per URI multiplied by the byte size of the responses.
You'll find that the data is skewed to a few objects, which will probably be binary photographs. unlink the photographs.


depending on the distribution of bytes / response objects, a reverse proxy might also help serve common, static objects.

Check that your conditional requests are actually working. If clients are sending IMS requests and your server is sending back objects when it should be a positive cache acknowledgment.

If processing is your bottleneck, depending on the level of access you have to the machine i'd probably install zend optimizer, it is apparently compatible with suhosin.

zend and apc. found xcache problematic



If the DB is your bottleneck, you might need to look towards replication - which is probably too expensive.

agreed


The main page http://www.aulro.com/ is FAR too large, 119.7k. Pull the CSS out into a separate file, each time someone loads the main page, they keep downloading the static CSS, it should be seperate and locally cached. Most/all of the javascript should also be placed in seperate objects.
I'd pull all the whitespace from it too. It's just wasted.

will look into that,..


think about removing dynamic content, like the time, from pages that are not otherwise changed. This will allow the content to be cached.


hmmmmmmmm yes and no... will be looking into some of the dynamics of the way it is setup.
...


are you serving any content other than this forum?
</jibberish>

dedicated p4 box with 4gig of ram running freebsd with raided sata hdd's

it is running an openads adserver for the system as well but that will be leaving shortly.

and postfix for mail

[Rob101]

Oh yes.
and I should say.
:D Spectacular forum. :D

[Scouse]

I understood nothing of the above except this :
Oh yes.
and I should say.
:D Spectacular forum. :D
Understatement of the year IMO
:):):)

[disco2hse]

Thinking of ways you can reduce the number of disk accesses.

You using RAID to increase volume or for data replication/disk duplication? If the former, this can add significantly to your read/write times. With the cost of large disks being so low now it may be wise to consider copying everything onto a larger disk, e.g. 500Gb <$200, 1Tb still expensive but will come down soon.

You might need to look at how big you have set the partitions (?). Can you move the swap disk to another drive (that'd be at least 4Gb on its own)?

What about storing BLOBS on another drive?

[incisor]

is a mirror for backup

i will be upgrading the drives to some of the new 32mb cache gear this year some time.

[Rob101]

will look into that,..


Sorry, I think that I got the size of the page wrong by looking at the xfer rate in wget, rather than the object size. But the point still holds water.
um, yeah. and now to stfu.

[disco2hse]

is a mirror for backup

Right, that shouldn't make too much difference, but with a large number of requests it could end up as a bottle neck as the disks sync with each other.


i will be upgrading the drives to some of the new 32mb cache gear this year some time.

You will probably notice a difference for a while. What are your storage requirements and will they increase by much?

The requirement to store images outside the forum may be a necessary step although having the images as a library resource may have historical value in years to come.

[disco2hse]

and now to stfu.

Not at all mate. Some good ideas in there :)