I have 4 instances of PsychoStats running on a Linux Celery 2.4. There are no game servers on the machine (it FTP downloads the log files then processes them).

These PsychoStats instances are to update once daily in the morning (each instance updates 1 hour apart [i.e. 1am, 2am, 3am, 4am]). I setup a cronjob to accomplish this (crontab -e, etc).

A couple times per week however I see the machine load is completely going crazy (between 30.00 and 60.00) and when I issue a top command, I see that there are about 25 instances of "stats.pl" running, and that of course is what is maxing out the processor.

Does anyone know a way to narrow down just as to why this is occuring? Thanks!

You have to use a debugger

Hope this helps
Matt

How big are your log files?

No ideas for yor error, but have you ever looked into running PyschoStreamer? It would cut out the FTP step for you.

Massive in some cases. There are stats pages with over 12,000 people on them.

Wow!

I'm not suprised your stats box is going nuts then. That's a lot for it to do.

What can I say? I have popular servers. lol

Can't you trim those logs or something ? I mean every few days ?

I'm about to try to set this up myself automated if anyone has any good info besides just RTM that is

You can setup Psychostats to only keep logs for a certain number of players. Really I would recommend this step, as setting up stats for players to see when they are regular visitors to your servers is better than just logging everyone that might come in.

It also gives people the incentive to play more often there in order to keep their stats up.

Do you run a map for 9999 minutes or something? thats one of the few reasons off the top of my head I can think your log files getting sooooooooooooooooo big!

Is there enough Matrix Avatars or what? Alot of people don't rotate their log files for whatever reason.

I was just about to suggest giving logrotate a try. That'll give you smaller log files to work with while keeping all the stats.

or...... Get a bigger machine

or a better stats processor...

Is this an option in PsychoStats? I have PS setup to compile the logs once per day, each log file isn't necessarily huge, there have just been a lot of players built up over time (the ~15,000 in the last 30 days).

You'd have to run Logrotate outside of PsychoStats. Since PsychoStats records to a MySQL db I assume that it will keep old logs in the db and just import new stats as they come in?