OK. After poking around my logs, seeing how big they are, here is what I decided my apache log/report policy should be:

1) rotate logs monthly, compressing the previous month's logs and deleting the earlier compressed logs... just leaving the current months logs (being added to) and the previous month's compressed log.

2) allow my customers to download the previous month's compressed log (giving them one month to do so before it is deleted).

3) I will personally ftp all their logs to a storage server and then, once a year actually mail them a cd with all of their previous year's logs burned on it (at least for the big clients). At the same time, I will make myself a cd and clean all the logs off my storage server, ready for the new year.

Also, I run urchin and it, in the best case scenario, produces reports that are about 10-12% of the size of the raw logs each month. Because of this, I can't keep them around forever. So my policy is to:

1) not count urchin reports against my account's disk quota
2) delete any urchin reports older than a year

I am pretty sure these policies will make even my most picky customers happy. Now the only question is, how do I get this to work in the cpanel context?

Specifically:
- how do I get the /home/account/tmp folder to not count against a customer's quota?
- which utility should I use to rotate raw logs and compress the previous months logs (and delete older)? WHM? logrotate? urchin?
- how can I make the previous month's logs easily available to clients?
- how does this all work with the cpanel options to archive previous months raw logs?
etc. etc.

Any answers to any of these questions would be appreciated by myself and, I'm guessing, to many others, too.

hi, i believe that if the owner of the file matches the site owner, the site owner's quota will be counted. so in order to remove the /home/account/tmp from the account owner's quota, you will need to change the owner for this directory and all files in this directory. for examples, you will do this command as root:

#chown -R admin /home/account/tmp
#chmod 774 -R /home/account/tmp

( you can create an account name without any quota limitation and use this user as the owner of any file you don't want a quota limitation)

sorry, i haven't test it, so don't apply it to a real account until you test it.

Apparently this would need to be done everytime cpanellogd is run (nightly) since it contains the following line:

system("chown","-R","$user:$group","$homedir/tmp");

I suppose you could change it to:

system("chown","-R","admin","$homedir/tmp");

to have it do your work for you... but this would be overwritten on every cpanel update.

And it would be important to make sure that the correct permissions are set before the quota script is run otherwise users may exceed their quota.

I entered this as a feature request at cpanel.net.

Apparently this won't work. Answer from CPanel feature request ticket:

Hello, actually I'm afraid this is a bit more complicated than that, it would require re-writing a lot of the cpanel code as it relies quite a bit on the files in the /tmp directory being owned by the user otherwise it will freak a few other programs out. So at this time I can't see this as being feisable to be added into the system. It is a good idea but just a bit hard to implement easily at the moment.
--
Thanks
Clifford,

I asked for a new feature (ability to turn mailman off) on 22/7. Yesterday (17/10) I received a reply from Clifford stating:


2 points I'd like to make:
1) The Feature was implemented by Nick about a week after i requested it via CPanel forums.
2) The best way to disable links is with the "Feature Manager" and is certainly not to create a custom theme.

From this I have to conclude that Clifford really doesn't know too much about cPanel, so take his advice/feedback with a BIG grain of salt.

Well... I asked about being able to manually set each person's archiving options (without using the cpanel) and set them as default for new accounts... and I was told this was not possible and should be entered as a feature request.

However, looking into cpanellogd more, this is what I found out:

The settings for archiving and removing old archives for log files are stored in: /home/{user}/.cpanel-logs

the contents are:

archive-logs=1
remove-old-archived-logs=1

So, I just need to copy this file into each home directory... and should be able to put this file in the skeleton directory to make it a default setting for all new accounts.

Then I can either disable this feature from the cpanels or edit the skins to not have this option editable (if I don't want my cpanel users to modify these settings).

Very cool. So we can have cpanel manage all the logs and, with these settings, have the first and second items of my desired policy taken care of (rotate the logs, let the users download the previous month, etc.)

Left to do:
a) figure out an easy way to download every user's raw logs from each month: an ftp script? or modify cpanellogd to make a copy into a backup directory after rotating.
b) figure out how to handle quotas... now raw archived logs will be stored in /home/{account}/logs and urchin logs, which are about 20% the size of the raw logs will be stored in /home/{account}/tmp/urchin/data. I would rather not have either of these applied to the user's quota since these are due to MY policy.
c) automate the deletion of urchin reports older than a year.

Definitely making some progress. Does anyone else care about this stuff?

well... in case anyone cares in the future...

ok... so far everything worked. I copied the .cpanel-logs file into all existing accounts and into the skel directory (which works for new accounts). At the end of last month, everything worked beautifully... each account now has a 'logs' directory with {domain}-Oct-2003.gz in it--which is the compressed log that used to be in domlogs. Now domlogs has rotated and is starting new for the new month. I decided to leave the cpanel log manager options available to users and count any space the zipped logs take against their account. That way they can keep as many as they want there by unchecking the 'delete previous months' option or they can keep no raw logs by unchecking the 'keep' option. Whatever they want.

Now that I have those logs to play with, I found out that getting all the logs into a single directory to easily ftp them out is as easy as:

cp /backup/logs/ /home/*/logs/*.gz

then *poof* all the logs are in the /backup/logs directory (which I made for this purpose. Now I just download all the logs from this directory (then delete it to save space) and I have a backup copy of all the logs and can do whatever I want with them... namely to gather them together for my big customers and burn them to a cd once a year.

I can add this to a script along with the rm -f /backup/logs/* (before it) and call it from a CRON job to run on the first of each month. So far, everything is automated except for having to ftp down the logs sometime during each month. I can just have a CRONed lftp script do it for me if I wanted...

I think the only thing left to figure out is how to not count Urchin reports against the account's quota (they will add up significantly over the year) and delete any Urchin reports older than a year (should be easy to do with a script).

If anyone has tried this, let me know! Apparently this might be able to be done by changing the ownership... but then they might not be able to see the reports... maybe a group?

Thanks, Payne, for sharing your experience. I haven't touched the logs, so I can't be of any help, but it's nice to see people sharing their knowledge.

I, too, care about this. I have had a few prospective users inquire as to whether or not log files constitute part of their storage allocation. I've seen log files grow to 500 megs, so this is a valid question, and could be a good marketing point.

Kudos to you, Payne.