Splitting up Apache Logs for Subdomains

If you have multiple virtual hosts in your server and you specified different log files for each, it is more organized and easy to locate, but the downside is Linux has so called file descriptors or the number of files that it can be handle. Basically , it can be increased but I think it’s not really recommended just for the sake of it.
Making your apache logs more organized can be done by many ways, one way is via piped logging and use rotatelogs. And based on my tips before named Automatic unlimited subdomains via Apache mod_rewrite, one issue is the log files. But there’s one method to solve this, and this is by splitting up your log files.

1. Make sure your multiple virtual hosts log to the same log file.
example:
@ your config file

ErrorLog “|/usr/sbin/rotatelogs -l /var/log/httpd/linuxtutorials/error_log.%Y%m%d 86400”
CustomLog logs/linuxtutorials//access_log combinedio
CustomLog “|/usr/sbin/rotatelogs -l /var/log/httpd/linuxtutorials//access_log.%Y%m%d 86400” combinedio

2. Change your Apache LogFormat
from let say:
#LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” combined

to:
LogFormat “%V %h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” \”%{HTTP_X_UP_CALLING_LINE_ID}e\” \”%{HTTP_X_WSB_IDENTITY}e\ %I %O” combinedio

where: (use %V or %v based on your httpd.conf setting)
%V =the server name according to the UseCanonicalName setting
%v=the canonical ServerName  of the server serving the request.

3. Split your log files into its component parts and use the perl script “split-logfile. This program can be found in the support folder of your Apache distro.

split-logfile < /logs/access_log

You can create script to automate this, e.g.

[root@dc1 log]# cat split.sh
#!/bin/bash

#define
TODAY=`date “+%Y%m%d”`
YEST=`date “+%Y%m%d” -d”1 day ago”`
LOGPATH=”/var/log/httpd/linuxtutorials”
ACCESSFILE=”$LOGPATH/access_log.$YEST”
CURFOLDER=”/root/log”

#split
cd $CURFOLDER
/darwin/log/split-logfile < $ACCESSFILE
echo $ACCESSFILE
#rename
/usr/bin/rename log log.$YEST *log

-> This script basically reads the rotated access log from the log path, and split the log files according to the server name

Sample Results

@original access logs

[root@flt log]# ls -la /var/log/httpd/linuxtutorials/
total 544
drwxr-xr-x 2 root root   4096 Apr 20 16:07 .
drwx—— 3 root root   4096 Apr 18 04:02 ..

-rw-r–r– 1 root root 103090 Apr 20 16:08 access_log
-rw-r–r– 1 root root  53770 Apr 13 18:11 access_log.20100420
-rw-r–r– 1 root root  48660 Apr 13 18:11 access_log.20100421
-rw-r–r– 1 root root    710 Apr 15 16:33 error_log.20100420
-rw-r–r– 1 root root    50 Apr 15 16:33 error_log.20100421

@splitted log files after executing the script

[root@dc1 log]# ls -la /root/log
total 292
drwxr-xr-x 2 root root  4096 Apr 21 01:01 .
drwxr-x— 8 root root  4096 Apr 13 15:56 ..
-rw-r–r– 1 root root  9278 Apr 15 01:01 admin.freelinuxtutorials.com.log.20100420
-rw-r–r– 1 root root   436 Apr 15 01:01 www.freelinuxtutorials.com.log.20100420
-rw-r–r– 1 root root  1296 Apr 20 01:01 m.freelinuxtutorials.com.log.20100420
-rw-r–r– 1 root root  6104 Apr 14 01:01 secure.freelinuxtutorials.com.log.20100420

Share

About the author

tux

View all posts

Leave a Reply