Segfault > IT > General > Keep track of network traffic

(05.Apr.2010)



Introduction

The site you're seeing right now has a maximum amount of monthly traffic included in the basic subscription. If I transmit or receive more than that amount, I have to pay more.
As I didn't have a clue about how far I was from that limit I tried to find a program which keeps track of these numbers.

Unluckily - most probably because I just didn't search long enough - I didn't find anything that matched my requirements. So without almost noticing I started to write my own script.

It was my first PHP-script, and as at the beginning I didn't think it would end up being so complicated I wrote it directly in a tiny terminal, so the end result is a bit messy and not nicely structured.
But in the end it looks like it's working :o)

The requirements of this traffic-usage-tracking-script were:

  • keep track of all traffic generated by a specific interface - not just e.g. the one generated by Apache or by SSH.
  • survive server reboots. If I reboot the server or restart the network interface the counters shouldn't reset.
  • do not be invasive - do not slow down the network interface or use a lot of CPU.
  • aggregate on a monthly basis.

 

This is the resulting script which I wrote.


How it works

As I said - the script is quite ugly :o).
Some lines are weird as I had some problems executing some external commands and converting strings to numbers.

Anyway, if you want to use it you should create somewhere a directory where the script will put its own files and set the name of that directory in the variable "$transfer_db".
It will create in there 3 intermediate files and a sqlite-database where it will store all intermediate results and the final monthly aggregation.

You should run the script often (e.g. using a cronjob, or just a continuous loop with a "sleep" event in between), as if anything happens between the previous and the new update that data won't be taken into account.
In my case I let the script run once every 10 minutes => it's impossible that I have a huge volume increase in just a 10-minutes timespan, therefore losing the volume of that timespan from time to time is still acceptable.

What the script does in the end is to have a look at the figures reported by the command "ifconfig", save them in the database and aggregate the difference of the previous run with the current total.
If it sees that the RX- and TX-numbers reported by "ifconfig" are smaller than the ones it saw previously, it means that the network interface was probably somehow reset (e.g. server reboot) but it will continue to aggregate the difference starting with the new numbers.

I am running this script since ~1 month and until now it seems that the results are correct.


How to use it

Have a look at the top of the script and change the variables according to your needs.

Change especially the values of "$transfer_db" and "$temp_file[123]" setting the directory and filenames you wish, then set for "$monthlyTrafficTotalMAX" your the maximum monthly traffic/bandwidth limit (in bytes) that your provider allows you to generate and finally the name of the network interface you want to monitor ("$ifconfig_if2search").

Once you've done that, run once the script with "/usr/bin/php /[your_directory]/check_rx_and_tx_code".
You should see that the 4 files were created in the directory you set.

Then generate some traffic (transfer some files or whatever) and run the script again.
To check the results go to the directory where the database is located and access it with the usual sqlite commands ("sqlite myBwidth_db").
If you list all tables (command ".tab") you'll see that there are 4 tables. The one you're interested in is called "bwidth_monthly_total".

Show the column names of the table (command ".headers on") and issue e.g. the command...

select year, month, in_bytes / 1024 / 1024 "mb_in", out_bytes / 1024 / 1024 "mb_out", tot_gb, tot_gb_max_now, warning from bwidth_monthly_total;

Here an example of what should appear:

year month mb_in mb_out tot_gb tot_gb_max_now warning
2010 03 114.819579124451 111.462564468384 0.220978655852377 500 N
2010 04 482.94416809082 131.63143157959 0.600171484053135 83.3333333333023 N

 

Today, being April 5th, having started the script for the first time in March, I get the two lines above.

  • The first one shows me that in the previous month I read 114MB and sent 111MB, which makes a total of 0.22GB and as the end of the month was reached I had as well the potential to reach as well my maximum volume which is 500GB per month.
  • The second line shows that until today my network interface got 482MB and sent 131MB, which makes a total of 0.60GB and that today's maximum critical amount would have been about 83GB (500GB monthly divided by 30 days multiplied by 5)
    If it goes on like this until the end of the month I will therefore be well below the limit of 500GB.

If you want to show all columns and don't know SQL, the command is "select * from bwidth_monthly_total". For more complicated stuff please refer to some SQL-reference.

To exit from "sqlplus" press [CTRL+D].

If you run the script every 10 minutes, the database size will grow by ~200KB per month.
I think I included in the script a line which deletes old stuff (here set to delete anything older than 365 days), but as I didn't test it and as normally stuff that I don't test doesn't work, it's most probably not working :oP


I hope it's useful!!
It is very raw, but it should be easy to customize to meet your needs (e.g. write more "syslog"-lines to alert you automatically if the limit has been reached).