Support Queries

Statistics from youtube_cache.log by day

Hello there,

This a simple example, and a contribution, to Cache Video project using bash script to get some statistics from youtube_cache.log.

For execute this, use: ./script.sh n
where n = number of day ago to get statistics.

#!/bin/sh
# lopan dot eti at gmail dot com (Author: Lopan)
# GPL2

#Variables
VIDEO_CACHE_DIR=/var/spool/video_cache
VIDEO_CACHE_LOG=/var/log/youtube_cache/youtube_cache.log*

#Arguments
if [ -z $1 ]
then
        DA=0    #if arg $1 = NULL use 0 (0 = today)
else
        DA=$1   #set n days ago
fi

#Search and sum as total for files are gets from cache
TDW=0
for DW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep 303:http | awk '{print $5}'); do
                VDW=0
                VDW=`find $VIDEO_CACHE_DIR -name $DW.flv -exec ls -l {} \\; | awk '{print $5}'`
                TDW=$((TDW+VDW))
                #Update the time of file (to date = $DA days ago) if the file are got on day.
                #
                #You can use this in future to delete old files.
                #The old files (if you use this command) represent the files are not accessed on cache.
                find $VIDEO_CACHE_DIR -iname $DW'*' -exec touch -t `date --date="$DA days ago" '+%m%d%H%M'` {} \\;
done

#Search and sum as total for files are download on day
FDDWT=0
for DDW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep "Video was downloaded and cached." | awk '{print $5}'); do
                FDDW=`find $VIDEO_CACHE_DIR -name $DDW.flv -exec ls -l {} \\; | awk '{print $5}'`
                FDDWT=$((FDDWT+FDDW))
done

#Print resume
echo "Daily economy (`date --date="$DA days ago" '+%Y-%m-%d'`): `echo $TDW/1024/1024 | bc` MB"
echo
echo "Total in MB of videos in cache: `du -csh $VIDEO_CACHE_DIR | grep total | awk '{print $1}'`"
echo "Total in MB of videos download by day (`date --date="$DA days ago" '+%Y-%m-%d'`): `echo $FDDWT/1024/1024 | bc` MB"

Lopan

6 Answers

Thanks for sharing the script. I'll make a scripts section very soon and will put it up in faqs as well :)

Hello Lopan,

I took the liberty to use your script to generate graphs in MRTG.

The file log_youtube.sh was placed in /etc with the following content:

#!/bin/sh
# lopan dot eti at gmail dot com (Author: Lopan)
# GPL2

#Variables
VIDEO_CACHE_DIR=/var/spool/video_cache
VIDEO_CACHE_LOG=/var/log/videocache/videocache.log*

#Arguments
if [ -z $1 ]
then
        DA=0    #if arg $1 = NULL use 0 (0 = today)
else
        DA=$1   #set n days ago
fi

#Search and sum as total for files are gets from cache
TDW=0
for DW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep 303:http | awk '{print $5}'); do
                VDW=0
                VDW=`find $VIDEO_CACHE_DIR -name $DW.flv -exec ls -l {} \; | awk '{print $5}'`
                TDW=$((TDW+VDW))
                #Update the time of file (to date = $DA days ago) if the file are got on day.
                #
                #You can use this in future to delete old files.
                #The old files (if you use this command) represent the files are not accessed on cache.
                find $VIDEO_CACHE_DIR -iname $DW'*' -exec touch -t `date --date="$DA days ago" '+%m%d%H%M'` {} \;
done

#Search and sum as total for files are download on day
FDDWT=0
for DDW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep "Video was downloaded and cached." | awk '{print $5}'); do
                FDDW=`find $VIDEO_CACHE_DIR -name $DDW.flv -exec ls -l {} \; | awk '{print $5}'`
                FDDWT=$((FDDWT+FDDW))
done

#Print resume
#Daily economy
echo "`echo $TDW/1024/1024 | bc`" 
#Total in MB of videos download today
echo "`echo $FDDWT/1024/1024 | bc`"

And put this in mrtg.cfg

#---------------------
# Monitoring videocache
# performance
#---------------------
Target[videocache]: `/etc/log_youtube.sh`
Title[videocache]: videocache Performance
PageTop[videocache]: <h1>Economy X Downloaded Files by day</h1>
Options[videocache]: growright,noinfo,gauge
MaxBytes[videocache]: 200
Legend1[videocache]: Daily Economy
Legend2[videocache]: Downloaded Today (in MB)
LegendI[videocache]: Economy
LegendO[videocache]: Downloaded  

The graph will show the results according to the range that you put
in MRTG (usually 5 minutes). Ie, 5 in 5 minutes he will make a daily average of the results that given moment.

Sorry for my English and thanks for your contribution Lopan.

this is wicked... one question thought, the bandwidth its showing is the amount sent from the cache to the clients... but it doesnt show overall how much youtube traffic their is including say... the ones that didnt get hits that still had to be downloaded (2x) it'd be nice if it could show the amount of hits made on day X and the amount of misses and then double the misses so that we can see actually if and when the cache starts moving into the green and saving bandwidth...

im not good with linux scripting is it possible one could modify a version for that?

Hi Rafael,

Thanks for the wonderful work :D You rock!!! I hope people will find it useful to visualize their videocache logs.

Thanks again !!

Hi Phantam,

You can try contacting Rafael using this contact mail form.

Thanks for using videocache :)

No worry Rafael!

Nice graphics! MRTG is cool. But, I just redirect the output to a text plain log file.

My last line:

echo "`date --date="$DA days ago" '+%Y-%m-%d'` `echo $TDW/1024/1024 | bc`MB `du -csh $VIDEO_CACHE_DIR | grep total | awk '{print $1}'` `echo $FDDWT/1024/1024 | bc`MB"

Rafael, are you from Brazil? Please, contact me by mail.