You are hereForums / Developer / Statistics from youtube_cache.log by day

Videocache 1.9.5 is available now. This version improves support for logging and fixes video caching problems for various sites like Vimeo, Break, Tube8 etc due to change in URL patterns. Also, from this version onwards, UrlGrabber has been removed from dependencies. Check Changelog for details. If you have purchased version 1.9.x previously, you can claim free upgrade by contacting using Contact Link. Please mention the email address using which you purchased Videocache. New users proceed to download page right now, secure your copy and enjoy caching :-)

Statistics from youtube_cache.log by day


6 replies [Last post]
lopan's picture
Offline
Joined: Nov 5 2008
Statistics from youtube_cache.log by day
Printer-friendly versionSend to friend

Hello there,

This a simple example, and a contribution, to Cache Video project using bash script to get some statistics from youtube_cache.log.

For execute this, use: ./script.sh n
where n = number of day ago to get statistics.

#!/bin/sh
# lopan dot eti at gmail dot com (Author: Lopan)
# GPL2
 
#Variables
VIDEO_CACHE_DIR=/var/spool/video_cache
VIDEO_CACHE_LOG=/var/log/youtube_cache/youtube_cache.log*
 
#Arguments
if [ -z $1 ]
then
        DA=0    #if arg $1 = NULL use 0 (0 = today)
else
        DA=$1   #set n days ago
fi
 
#Search and sum as total for files are gets from cache
TDW=0
for DW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep 303:http | awk '{print $5}'); do
                VDW=0
                VDW=`find $VIDEO_CACHE_DIR -name $DW.flv -exec ls -l {} \; | awk '{print $5}'`
                TDW=$((TDW+VDW))
                #Update the time of file (to date = $DA days ago) if the file are got on day.
                #
                #You can use this in future to delete old files.
                #The old files (if you use this command) represent the files are not accessed on cache.
                find $VIDEO_CACHE_DIR -iname $DW'*' -exec touch -t `date --date="$DA days ago" '+%m%d%H%M'` {} \;
done
 
#Search and sum as total for files are download on day
FDDWT=0
for DDW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep "Video was downloaded and cached." | awk '{print $5}'); do
                FDDW=`find $VIDEO_CACHE_DIR -name $DDW.flv -exec ls -l {} \; | awk '{print $5}'`
                FDDWT=$((FDDWT+FDDW))
done
 
#Print resume
echo "Daily economy (`date --date="$DA days ago" '+%Y-%m-%d'`): `echo $TDW/1024/1024 | bc` MB"
echo
echo "Total in MB of videos in cache: `du -csh $VIDEO_CACHE_DIR | grep total | awk '{print $1}'`"
echo "Total in MB of videos download by day (`date --date="$DA days ago" '+%Y-%m-%d'`): `echo $FDDWT/1024/1024 | bc` MB"

Lopan

admin's picture
Offline
Joined: Nov 2 2008

Thanks for sharing the script. I'll make a scripts section very soon and will put it up in faqs as well :)

Rafael Guedes's picture
Offline
Joined: Jan 9 2009

Hello Lopan,

I took the liberty to use your script to generate graphs in MRTG.

The file log_youtube.sh was placed in /etc with the following content:

#!/bin/sh
# lopan dot eti at gmail dot com (Author: Lopan)
# GPL2
 
#Variables
VIDEO_CACHE_DIR=/var/spool/video_cache
VIDEO_CACHE_LOG=/var/log/videocache/videocache.log*
 
#Arguments
if [ -z $1 ]
then
        DA=0    #if arg $1 = NULL use 0 (0 = today)
else
        DA=$1   #set n days ago
fi
 
#Search and sum as total for files are gets from cache
TDW=0
for DW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep 303:http | awk '{print $5}'); do
                VDW=0
                VDW=`find $VIDEO_CACHE_DIR -name $DW.flv -exec ls -l {} \; | awk '{print $5}'`
                TDW=$((TDW+VDW))
                #Update the time of file (to date = $DA days ago) if the file are got on day.
                #
                #You can use this in future to delete old files.
                #The old files (if you use this command) represent the files are not accessed on cache.
                find $VIDEO_CACHE_DIR -iname $DW'*' -exec touch -t `date --date="$DA days ago" '+%m%d%H%M'` {} \;
done
 
#Search and sum as total for files are download on day
FDDWT=0
for DDW in $(cat $VIDEO_CACHE_LOG | grep -w `date --date="$DA days ago" '+%Y-%m-%d'` | grep "Video was downloaded and cached." | awk '{print $5}'); do
                FDDW=`find $VIDEO_CACHE_DIR -name $DDW.flv -exec ls -l {} \; | awk '{print $5}'`
                FDDWT=$((FDDWT+FDDW))
done
 
#Print resume
#Daily economy
echo "`echo $TDW/1024/1024 | bc`" 
#Total in MB of videos download today
echo "`echo $FDDWT/1024/1024 | bc`"



And put this in mrtg.cfg
#---------------------
# Monitoring videocache
# performance
#---------------------
Target[videocache]: `/etc/log_youtube.sh`
Title[videocache]: videocache Performance
PageTop[videocache]: <h1>Economy X Downloaded Files by day</h1>
Options[videocache]: growright,noinfo,gauge
MaxBytes[videocache]: 200
Legend1[videocache]: Daily Economy
Legend2[videocache]: Downloaded Today (in MB)
LegendI[videocache]: Economy
LegendO[videocache]: Downloaded  


 The graph will show the results according to the range that you put
in MRTG (usually 5 minutes).
Ie, 5 in 5 minutes he will make a daily average of the results that
given moment.
Sorry for my English and thanks for your contribution Lopan.
admin's picture
Offline
Joined: Nov 2 2008

Hi Rafael,

Thanks for the wonderful work :D You rock!!! I hope people will find it useful to visualize their videocache logs.

Thanks again !!

lopan's picture
Offline
Joined: Nov 5 2008

No worry Rafael!

Nice graphics! MRTG is cool. But, I just redirect the output to a text plain log file.

My last line:

echo "`date --date="$DA days ago" '+%Y-%m-%d'` `echo $TDW/1024/1024 | bc`MB `du -csh $VIDEO_CACHE_DIR | grep total | awk '{print $1}'` `echo $FDDWT/1024/1024 | bc`MB"

Rafael, are you from Brazil? Please, contact me by mail.

Phantam (not verified)

this is wicked... one question thought, the bandwidth its showing is the amount sent from the cache to the clients... but it doesnt show overall how much youtube traffic their is including say... the ones that didnt get hits that still had to be downloaded (2x) it'd be nice if it could show the amount of hits made on day X and the amount of misses and then double the misses so that we can see actually if and when the cache starts moving into the green and saving bandwidth...

im not good with linux scripting is it possible one could modify a version for that?

admin's picture
Offline
Joined: Nov 2 2008

Hi Phantam,

You can try contacting Rafael using this contact mail form.

Thanks for using videocache :)

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <code>
  • Lines and paragraphs break automatically.
  • HTML tags will be transformed to conform to HTML standards.

More information about formatting options

CAPTCHA
Are you a human visitor ?
Image CAPTCHA
Enter the characters (without spaces) shown in the image.

Buy Now

Videocache Bundle (RPM and tar archive) is available. Secure your copy now by clicking the button below.
No. of Servers?
For details, check this page.

Recent comments

Spread it

Thank you for spreading the word!

Who's online

There are currently 0 users and 70 guests online.