VideoCache

Python statistics program

by Anonymous on 22 Nov 2008

I've created a python program to generate some slightly advanced statistics from the cache logs. It's located here: http://zrwr.tuxfamily.org/programming/youtube-cache-stats.tar.gz

It lists video totals, sizes, download dates, times watched, and bandwidth saved. Unfortunately, it's only accurate based on the logs it can read.

Edit: Updated link

13 Answers

by Kulbir Saini on 22 Nov 2008

Thanks a ton for writing the cache analyzer man!! Did you test it? If so, please share some statistics and also can you write a "How to use stat analyzer?", so that people can get help with different options. Also, put a copyright notice in header after GPL comment like

(C) Copyright 2008

by chris on 23 Nov 2008

You're welcome! I fixed a condition when a video was found the in logs but not in the cache, and I added options parsing. It seems to work fine on my system. It's now a tarball of a bazaar branch so changes can be tracked in the future.

By default, it works without any options. There are other options to control output but they might change over time so it's best to use: ./youtube-cache-stats.py --help to see all the options.

Since my old logs were being deleted in /var/log/squid (but it seems to not occur in /var/log/youtube_cache after upgrading to 1.5) and I'm the only one using the cache, I'll post up stats later when I watch more videos :)

by Kulbir Saini on 23 Nov 2008

Thanks man!! I am glad to see that you create a code repo for the same :) I'll try to include it in plugin in next version.

by Alexandre Correa on 7 Jan 2009

if you get this error:

Traceback (most recent call last):
  File "./videocache-stats.py", line 364, in ?
    run()
  File "./videocache-stats.py", line 357, in run
    analyze_logs(cache_record, log_files, options.verbose)
  File "./videocache-stats.py", line 163, in analyze_logs
    fi = fileinput.hook_compressed(log_path, "r")
AttributeError: 'module' object has no attribute 'hook_compressed'

your python version are < 2.5 !! to solve this.. apply patch attached

patch -p1 < videocache-stats.patch

######## file videocache-stats.patch #########

--- videocache-stats.py 2009-01-07 04:59:50.000000000 -0200
+++ videocache-stats_24.py      2009-01-07 04:59:38.000000000 -0200
@@ -160,7 +160,7 @@

        # For each log file
        for log_path in log_paths:
-               fi = fileinput.hook_compressed(log_path, "r")
+               fi = fileinput.input(log_path, "r")

                if verbose:
                        print "Reading", log_path

############ end ###########

hook_compressed featurea available for python >= 2.5 ... the difference between hook_compressed and input is that script canĀ“t support gzipped/bzipped files... :)

by chris on 9 Jan 2009

Thanks for the feedback, but that patch just wipes out the videocache logs. The argument "r" gets passed into the "inplace" parameter. I hope you didn't lose any data! Unfortunately, I never backed up my /var directory but it's not a big deal.

I'll find a solution to reading multiple compressed files for python version < 2.5 within a few days.

by Anonymous on 9 Jun 2009

Link is not working..

by Kulbir Saini on 27 Jun 2009

Hi!

For me its working fine. You want to try it again.

Thank You!

by mynullvoid on 29 Jun 2009

Hi, I extract your script and localize to my enviroment; but why I get 0 MB of bandwidth was saved in the summary:

Report approximately from Wed Jun 24 17:54:38 2009

CACHE SUMMARY
20 videos cached
318 MB of videos
0 MB of bandwidth was saved

At the detail I got this for the bandwidth saving:

= BANDWIDTH SAVING =
NAME                     SIZE MB DOWNLOAD      LAST WATCHED   WATCHED  SAVED MB
3a1b469c26bcc769              19 none          none                 0         0
088374622da0d8f7               1 3.1 days ago  3.1 days ago         1         0
85c68d67888748cc               5 3.0 days ago  3.0 days ago         1         0
20234911f0a01e7e               9 3.9 days ago  3.9 days ago         1         0
8fc843a666f70096               9 3.0 days ago  3.0 days ago         1         0
3c696f7ced8302bf              10 3.0 days ago  3.0 days ago         1         0
75a9cc8b99b5a314              10 2.9 days ago  2.9 days ago         1         0
71b18d5472628008              10 3.1 days ago  3.1 days ago         1         0
857b1de5ac3ca1ab              11 3.0 days ago  3.0 days ago         1         0
8eab6c2162d2da11              13 3.0 days ago  3.0 days ago         1         0
NAME                     SIZE MB DOWNLOAD      LAST WATCHED   WATCHED  SAVED MB
3a1b469c26bcc769              19 none          none                 0         0
088374622da0d8f7               1 3.1 days ago  3.1 days ago         1         0
85c68d67888748cc               5 3.0 days ago  3.0 days ago         1         0
20234911f0a01e7e               9 3.9 days ago  3.9 days ago         1         0
8fc843a666f70096               9 3.0 days ago  3.0 days ago         1         0
3c696f7ced8302bf              10 3.0 days ago  3.0 days ago         1         0
75a9cc8b99b5a314              10 2.9 days ago  2.9 days ago         1         0
71b18d5472628008              10 3.1 days ago  3.1 days ago         1         0
857b1de5ac3ca1ab              11 3.0 days ago  3.0 days ago         1         0
8eab6c2162d2da11              13 3.0 days ago  3.0 days ago         1         0

Please assist

by Kulbir Saini on 29 Jun 2009

Mynullvoid,

No video was watched more than once, so no saving.

Thank You!

by mynullvoid on 2 Jul 2009

I had been testing it by opening the same video over and over, and the result still shows 0 MB savings :(

by Kulbir Saini on 2 Jul 2009

Mynullvoid,

Do you see entries with CACHE_HIT in your videocache.log file?

Thank You!

by Ryz on 16 Jan 2010

Hi, where can I download the python script ?

regards

Ryz

by Anonymous on 26 Oct 2010

help pless

Traceback (most recent call last):
  File "/usr/local/share/videocache/videocache.py", line 847, in <module>
    log(format%(os.getpid(), '-', '-', 'LOG_ROTATE_ERR', '-', 'Could not rotate logfiles.'))
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
  File "/usr/local/share/videocache/videocache.py", line 847, in <module>
    log(format%(os.getpid(), '-', '-', 'LOG_ROTATE_ERR', '-', 'Could not rotate logfiles.'))
TypeError: 'NoneType' object is not callable

this my but install cache video in squid version 2.7.STABLE7

You need to sign in. Please sign in to add answer to this question.