You are hereForums / Developer / Python statistics program

Version 1.9.9 is out. We have added support for Youtube mobile videos, Facebook, AOL and MySpace videos as well. Fixed caching issues for some already supported websites. Options to control cache directory selection algorithm, bandwidth usage by videocache, etc. More robust videocache scheduler with a lot more options and features. Have a look at changelog for more details. If you have purchased Videocache in last one year, Contact Us to claim your free upgrade. Please mention the email address using which you purchased Videocache. New users proceed to Pricing Page right now, secure your copy and enjoy caching :-) Also, check our new and simplified upgrade policy :-)

Python statistics program

Post Reply

13 replies [Last post]

22 November, 2008 - 09:43

chris

Offline

Joined: Nov 22 2008

Python statistics program

I've created a python program to generate some slightly advanced statistics from the cache logs. It's located here: http://zrwr.tuxfamily.org/programming/youtube-cache-stats.tar.gz

It lists video totals, sizes, download dates, times watched, and bandwidth saved. Unfortunately, it's only accurate based on the logs it can read.

Edit: Updated link

22 November, 2008 - 12:07

admin

Online

Joined: Nov 2 2008

Thanks a ton for writing the cache analyzer man!! Did you test it? If so, please share some statistics and also can you write a "How to use stat analyzer?", so that people can get help with different options. Also, put a copyright notice in header after GPL comment like

23 November, 2008 - 07:39

chris

Offline

Joined: Nov 22 2008

You're welcome! I fixed a condition when a video was found the in logs but not in the cache, and I added options parsing. It seems to work fine on my system. It's now a tarball of a bazaar branch so changes can be tracked in the future.

By default, it works without any options. There are other options to control output but they might change over time so it's best to use: ./youtube-cache-stats.py --help to see all the options.

Since my old logs were being deleted in /var/log/squid (but it seems to not occur in /var/log/youtube_cache after upgrading to 1.5) and I'm the only one using the cache, I'll post up stats later when I watch more videos :)

23 November, 2008 - 11:42

admin

Online

Joined: Nov 2 2008

Thanks man!! I am glad to see that you create a code repo for the same :) I'll try to include it in plugin in next version.

7 January, 2009 - 12:33

Alexandre Correa (not verified)

if you get this error:

Traceback (most recent call last):
File "./videocache-stats.py", line 364, in ?
run()
File "./videocache-stats.py", line 357, in run
analyze_logs(cache_record, log_files, options.verbose)
File "./videocache-stats.py", line 163, in analyze_logs
fi = fileinput.hook_compressed(log_path, "r")
AttributeError: 'module' object has no attribute 'hook_compressed'

your python version are < 2.5 !! to solve this.. apply patch attached

patch -p1 < videocache-stats.patch

######## file videocache-stats.patch #########

--- videocache-stats.py 2009-01-07 04:59:50.000000000 -0200
+++ videocache-stats_24.py 2009-01-07 04:59:38.000000000 -0200
@@ -160,7 +160,7 @@

# For each log file
for log_path in log_paths:
- fi = fileinput.hook_compressed(log_path, "r")
+ fi = fileinput.input(log_path, "r")

if verbose:
print "Reading", log_path

############ end ###########

hook_compressed featurea available for python >= 2.5 ... the difference between hook_compressed and input is that script can´t support gzipped/bzipped files... :)

9 January, 2009 - 23:31

chris

Offline

Joined: Nov 22 2008

Thanks for the feedback, but that patch just wipes out the videocache logs. The argument "r" gets passed into the "inplace" parameter. I hope you didn't lose any data! Unfortunately, I never backed up my /var directory but it's not a big deal.

I'll find a solution to reading multiple compressed files for python version < 2.5 within a few days.

9 June, 2009 - 09:15

Anonymous (not verified)

Link is not working..

27 June, 2009 - 19:07

admin

Online

Joined: Nov 2 2008

Hi!

For me its working fine. You want to try it again.

Thank You!

29 June, 2009 - 07:59

mynullvoid (not verified)

Hi, I extract your script and localize to my enviroment; but why I get 0 MB of bandwidth was saved in the summary:

Report approximately from Wed Jun 24 17:54:38 2009

CACHE SUMMARY
20 videos cached
318 MB of videos
0 MB of bandwidth was saved

At the detail I got this for the bandwidth saving:

= BANDWIDTH SAVING =
NAME SIZE MB DOWNLOAD LAST WATCHED 3a1b469c26bcc769 19 none none 088374622da0d8f7 1 3.1 days ago 3.1 days ago 85c68d67888748cc 5 3.0 days ago 3.0 days ago 20234911f0a01e7e 9 3.9 days ago 3.9 days ago 8fc843a666f70096 9 3.0 days ago 3.0 days ago 3c696f7ced8302bf 10 3.0 days ago 3.0 days ago 75a9cc8b99b5a314 10 2.9 days ago 2.9 days ago 71b18d5472628008 10 3.1 days ago 3.1 days ago 857b1de5ac3ca1ab 11 3.0 days ago 3.0 days ago 8eab6c2162d2da11 13 3.0 days ago 3.0 days ago NAME SIZE MB DOWNLOAD LAST WATCHED 3a1b469c26bcc769 19 none none 088374622da0d8f7 1 3.1 days ago 3.1 days ago 85c68d67888748cc 5 3.0 days ago 3.0 days ago 20234911f0a01e7e 9 3.9 days ago 3.9 days ago 8fc843a666f70096 9 3.0 days ago 3.0 days ago 3c696f7ced8302bf 10 3.0 days ago 3.0 days ago 75a9cc8b99b5a314 10 2.9 days ago 2.9 days ago 71b18d5472628008 10 3.1 days ago 3.1 days ago 857b1de5ac3ca1ab 11 3.0 days ago 3.0 days ago 8eab6c2162d2da11 13 3.0 days ago 3.0 days ago WATCHED SAVED MB
0 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
WATCHED SAVED MB
0 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0
1 0

Please assist

29 June, 2009 - 09:39

admin

Online

Joined: Nov 2 2008

Mynullvoid,

No video was watched more than once, so no saving.

Thank You!

2 July, 2009 - 13:24

#10

mynullvoid

Offline

Joined: Jul 1 2009

I had been testing it by opening the same video over and over, and the result still shows 0 MB savings :(

2 July, 2009 - 17:38

#11

admin

Online

Joined: Nov 2 2008

Mynullvoid,

Do you see entries with CACHE_HIT in your videocache.log file?

Thank You!

17 January, 2010 - 02:00

#12

Ryz (not verified)

Hi, where can I download the python script ?

regards

Ryz

26 October, 2010 - 16:21

#13

Anonymous (not verified)

help pless

Traceback (most recent call last):
File "/usr/local/share/videocache/videocache.py", line 847, in
log(format%(os.getpid(), '-', '-', 'LOG_ROTATE_ERR', '-', 'Could not rotate logfiles.'))
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
File "/usr/local/share/videocache/videocache.py", line 847, in
log(format%(os.getpid(), '-', '-', 'LOG_ROTATE_ERR', '-', 'Could not rotate logfiles.'))
TypeError: 'NoneType' object is not callable

this my but install cache video in squid version 2.7.STABLE7

Introduction

VideoCache Hacks

Python statistics program

Post new comment

My Book on Squid

Purchase/Renew

New forum topics

Recent comments

Who's online

Introduction

User login

VideoCache Hacks

Python statistics program

Post new comment

My Book on Squid

Purchase/Renew

New forum topics

Recent comments

Who's online