How To - CacheVideos on PFsense

Thanks Juan for writing the wonderful howto :). This is an awesome contribution to videocache.

Hi,

I have followed the instruction given. installations successfully. But, my squid cannot start. This is the errors :

squid[4688]: The url_rewriter helpers are crashing too rapidly, need help!

Zaf,

Please inspect your /var/log/squid/cache.log file for python tracebacks. That will give me exact idea of what is wrong with your installation.

Thank you for considering videocache :)

Hello Zaf,

Use the following command

chmod -R 755 /var/log/videocache/

After that restart squid and I am sure it'll be fine :)

Thank you for using videocache :)

hi,

thank you for your reply. :)

it is very strange. after read the logs' "permission denied", the chmod should be fine. but it isn't. like nothing happened.

the same error comeout at cache.log and there is nothing in /var/log/videocache/*

*edit*
i try it be : chmod -R 777 /var/log/videocache
and it's working. but i don't know it is okay or not.

*edit2*
i have check at /var/log/videocache/videocache.log
there is too many log lines like below. and it makes my CPU usage to 100% all the time.

RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.

thank you.

Hi,

I have followed your instruction. plus, im sorry because i have not set /etc/videocache.conf and change to my proxy ip.

after restart squid, there is no more error logs. i wonder it is working. but i dont know how to prove it's working or not. then i try to test with traditional method by open a same video from youtube with 2 different computer. but it looks like the 2nd computer still download the video through the internet.

thank you for you time. :)

Hi, there is nothing in /var/spool/videocache/youtube/

also, there is nothing in /var/log/videocache/videocache.log
but, when i ls in /var/log/videocache/ ,there is a files :
videocache.log
videocache.log.1
videocache.log.2
videocache.log.3

in *.1 , *.2, *.3; too many lines like this :
2008-12-24 05:12:17,547 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.

thank you.

Sir,

I'm using pfsense 1.2.2 and I have a problem :

# pkg_add -r phyton
Error: FTP Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.0-release/Latest... File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.0-release/Latest/phyton.tbz' by URL

the phyton package is not available....
Any suggestion?

Thanks

The rpm creation for urlgrabber seems to be broken somehow for newer pfsense versions. I have 1.2.1-rc2 (openbsd7) and 1.2.3 (openbsd 7.1) and both give me this error after python setup.py bdist_rpm

building RPMs
rpm: not found
error: Failed to execute: "rpm -q --qf '%{name}-%{version}-%{release}.src.rpm %{arch}/%{name}-%{version}-%{release}.%{arch}.rpm\\n' --specfile 'build/bdist.freebsd-7.1-RELEASE-p2-i386/rpm/SPECS/urlgrabber.spec'"

Any ideas on how to fix or if I need some other packages for pfsense?

I went ahead with the install without the rpm's. After installing the everything I had the same problems as zaf did (and used the touch,chmod etc fixes). After I did all that I ran also to the same problem as he: 100% cpu usage (python processes) and lots of reloading of videocache.

I tested this with two very different networks and like said before using 1.2.1-RC(freebsd7) and 1.2.3(freebsd7.1).

Any ideas ?

Niko,

How many redirect_programs children are you using? Try to keep it 5-10.

Thank You!

Hello!

BSD systems doesn't use rpm packages as default. pfSense is FreeBSD based.

If the package doesn't exist for FreeBSD you have to download the tar.gz source and follow installation instructions.

See VIDEOCACHE HACKS, at lef. There is a hack for pfSense and another for FreeBSD.

Regards,

Josep Pujadas

admin: I used the default config so 10 is the maximum I had.

bellera: Indeed but the guide tells you to create(?) rpm's with python setup.py bdist_rpm

Niko,

Can you please inspect your /var/log/squid/cache.log and /var/log/squid/squid.out and videocache logfile /var/log/videocache.log to infer whats going on with videocache?

Thank You!

cache.log after squid start:

2009/03/07 12:51:53| parseHttpRequest: Requestheader contains NULL characters
2009/03/07 12:51:53| parseHttpRequest: Unsupported method '▒c~]▒,1N.▒▒[v▒$▒▒▒▒t▒?▒▒+▒:1▒7*▒▒▒▒J▒ۨj+O▒g▒'
2009/03/07 12:51:53| clientReadRequest: FD 67 (192.168.2.2:54123) Invalid Request
2009/03/07 12:52:51| WARNING: url_rewriter #10 (FD 54) exited
2009/03/07 12:52:52| WARNING: url_rewriter #7 (FD 51) exited
2009/03/07 12:52:53| WARNING: url_rewriter #2 (FD 46) exited
2009/03/07 12:52:53| WARNING: url_rewriter #8 (FD 52) exited
2009/03/07 12:52:53| WARNING: url_rewriter #4 (FD 48) exited
2009/03/07 12:52:53| Too few url_rewriter processes are running
2009/03/07 12:52:53| Starting new helpers
2009/03/07 12:52:53| helperOpenServers: Starting 10 'python' processes
2009/03/07 12:52:54| WARNING: url_rewriter #3 (FD 47) exited
2009/03/07 12:52:54| WARNING: url_rewriter #6 (FD 50) exited
2009/03/07 12:52:54| WARNING: url_rewriter #1 (FD 45) exited
2009/03/07 12:52:55| WARNING: url_rewriter #5 (FD 49) exited
2009/03/07 12:52:56| WARNING: url_rewriter #9 (FD 53) exited

Dont know about the weird symbols, those are on the log for some reason

Dont have any .out.

Videocache.log.1 is full of these:

2009-03-07 13:02:18,385 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,386 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,388 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,389 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,390 10422 - - RELOAD - videocache plugin was reloaded.

Videocache.log :

2009-03-07 12:56:19,335 11459 192.168.2.4 - REQUEST - http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=3...
2009-03-07 12:56:21,481 11459 192.168.2.4 de2d90e7af1a4fc3 URL_HIT YOUTUBE http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=3...
2009-03-07 12:56:21,514 11459 192.168.2.4 de2d90e7af1a4fc3 CACHE_MISS YOUTUBE Requested video was not found in cache.
2009-03-07 12:56:21,531 11459 192.168.2.4 de2d90e7af1a4fc3 NEW_URL YOUTUBE http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=3...

Seems to be logging my attempt to cache a video. I opened it on one machine(192.168.2.3) and after it had downloaded completely I tried opening it on a another machine(192.168.2.4, mentioned in this log). Some around that time cpu hit 100% and after 15minutes it havent gone down. Also the speed for the video wasnt any better on the other machine than straming from inet so I think it didnt even come from cache

Niko,

What is the value of base_dir option in /etc/videocache.conf ? The disk size calculation might be causing the problem.

Thank You!

It is the default base_dir = /var/spool/videocache/

I am using totally default configs with only the things mentioned in the guide and this thread.

Should I try changing this?

Only thing interesting I saw was that squid user is usually proxy (and the group is also proxy). The dir /var/spool/videocache is owned by user squid (and not proxy).

The directory exists tough, and the directories for youtube,youporn etc etc are there. Nothing in youtube dir altough I tried couple videos with videocache enabled.

Niko,

Then you need to change the user:group for /var/spool/videocache/ and subdirs to proxy:proxy. Those directories should be owned by squid user (for a lot of distros it is squid by default).

Thank You!

Yup, that fixed the cpu problem. It was probably trying to write that in to that directory and without the rights it crapped itself :)

Now when a video is being cached it goes to that directory and the log says for example

2009-03-08 10:03:31,411 34372 192.168.2.4 2a670f7ae7f9697a CACHE_HIT YOUTUBE Video was served from cache.

when I open the same video from another machine BUT the video doesnt open. It only says "We're sorry the requested video is no longer available". It isnt being streamd in any way, I was monitoring bandwidth use when trying this.

Any ideas on that? :)

It probably has something to do with the webserver and I find only this related to it:

2009-03-08 10:24:48,500 1440 - - START_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
2009-03-08 10:24:48,519 1440 - - SCHEDULEDER - Download Scheduler starting.

I tried this http://cachevideos.com/forum/post/solved-startxmlrpcservererr-cannot-sta... altough its not really the same because I got this errors just after rebooting. After this threads 'cleanup' I got this:

2009-03-08 10:30:23,608 4841 - - RELOAD - videocache plugin was reloaded.
2009-03-08 10:34:26,914 7366 - - XMLRPCSERVER - Starting XMLRPCServer on port 9100.

and 2009-03-08 10:34:31,471 7368 - - START_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting

And after those when I tried to open youtube videos I get the 'normal' messages which I described in the previous posts. Hit, being served but still "we're sorry..."

Niko,

Thats not an XMLRPC server problem. Its a problem with your webserver. Did you change the cache_host option in /etc/videocache.conf ? If not read the cache_host description carefully and change it. If you still face the same problem, report it back here.

Thank You!

The pfsense box ip is 192.168.2.1 and I have these:

cache_host = 192.168.2.1
proxy = http://192.168.2.1:3128/

and squid is using default port so 3128 should be ok. And https has been turned off

Dont know if this helps you but in squid.inc I have:

acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255

And in access.log when trying to access a cached youtube video I get:

1236503411.830 201 192.168.2.3 TCP_MISS/303 647 GET http://www.youtube.com/get_video? - DIRECT/208.65.153.238 text/html
1236503411.860 26 192.168.2.3 TCP_MISS/303 366 GET http://v15.cache.googlevideo.com/videoplayback? - NONE/- -

Niko,

Are you able to access http://192.168.2.1/videocache/ or http://192.168.2.1/videocache/0/ ? If not then your webserver is not working as expected.

Can you please paste a few lines are CACHE_HIT line in videocache log which redirects request to the video on your webserver?

Thank You!

Nope, 404 with both links.

2009-03-08 10:36:38,670 7461 192.168.2.3 48bdf067ab5d18ea URL_HIT YOUTUBE http://v22.cache.googlevideo.com/videoplayback?id=48bdf067ab5d18ea&itag=...
2009-03-08 10:36:38,682 7461 192.168.2.3 48bdf067ab5d18ea CACHE_HIT YOUTUBE Video was served from cache.
2009-03-08 10:36:38,683 7461 192.168.2.3 48bdf067ab5d18ea NEW_URL YOUTUBE 303:http://192.168.2.1/videocache/0/youtube/48bdf067ab5d18ea?id=48bdf067ab5d18ea&itag=34&ip=88.192.75.137&region=0&signature=419DF188CF1FC430AA272B7737A89B67141F1047.6000006349CB2D65B9C869AD42BB4D9382C2C107&sver=2&expire=1236522998&key=yt1&ipbits=0

2009-03-08 11:08:00,285 7461 192.168.2.3 2a670f7ae7f9697a URL_HIT YOUTUBE http://v15.cache.googlevideo.com/videoplayback?id=2a670f7ae7f9697a&itag=...
2009-03-08 11:08:00,296 7461 192.168.2.3 2a670f7ae7f9697a CACHE_HIT YOUTUBE Video was served from cache.
2009-03-08 11:08:00,297 7461 192.168.2.3 2a670f7ae7f9697a NEW_URL YOUTUBE 303:http://192.168.2.1/videocache/0/youtube/2a670f7ae7f9697a?id=2a670f7ae7f9697a&itag=34&ip=88.192.75.137&region=0&signature=51BB2302CFEADFE09A1E724E6E0C5D26F221D33F.1F2E01A16732BF24EDE5280594F486FD843E4CC7&sver=2&expire=1236524880&key=yt1&ipbits=0

Niko,

What are the contents of your /etc/httpd/conf.d/videocache.conf file? If application specific Apache configurations for your distro are to be stored in /etc/apache2/conf.d/, then copy the following lines to your /etc/apache2/conf.d/videocache.conf file and restart Apache webserver.

Alias /videocache/0 /var/spool/videocache
<Directory /var/spool/videocache>
  Options +Indexes
  Order Allow,Deny
  Allow from all
</Directory>

Thank You!

Ok, I will see what I can do about that. The how to doesnt mention this, only squid. Also the how to is broken in so many ways it probably would be better to mention that it doesnt work.

Also theres the guide for ubuntu+lighthttpd but I dont know if that will be any use since pfsense's lighthttpd seems to be quite different from default install.

But thanks for all the help anyway, I will see what I can do about all this. Would be really useful to get videocache working.

Just installed apache2 but I have no idea on how to configure it.

Atleast this guide didnt help http://www.bellera.cat/josep/videocache/

Niko,

Apache is default configuration works out of the box. You need to follow this. And hopefully videocache will work fine.

Thank You!

And theres nothing in httpd-access.log and configtest (with apache) went trough just fine.

Yup just what I tought. When I manually bind apache to the lan-interface (192.168.2.1) it blocks the normal webgui for pfsense.

Youtube seems to be working fine tough

BUT if you change pfsense webgui to use https (and of course different port, 443) everything works fine.

So in the end I have an working system!

Thanks for this great addon, I will begin testing it more and maybe write a how to for you on how it really works on pfsense.. The how to you got now isnt even close ;)

Niko,

Congratulations for the setup. If you can write the HowTo for PFSense, it would be a great contribution to videocache documentation.

Thank you for using videocache :)

Hi, I've gotten videocache to work on pfSense's lighttpd without installing apache additionally.

I am running pfsense 1.2.2 with the squid package installed and I followed the instructions here - 'http://cachevideos.com/install-videocache-pfsense' - to install videocache.

Following this thread, it was mentioned the need to install apache in order serve video from /var/spool/videocache using the alias function. Since my pfSense router box have limited ram, using the existing lighttpd was more economical.

My solution involves getting a version compatible 'mod_alias.so' module into '/usr/local/lib/lighttpd'. pfSense 1.2.2 is running lighttpd 1.4.20
and the default installation doesn't come with 'mod_alias.so'.

I compiled lighttpd 1.4.20 on another freebsd 7.1 box and scp 'mod_alias.so' over to 'pfsense:/usr/local/lib/lighttpd'

Next I added the following 2 modifications to '/etc/inc/system.inc'. Modifications are in bold.

At /etc/inc/system.inc(796)

server.modules = (
{$captive_portal_module}
"mod_access", "mod_accesslog", "mod_alias",
{$module}{$captiveportal}

And at /etc/inc/system.inc(913)

## virtual directory listings
server.dir-listing = "disable"

## alias
alias.url = ( "/videocache" => "/var/spool/videocache" )

## enable debugging
debug.log-request-header = "disable"
debug.log-response-header = "disable"
debug.log-request-handling = "disable"
debug.log-file-not-found = "disable"

Then exit from shell to 'pfSense console setup' menu and select '11' to Restart webConfigurator.

Verify in '/var/etc/lighty-webConfigurator.conf' that your changes have been appened.

From here you might need to restart squid to get videocache working. I found the caching behavior inconsistent now with some videos cached while some ignored.

To fix that I replaced in '/usr/local/pkg/squid.inc':


# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 10
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video
acl videocache_allow_url url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/videoplayback
acl videocache_allow_url url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/get_video
acl videocache_allow_url url_regex -i proxy\-[0-9][0-9]\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_dom dstdomain v.mccont.com vp.video.google.com dl.redtube.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid


which is shown from 'http://cachevideos.com/install-videocache-pfsense' with:


# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 7
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid


from 'http://cachevideos.com/installation' which seemed more up to date.
Restart squid after the change.

While solving the above I found that any slight misconfiguration will cause videocache to run many instance of 'python' and fully utilizing the CPU. My '/var/log/videocache/videocache.log' is filled with over 100MB of repeating lines of 'RELOAD - videocache plugin was reloaded' when it happened.

At the moment everything seems fine but I've only been running it for a few hours. Will report back if anything arises.

Tekkon,

Thats cool! Congrats for the setup and enjoy caching!

Thank You!

Oopss.. no good. In less than 4 hours, the system began to run many instance of python again:


PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
1085 proxy 2 97 0 11048K 7464K RUN 17:41 6.15% python
4402 proxy 1 97 0 10664K 7384K RUN 16:34 5.86% python
1087 proxy 2 97 0 11048K 7464K RUN 17:37 5.76% python
1084 proxy 2 97 0 11048K 7464K RUN 17:36 5.57% python
4401 proxy 1 97 0 10664K 7384K RUN 16:42 5.57% python
1086 proxy 2 97 0 11048K 7464K RUN 17:38 5.37% python
1082 proxy 2 97 0 11048K 7464K RUN 17:34 5.27% python
4399 proxy 1 97 0 10664K 7384K RUN 16:36 5.18% python
1083 proxy 2 97 0 11048K 7464K RUN 17:41 4.98% python
4397 proxy 1 97 0 10664K 7384K RUN 16:38 4.98% python
4400 proxy 1 97 0 10664K 7384K RUN 16:30 4.88% python
4398 proxy 1 97 0 10664K 7384K RUN 16:36 4.79% python
4403 proxy 1 97 0 10664K 7384K RUN 16:34 4.59% python
2027 proxy 1 116 20 10664K 7384K RUN 9:24 3.27% python
2026 proxy 1 116 20 10664K 7384K RUN 9:24 3.17% python
2023 proxy 1 116 20 10664K 7384K RUN 9:23 3.17% python
1081 proxy 4 96 0 11048K 7540K RUN 11:22 2.98% python
2022 proxy 1 116 20 10664K 7384K RUN 9:26 2.88% python
2025 proxy 1 116 20 10664K 7384K RUN 9:25 2.78% python
2028 proxy 1 116 20 10664K 7384K RUN 9:26 2.69% python
2024 proxy 1 116 20 10664K 7384K RUN 9:25 2.69% python


Here are 21 intense of python. So I guess my setup is still buggy.

I pasted my /var/log/videocache/videocache.log here - http://pastebin.com/m27d0dadc

and /var/squid/log/cache.log here - http://pastebin.com/m57361dd5

for analyzing.

Following the timestamps of the above log files, video cache was working nicely from around 05:53 after a restart. Then, until 09:23 the crash started.

Up until now my system has been running multiple python instance choking the CPU. At this moment ps -aux | grep python shows - http://pastebin.com/d49b7d3e0 - 28 instance of python.

My url_rewrite_children is set to 7. I think i'm seeing the running python processes in multiples of 7. Don't know if that mean anything. Any suggestions?

Tekkon,

To cache a video, videocache forks a daemon which is a python process. The default limit of parallel forking is 30. See max_parallel_downloads in /etc/videocache.conf for more details. Set it to a lower value like 10 or something.

Thank You!

Tekkon,

Open file /usr/share/videocache/videocache.py in any editor and do a search for line containing 'os.kill('. Comment all these lines or replace with a pass statement.

See if there are changes in behavior.

Thank You!

Updated Guide
It took me hours and now days to finally get to a working install. I have created a guide for installing on pfSense 1.2.2 that worked for me and currently is running with low CPU usage and is properly caching videos. It even works on iPhone YouTube videos. Sorry for the bad wiki formatting. I'll figure out how to fix the broken numbered list later.
 
http://wiki.ecnet.org/wiki/PfSenseVideoCache

LTitus,

Awesome tutorial man!!! Linking it in the "Videocache Hacks box" on the left sidebar. Keep up the good work!!!

Thank You!

Kulbir Saini,

After sometime I still have the same issues, many python instances until I run pkill python.

I tried editing /usr/share/videocache/videocache.py according to your instructions but the problem still persists.

I found that it only happens when squid loads during startup or restarts. Even though during the time of the bug when many python instance are eating away the CPU cycle, videocache is actually still caching video normally.

I've tried applying chown -R 777 to /var/log/videocache and /var/spool/videocache but it didn't helped.

LTitus210,

You mentioned in your wiki guide that /var/log/videocache/videocache.log is filling up to over 200MB in 5 minutes.

Whats the filling up with? Is it normal video caching log or are you getting any errors like 'RELOAD - videocache plugin was reloaded', similar to what I am getting?

After a reboot it seems to be working. The problem I mentioned before involves lighthttpd settings. It doesnt allow any directories to be listed (so you can access the videos but you cant list them )

Hi every body.

I had installed PFsense and try to configure videocache for 2 day but it still don't work T_T

first, I try on this link "http://cachevideos.com/install-videocache-pfsense" but system show me (system->service) that squid had been stoped
One again I try follow this link "http://wiki.ecnet.org/wiki/PfSenseVideoCache".
After finish system show me that squid still start until I had reboot Pfsense then system show me that squid had stop again

Notice that I use pfsense 1.2.3 RC1 and somehow squidguard didn't work property (I can access my black list's web).
I think I setup squidguard correctly.
Does my problem was squidguard ?

PS. I had no exp. on proxy server (actually, I had used Pfsense since 2 week ago)
PS2. Sorry for my bad english.