How To - CacheVideos on PFsense
by jchuerta on 1 Dec 2008Kulbir Great Product congratulations! I will try the how-to on PFsense if you look something wrong please help me Thanks !
This howto has been moved to How to install Video Cache on PFSense?. Please visit to see details.
Use this forum topic to comment on the howto or reporting problems with howto.
Good Luck !
70 Answers
Thanks Juan for writing the wonderful howto :). This is an awesome contribution to videocache.
Hi,
I have followed the instruction given. installations successfully. But, my squid cannot start. This is the errors :
squid[4688]: The url_rewriter helpers are crashing too rapidly, need help!
i have changed my squid from squid2 to squid3. and the problem i mentioned above, has gone. But there is another error.
squid: redirect_program /usr/bin/python: (2) No such file or directory
can somebody help?
Hi Zaf,
We don't use redirect_program directive for videocache. We use url_rewrite_program directive instead. Please go through the "Configure Squid" part on Videocache on PFSense again and add correct lines to your squid.conf file.
Thank you for considering videocache :)
Hi,
thank you for your reply. i have put same like the configuration guide. in the system logs, it show :
(squid): The redirector helpers are crashing to rapidly, need help!
and the warning shows every seconds. squid cannot start. this all stop when i delete the video cache configuration lines and squid can start again.
thank you.
Zaf,
Please inspect your /var/log/squid/cache.log file for python tracebacks. That will give me exact idea of what is wrong with your installation.
Thank you for considering videocache :)
my squid logs located in : /var/squid/log/cache.log
Here is the cache log after i inserted the conf.
2008/12/22 21:55:15| Preparing for shutdown after 0 requests
2008/12/22 21:55:15| Waiting 3 seconds for active connections to finish
2008/12/22 21:55:15| FD 20 Closing HTTP connection
2008/12/22 21:55:15| FD 21 Closing HTTP connection
2008/12/22 21:55:15| Closing Pinger socket on FD 24
2008/12/22 21:55:19| Shutting down...
2008/12/22 21:55:19| Closing unlinkd pipe on FD 22
2008/12/22 21:55:19| storeDirWriteCleanLogs: Starting...
2008/12/22 21:55:19| 65536 entries written so far.
2008/12/22 21:55:20| 131072 entries written so far.
2008/12/22 21:55:20| 196608 entries written so far.
2008/12/22 21:55:20| 262144 entries written so far.
2008/12/22 21:55:20| 327680 entries written so far.
2008/12/22 21:55:20| 393216 entries written so far.
2008/12/22 21:55:20| 458752 entries written so far.
2008/12/22 21:55:20| 524288 entries written so far.
2008/12/22 21:55:20| 589824 entries written so far.
2008/12/22 21:55:20| 655360 entries written so far.
2008/12/22 21:55:20| 720896 entries written so far.
2008/12/22 21:55:20| 786432 entries written so far.
2008/12/22 21:55:20| 851968 entries written so far.
2008/12/22 21:55:20| 917504 entries written so far.
2008/12/22 21:55:20| 983040 entries written so far.
2008/12/22 21:55:20| 1048576 entries written so far.
2008/12/22 21:55:20| 1114112 entries written so far.
2008/12/22 21:55:20| 1179648 entries written so far.
2008/12/22 21:55:20| Finished. Wrote 1228109 entries.
2008/12/22 21:55:20| Took 0.96 seconds (1282671.27 entries/sec).
CPU Usage: 10.421 seconds = 3.560 user + 6.861 sys
Maximum Resident Size: 115644 KB
Page faults with physical i/o: 0
2008/12/22 21:55:20| Squid Cache (Version 3.0.STABLE8): Exiting normally.
other log is in pfsense system logs.
Dec 22 21:56:01 squid: redirect_program /usr/bin/python: (2) No such file or directory
I think it is because squid cannot reach python location. im also have check at /usr/bin there is no python file or folder but python is installed. maybe it's located in another place, but i dont know where it is.
Zaf,
Try locating python in your system. Use 'locate' command if available in your system and specify the correct python path to url_rewrite_program. Other command you can use is 'which'.
Thank You.
hi,
i have use which python, and it's located /usr/local/bin/python
good news squid started, but a few seconds then, it stop itself.
(squid): The redirector helpers are crashing too rapidly, need help!
:)
Hi Zaf,
Any python tracebacks in /var/log/squid/cache.log ? If they are crashing, they must leave some python tracebacks in cache.log. Also can you please paste your /var/log/videocache/videocache.log on http://pastebin.com/ .
Thank You.
Hello Zaf,
Use the following command
chmod -R 755 /var/log/videocache/
After that restart squid and I am sure it'll be fine :)
Thank you for using videocache :)
hi,
thank you for your reply. :)
it is very strange. after read the logs' "permission denied", the chmod should be fine. but it isn't. like nothing happened.
the same error comeout at cache.log and there is nothing in /var/log/videocache/*
edit
i try it be : chmod -R 777 /var/log/videocache
and it's working. but i don't know it is okay or not.
edit2
i have check at /var/log/videocache/videocache.log
there is too many log lines like below. and it makes my CPU usage to 100% all the time.
RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 04:51:03,960 INFO - - RELOAD - videocache plugin was reloaded.
thank you.
Hi zaf,
777 for
Replace USER with the squid user in your system and GROUP with squid group in your system.
touch /var/log/videocache/videocache.log
chown -R USER:GROUP /var/log/videocache/
chmod -R 755 /var/log/videocache/
The access denied error is occurring probably because in your system squid user and squid group are not squid, squid. Please find out the same and use above commands to rectify the problem.
Also the RELOAD line in squid states that the plugin instance was reloaded. It occurs when you restart/reload your squid service. There is nothing wrong with these lines in log, so don't worry about them.
CPU usage may be reaching 100% due to other processes or may be because some process is hanged due to permission problems.
Please rectify the permission problem first and then we'll see whats wrong with other things.
Thank you for using videocache :)
Hi,
I have followed your instruction. plus, im sorry because i have not set /etc/videocache.conf and change to my proxy ip.
after restart squid, there is no more error logs. i wonder it is working. but i dont know how to prove it's working or not. then i try to test with traditional method by open a same video from youtube with 2 different computer. but it looks like the 2nd computer still download the video through the internet.
thank you for you time. :)
Hello Zaf,
Please check if there are some files cached in /var/spool/videocache/youtube/. Or you can verify caching by having a look at /var/log/videocache/videocache.log file.
Thank you for using videocache :)
Hi, there is nothing in /var/spool/videocache/youtube/
also, there is nothing in /var/log/videocache/videocache.log
but, when i ls in /var/log/videocache/ ,there is a files :
videocache.log
videocache.log.1
videocache.log.2
videocache.log.3
in *.1 , *.2, *.3; too many lines like this :
2008-12-24 05:12:17,547 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
2008-12-24 05:12:17,548 INFO - - RELOAD - videocache plugin was reloaded.
thank you.
Zaf,
Your videocache problem looks strange. Contact me using Contact tab at the top.
Thank You.
Sir,
I'm using pfsense 1.2.2 and I have a problem :
# pkg_add -r phyton
Error: FTP Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.0-release/Latest/phyton.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.0-release/Latest/phyton.tbz' by URL
the phyton package is not available....
Any suggestion?
Thanks
Hi Dedi,
I am not really sure if this problem can be addressed here. Please try asking in pfsense forums.
Thank you for considering videocache!
The rpm creation for urlgrabber seems to be broken somehow for newer pfsense versions. I have 1.2.1-rc2 (openbsd7) and 1.2.3 (openbsd 7.1) and both give me this error after python setup.py bdist_rpm
building RPMs
rpm: not found
error: Failed to execute: "rpm -q --qf '%{name}-%{version}-%{release}.src.rpm %{arch}/%{name}-%{version}-%{release}.%{arch}.rpm\\\\n' --specfile 'build/bdist.freebsd-7.1-RELEASE-p2-i386/rpm/SPECS/urlgrabber.spec'"
Any ideas on how to fix or if I need some other packages for pfsense?
I went ahead with the install without the rpm's. After installing the everything I had the same problems as zaf did (and used the touch,chmod etc fixes). After I did all that I ran also to the same problem as he: 100% cpu usage (python processes) and lots of reloading of videocache.
I tested this with two very different networks and like said before using 1.2.1-RC(freebsd7) and 1.2.3(freebsd7.1).
Any ideas ?
Niko,
How many redirect_programs children are you using? Try to keep it 5-10.
Thank You!
Hello!
BSD systems doesn't use rpm packages as default. pfSense is FreeBSD based.
If the package doesn't exist for FreeBSD you have to download the tar.gz source and follow installation instructions.
See VIDEOCACHE HACKS, at lef. There is a hack for pfSense and another for FreeBSD.
Regards,
Josep Pujadas
admin: I used the default config so 10 is the maximum I had.
bellera: Indeed but the guide tells you to create(?) rpm's with python setup.py bdist_rpm
Niko,
Can you please inspect your /var/log/squid/cache.log and /var/log/squid/squid.out and videocache logfile /var/log/videocache.log to infer whats going on with videocache?
Thank You!
cache.log after squid start:
2009/03/07 12:51:53| parseHttpRequest: Requestheader contains NULL characters
2009/03/07 12:51:53| parseHttpRequest: Unsupported method '▒c~]▒,1N.▒▒[v▒$▒▒▒▒t▒?▒▒+▒:1▒7*▒▒▒▒J▒ۨj+O▒g▒'
2009/03/07 12:51:53| clientReadRequest: FD 67 (192.168.2.2:54123) Invalid Request
2009/03/07 12:52:51| WARNING: url_rewriter #10 (FD 54) exited
2009/03/07 12:52:52| WARNING: url_rewriter #7 (FD 51) exited
2009/03/07 12:52:53| WARNING: url_rewriter #2 (FD 46) exited
2009/03/07 12:52:53| WARNING: url_rewriter #8 (FD 52) exited
2009/03/07 12:52:53| WARNING: url_rewriter #4 (FD 48) exited
2009/03/07 12:52:53| Too few url_rewriter processes are running
2009/03/07 12:52:53| Starting new helpers
2009/03/07 12:52:53| helperOpenServers: Starting 10 'python' processes
2009/03/07 12:52:54| WARNING: url_rewriter #3 (FD 47) exited
2009/03/07 12:52:54| WARNING: url_rewriter #6 (FD 50) exited
2009/03/07 12:52:54| WARNING: url_rewriter #1 (FD 45) exited
2009/03/07 12:52:55| WARNING: url_rewriter #5 (FD 49) exited
2009/03/07 12:52:56| WARNING: url_rewriter #9 (FD 53) exited
Dont know about the weird symbols, those are on the log for some reason
Dont have any .out.
Videocache.log.1 is full of these:
2009-03-07 13:02:18,385 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,386 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,388 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,389 10422 - - RELOAD - videocache plugin was reloaded.
2009-03-07 13:02:18,390 10422 - - RELOAD - videocache plugin was reloaded.
Videocache.log :
2009-03-07 12:56:19,335 11459 192.168.2.4 - REQUEST - http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=34&ip=88.192.75.137®ion=0&signature=9EF4201B5FADCE8EEBAE8ECF606C31C50D2B647D.8D81F9E2DF466027E0BE4ED58C4178B4B218835F&sver=2&expire=1236444979&key=yt1&ipbits=0
2009-03-07 12:56:21,481 11459 192.168.2.4 de2d90e7af1a4fc3 URL_HIT YOUTUBE http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=34&ip=88.192.75.137®ion=0&signature=9EF4201B5FADCE8EEBAE8ECF606C31C50D2B647D.8D81F9E2DF466027E0BE4ED58C4178B4B218835F&sver=2&expire=1236444979&key=yt1&ipbits=0
2009-03-07 12:56:21,514 11459 192.168.2.4 de2d90e7af1a4fc3 CACHE_MISS YOUTUBE Requested video was not found in cache.
2009-03-07 12:56:21,531 11459 192.168.2.4 de2d90e7af1a4fc3 NEW_URL YOUTUBE http://v9.cache.googlevideo.com/videoplayback?id=de2d90e7af1a4fc3&itag=34&ip=88.192.75.137®ion=0&signature=9EF4201B5FADCE8EEBAE8ECF606C31C50D2B647D.8D81F9E2DF466027E0BE4ED58C4178B4B218835F&sver=2&expire=1236444979&key=yt1&ipbits=0
Seems to be logging my attempt to cache a video. I opened it on one machine(192.168.2.3) and after it had downloaded completely I tried opening it on a another machine(192.168.2.4, mentioned in this log). Some around that time cpu hit 100% and after 15minutes it havent gone down. Also the speed for the video wasnt any better on the other machine than straming from inet so I think it didnt even come from cache
Niko,
What is the value of base_dir option in /etc/videocache.conf ? The disk size calculation might be causing the problem.
Thank You!
It is the default base_dir = /var/spool/videocache/
I am using totally default configs with only the things mentioned in the guide and this thread.
Should I try changing this?
Only thing interesting I saw was that squid user is usually proxy (and the group is also proxy). The dir /var/spool/videocache is owned by user squid (and not proxy).
The directory exists tough, and the directories for youtube,youporn etc etc are there. Nothing in youtube dir altough I tried couple videos with videocache enabled.
Niko,
Then you need to change the user:group for /var/spool/videocache/ and subdirs to proxy:proxy. Those directories should be owned by squid user (for a lot of distros it is squid by default).
Thank You!
Yup, that fixed the cpu problem. It was probably trying to write that in to that directory and without the rights it crapped itself :)
Now when a video is being cached it goes to that directory and the log says for example
2009-03-08 10:03:31,411 34372 192.168.2.4 2a670f7ae7f9697a CACHE_HIT YOUTUBE Video was served from cache.
when I open the same video from another machine BUT the video doesnt open. It only says "We're sorry the requested video is no longer available". It isnt being streamd in any way, I was monitoring bandwidth use when trying this.
Any ideas on that? :)
It probably has something to do with the webserver and I find only this related to it:
2009-03-08 10:24:48,500 1440 - - START_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
2009-03-08 10:24:48,519 1440 - - SCHEDULEDER - Download Scheduler starting.
I tried this http://cachevideos.com/forum/post/solved-startxmlrpcservererr-cannot-start-xmlrpc-server-exiting altough its not really the same because I got this errors just after rebooting. After this threads 'cleanup' I got this:
2009-03-08 10:30:23,608 4841 - - RELOAD - videocache plugin was reloaded.
2009-03-08 10:34:26,914 7366 - - XMLRPCSERVER - Starting XMLRPCServer on port 9100.
2009-03-08 10:34:31,471 7368 - - START_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
And after those when I tried to open youtube videos I get the 'normal' messages which I described in the previous posts. Hit, being served but still "we're sorry..."
Niko,
Thats not an XMLRPC server problem. Its a problem with your webserver. Did you change the cache_host option in /etc/videocache.conf ? If not read the cache_host description carefully and change it. If you still face the same problem, report it back here.
Thank You!
The pfsense box ip is 192.168.2.1 and I have these:
cache_host = 192.168.2.1
proxy = http://192.168.2.1:3128/
and squid is using default port so 3128 should be ok. And https has been turned off
Dont know if this helps you but in squid.inc I have:
acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
And in access.log when trying to access a cached youtube video I get:
1236503411.830 201 192.168.2.3 TCP_MISS/303 647 GET http://www.youtube.com/get_video? - DIRECT/208.65.153.238 text/html
1236503411.860 26 192.168.2.3 TCP_MISS/303 366 GET http://v15.cache.googlevideo.com/videoplayback? - NONE/- -
Niko,
Are you able to access http://192.168.2.1/videocache/ or http://192.168.2.1/videocache/0/ ? If not then your webserver is not working as expected.
Can you please paste a few lines are CACHE_HIT line in videocache log which redirects request to the video on your webserver?
Thank You!
Nope, 404 with both links.
2009-03-08 10:36:38,670 7461 192.168.2.3 48bdf067ab5d18ea URL_HIT YOUTUBE http://v22.cache.googlevideo.com/videoplayback?id=48bdf067ab5d18ea&itag=34&ip=88.192.75.137®ion=0&signature=419DF188CF1FC430AA272B7737A89B67141F1047.6000006349CB2D65B9C869AD42BB4D9382C2C107&sver=2&expire=1236522998&key=yt1&ipbits=0
2009-03-08 10:36:38,682 7461 192.168.2.3 48bdf067ab5d18ea CACHE_HIT YOUTUBE Video was served from cache.
2009-03-08 10:36:38,683 7461 192.168.2.3 48bdf067ab5d18ea NEW_URL YOUTUBE 303:http://192.168.2.1/videocache/0/youtube/48bdf067ab5d18ea?id=48bdf067ab5d18ea&itag=34&ip=88.192.75.137®ion=0&signature=419DF188CF1FC430AA272B7737A89B67141F1047.6000006349CB2D65B9C869AD42BB4D9382C2C107&sver=2&expire=1236522998&key=yt1&ipbits=0
2009-03-08 11:08:00,285 7461 192.168.2.3 2a670f7ae7f9697a URL_HIT YOUTUBE http://v15.cache.googlevideo.com/videoplayback?id=2a670f7ae7f9697a&itag=34&ip=88.192.75.137®ion=0&signature=51BB2302CFEADFE09A1E724E6E0C5D26F221D33F.1F2E01A16732BF24EDE5280594F486FD843E4CC7&sver=2&expire=1236524880&key=yt1&ipbits=0
2009-03-08 11:08:00,296 7461 192.168.2.3 2a670f7ae7f9697a CACHE_HIT YOUTUBE Video was served from cache.
2009-03-08 11:08:00,297 7461 192.168.2.3 2a670f7ae7f9697a NEW_URL YOUTUBE 303:http://192.168.2.1/videocache/0/youtube/2a670f7ae7f9697a?id=2a670f7ae7f9697a&itag=34&ip=88.192.75.137®ion=0&signature=51BB2302CFEADFE09A1E724E6E0C5D26F221D33F.1F2E01A16732BF24EDE5280594F486FD843E4CC7&sver=2&expire=1236524880&key=yt1&ipbits=0
Niko,
What are the contents of your /etc/httpd/conf.d/videocache.conf file? If application specific Apache configurations for your distro are to be stored in /etc/apache2/conf.d/, then copy the following lines to your /etc/apache2/conf.d/videocache.conf file and restart Apache webserver.
Alias /videocache/0 /var/spool/videocache
<Directory /var/spool/videocache>
Options +Indexes
Order Allow,Deny
Allow from all
</Directory>
Thank You!
Alias /videocache/0 /var/spool/videocache
<Directory /var/spool/videocache>
Options +Indexes
Order Allow,Deny
#Comment the following line and uncomment the next for public use
#Deny from all
Allow from all
</Directory>
No apache folders (pfsense only has lighthttpd)
Niko,
Apache is listed as one of the dependencies on the installation page. I think you overlooked that. You can make videocache work with lighttpd but I don't know the procedure for that. Installing Apache will be a quick solution for you.
Thank You!
Ok, I will see what I can do about that. The how to doesnt mention this, only squid. Also the how to is broken in so many ways it probably would be better to mention that it doesnt work.
Also theres the guide for ubuntu+lighthttpd but I dont know if that will be any use since pfsense's lighthttpd seems to be quite different from default install.
But thanks for all the help anyway, I will see what I can do about all this. Would be really useful to get videocache working.
Just installed apache2 but I have no idea on how to configure it.
Atleast this guide didnt help http://www.bellera.cat/josep/videocache/
Niko,
Apache is default configuration works out of the box. You need to follow this. And hopefully videocache will work fine.
Thank You!
Yup, but theres no such directories in freebsd apache2.
/usr/local/etc/apache22/ directory has httpd.conf. I tried putting videocache.conf there and also in a directory called Includes which is under it -> no effect. I dont know if lighthttpd messes things up or if there should be something to be done in httpd.conf. And yes, apache is loaded ;)
I think the Includes directory is same as conf.d since httpd.conf has this: Include etc/apache22/Includes/*.conf
Maybe this helps: http://wiki.apache.org/httpd/DistrosDefaultLayout#head-cbf989510766645be58fcfe888efe08427dee9a7
And theres nothing in httpd-access.log and configtest (with apache) went trough just fine.
Yup just what I tought. When I manually bind apache to the lan-interface (192.168.2.1) it blocks the normal webgui for pfsense.
Youtube seems to be working fine tough
BUT if you change pfsense webgui to use https (and of course different port, 443) everything works fine.
So in the end I have an working system!
Thanks for this great addon, I will begin testing it more and maybe write a how to for you on how it really works on pfsense.. The how to you got now isnt even close ;)
Niko,
Congratulations for the setup. If you can write the HowTo for PFSense, it would be a great contribution to videocache documentation.
Thank you for using videocache :)
Hi, I've gotten videocache to work on pfSense's lighttpd without installing apache additionally.
I am running pfsense 1.2.2 with the squid package installed and I followed the instructions here - 'http://cachevideos.com/install-videocache-pfsense' - to install videocache.
Following this thread, it was mentioned the need to install apache in order serve video from /var/spool/videocache using the alias function. Since my pfSense router box have limited ram, using the existing lighttpd was more economical.
My solution involves getting a version compatible 'mod_alias.so' module into '/usr/local/lib/lighttpd'. pfSense 1.2.2 is running lighttpd 1.4.20
and the default installation doesn't come with 'mod_alias.so'.
I compiled lighttpd 1.4.20 on another freebsd 7.1 box and scp 'mod_alias.so' over to 'pfsense:/usr/local/lib/lighttpd'
Next I added the following 2 modifications to '/etc/inc/system.inc'. Modifications are in bold.
At /etc/inc/system.inc(796)
server.modules = (
{$captive_portal_module}
"mod_access", "mod_accesslog", <strong>"mod_alias",</strong>
{$module}{$captiveportal}
And at /etc/inc/system.inc(913)
## virtual directory listings
server.dir-listing = "disable"
## alias
alias.url = ( "/videocache" => "/var/spool/videocache" )</strong>
## enable debugging
debug.log-request-header = "disable"
debug.log-response-header = "disable"
debug.log-request-handling = "disable"
debug.log-file-not-found = "disable"
Then exit from shell to 'pfSense console setup' menu and select '11' to Restart webConfigurator.
Verify in '/var/etc/lighty-webConfigurator.conf' that your changes have been appened.
From here you might need to restart squid to get videocache working. I found the caching behavior inconsistent now with some videos cached while some ignored.
To fix that I replaced in '/usr/local/pkg/squid.inc':
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 10
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video
acl videocache_allow_url url_regex -i \\.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.googlevideo\\.com\\/videoplayback
acl videocache_allow_url url_regex -i \\.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.googlevideo\\.com\\/get_video
acl videocache_allow_url url_regex -i proxy\\-[0-9][0-9]\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\\.xtube\\.com\\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\\.vimeo\\.com\\/vimeo\\/videos\\/
acl videocache_allow_url url_regex -i va\\.wrzuta\\.pl\\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \\.files\\.youporn\\.com\\/(.*)\\/flv\\/
acl videocache_allow_url url_regex -i \\.msn\\.com\\.edgesuite\\.net\\/(.*)\\.flv
acl videocache_allow_dom dstdomain v.mccont.com vp.video.google.com dl.redtube.com
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid
which is shown from 'http://cachevideos.com/install-videocache-pfsense' with:
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 7
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.googlevideo\\.com\\/videoplayback \\.googlevideo\\.com\\/videoplay \\.googlevideo\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.google\\.com\\/videoplayback \\.google\\.com\\/videoplay \\.google\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.google\\.[a-z][a-z]\\/videoplayback \\.google\\.[a-z][a-z]\\/videoplay \\.google\\.[a-z][a-z]\\/get_video\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplayback\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplay\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/get_video\\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\\-][a-z0-9][a-z0-9][a-z0-9]?\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i vid\\.akm\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\\.xtube\\.com\\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\\.vimeo\\.com\\/vimeo\\/videos\\/
acl videocache_allow_url url_regex -i va\\.wrzuta\\.pl\\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \\.files\\.youporn\\.com\\/(.*)\\/flv\\/
acl videocache_allow_url url_regex -i \\.msn\\.com\\.edgesuite\\.net\\/(.*)\\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.tube8\\.com\\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.tube8\\.com\\/
acl videocache_allow_url url_regex -i \\.mais\\.uol\\.com\\.br\\/(.*)\\.flv
acl videocache_allow_url url_regex -i \\.video[a-z0-9]?[a-z0-9]?\\.blip\\.tv\\/(.*)\\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\\.break\\.com\\/(.*)\\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid
from 'http://cachevideos.com/installation' which seemed more up to date.
Restart squid after the change.
While solving the above I found that any slight misconfiguration will cause videocache to run many instance of 'python' and fully utilizing the CPU. My '/var/log/videocache/videocache.log' is filled with over 100MB of repeating lines of 'RELOAD - videocache plugin was reloaded' when it happened.
At the moment everything seems fine but I've only been running it for a few hours. Will report back if anything arises.
Tekkon,
Thats cool! Congrats for the setup and enjoy caching!
Thank You!
Oopss.. no good. In less than 4 hours, the system began to run many instance of python again:
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
1085 proxy 2 97 0 11048K 7464K RUN 17:41 6.15% python
4402 proxy 1 97 0 10664K 7384K RUN 16:34 5.86% python
1087 proxy 2 97 0 11048K 7464K RUN 17:37 5.76% python
1084 proxy 2 97 0 11048K 7464K RUN 17:36 5.57% python
4401 proxy 1 97 0 10664K 7384K RUN 16:42 5.57% python
1086 proxy 2 97 0 11048K 7464K RUN 17:38 5.37% python
1082 proxy 2 97 0 11048K 7464K RUN 17:34 5.27% python
4399 proxy 1 97 0 10664K 7384K RUN 16:36 5.18% python
1083 proxy 2 97 0 11048K 7464K RUN 17:41 4.98% python
4397 proxy 1 97 0 10664K 7384K RUN 16:38 4.98% python
4400 proxy 1 97 0 10664K 7384K RUN 16:30 4.88% python
4398 proxy 1 97 0 10664K 7384K RUN 16:36 4.79% python
4403 proxy 1 97 0 10664K 7384K RUN 16:34 4.59% python
2027 proxy 1 116 20 10664K 7384K RUN 9:24 3.27% python
2026 proxy 1 116 20 10664K 7384K RUN 9:24 3.17% python
2023 proxy 1 116 20 10664K 7384K RUN 9:23 3.17% python
1081 proxy 4 96 0 11048K 7540K RUN 11:22 2.98% python
2022 proxy 1 116 20 10664K 7384K RUN 9:26 2.88% python
2025 proxy 1 116 20 10664K 7384K RUN 9:25 2.78% python
2028 proxy 1 116 20 10664K 7384K RUN 9:26 2.69% python
2024 proxy 1 116 20 10664K 7384K RUN 9:25 2.69% python
Here are 21 intense of python. So I guess my setup is still buggy.
I pasted my /var/log/videocache/videocache.log here - http://pastebin.com/m27d0dadc
and /var/squid/log/cache.log here - http://pastebin.com/m57361dd5
for analyzing.
Following the timestamps of the above log files, video cache was working nicely from around 05:53 after a restart. Then, until 09:23 the crash started.
Up until now my system has been running multiple python instance choking the CPU. At this moment ps -aux | grep python shows - http://pastebin.com/d49b7d3e0 - 28 instance of python.
My url_rewrite_children is set to 7. I think i'm seeing the running python processes in multiples of 7. Don't know if that mean anything. Any suggestions?
Tekkon,
To cache a video, videocache forks a daemon which is a python process. The default limit of parallel forking is 30. See max_parallel_downloads in /etc/videocache.conf for more details. Set it to a lower value like 10 or something.
Thank You!
Thanks for the suggestion.
After days of testing my setup, I still find it buggy. From time to time the system will spawn many instants of 'dnsserver' and 'python'. It'll consume most of the remaining RAM and all CPU cycles, slowing down the system.
It always happen during system startup and I have to use 'pkill python' many times until the python process will stop consuming all resources. After that, the system works normally caching videos.
Other time when it happens are mostly after squid is restarted. I have set max_parallel_downloads to 12 few days ago but the results are still the same. I have also tried 'update-vc' but it also didn't helped.
During the time of the anomaly,
in my '/etc/squid/log/cache.log' I find this:
2009/04/10 01:41:03| Ready to serve requests.
2009/04/10 01:41:04| Store rebuilding is 6.0% complete
2009/04/10 01:41:06| WARNING: url_rewriter #4 (FD 47) exited
2009/04/10 01:41:06| WARNING: url_rewriter #5 (FD 48) exited
2009/04/10 01:41:06| WARNING: url_rewriter #1 (FD 44) exited
2009/04/10 01:41:06| WARNING: url_rewriter #6 (FD 49) exited
2009/04/10 01:41:06| Too few url_rewriter processes are running
FATAL: The url_rewriter helpers are crashing too rapidly, need help!
Squid Cache (Version 2.6.STABLE21): Terminated abnormally.
and in /var/spool/videocache.log I'll get repeating lines of:
2009-04-06 21:49:59,001 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,003 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,006 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,007 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,009 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,010 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,012 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,014 1108 - - RELOAD - videocache plugin was reloaded.
2009-04-06 21:49:59,016 1108 - - RELOAD - videocache plugin was reloaded.
the 'RELOAD - videocache plugin was reloaded' line are generated at almost 1000 lines per second, causing my videocache.log to grow in megabytes in minutes.
Because these are the only two noticeable error messages I could find. I can't really tell what is going on. Besides the 100% CPU consumption and using "pkill python" from time to time. The video caching is working ok. Can you help me figure out what's wrong?
Tekkon,
Open file /usr/share/videocache/videocache.py in any editor and do a search for line containing 'os.kill('. Comment all these lines or replace with a pass statement.
See if there are changes in behavior.
Thank You!
Updated Guide
It took me hours and now days to finally get to a working install. I have created a guide for installing on pfSense 1.2.2 that worked for me and currently is running with low CPU usage and is properly caching videos. It even works on iPhone YouTube videos. Sorry for the bad wiki formatting. I'll figure out how to fix the broken numbered list later.
LTitus,
Awesome tutorial man!!! Linking it in the "Videocache Hacks box" on the left sidebar. Keep up the good work!!!
Thank You!
Thank you for the suggestion.
I tried commenting out the lines but gotten a syntax error in '/var/squid/log/cache.log' I used 'pass' instead. However the behavior is still the same.
2009/04/10 08:41:52| WARNING: url_rewriter #7 (FD 44) exited
2009/04/10 08:41:52| WARNING: url_rewriter #6 (FD 43) exited
2009/04/10 08:41:52| WARNING: url_rewriter #3 (FD 40) exited
2009/04/10 08:41:52| WARNING: url_rewriter #5 (FD 42) exited
2009/04/10 08:41:52| Too few url_rewriter processes are running
2009/04/10 08:41:52| Starting new helpers
2009/04/10 08:41:52| helperOpenServers: Starting 7 'python' processes
2009/04/10 08:41:53| WARNING: url_rewriter #4 (FD 41) exited
2009/04/10 08:41:53| WARNING: url_rewriter #2 (FD 39) exited
2009/04/10 08:41:53| WARNING: url_rewriter #1 (FD 38) exited
2009/04/10 08:41:53| WARNING: url_rewriter #1 (FD 40) exited
2009/04/10 08:41:54| WARNING: url_rewriter #2 (FD 43) exited
2009/04/10 08:41:54| WARNING: url_rewriter #3 (FD 44) exited
2009/04/10 08:41:54| WARNING: url_rewriter #4 (FD 66) exited
2009/04/10 08:41:54| Too few url_rewriter processes are running
2009/04/10 08:41:54| storeDirWriteCleanLogs: Starting...
2009/04/10 08:41:54| 65536 entries written so far.
2009/04/10 08:41:54| Finished. Wrote 72108 entries.
2009/04/10 08:41:54| Took 0.3 seconds (287436.3 entries/sec).
FATAL: The url_rewriter helpers are crashing too rapidly, need help!
Squid Cache (Version 2.6.STABLE21): Terminated abnormally.
2009-04-10 08:41:52,427 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,429 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,430 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,434 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,436 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,439 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,440 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,442 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,443 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,445 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,446 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,448 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:41:52,450 25631 - - RELOAD - videocache plugin was reloaded.
2009-04-10 08:43:27,187 26960 - - XMLRPCSERVER - Starting XMLRPCServer on port 9100.
2009-04-10 08:43:27,194 26960 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:27,201 26961 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:27,928 26963 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:28,354 27622 - - UPDATE_SIZE - Size of all caching directories updated successfully.
2009-04-10 08:43:28,585 26956 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:28,904 26959 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:29,394 26957 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:32,139 26955 - - SCHEDULEDER - Download Scheduler starting.
2009-04-10 08:43:32,143 26955 10.11.12.31 - REQUEST - http://v23.lscache1.googlevideo.com/videoplayback?id=226ed6d409c39a00&i
tag=34&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag&ip=60.52.125.176&signature=CBAF523E03590F813C504CF564EA22382E5CA124.249689606
ABA90EA65541031220078EE8BE7EE5A&sver=3&expire=1239345793&key=yt1&ipbits=0
2009-04-10 08:43:32,151 26955 10.11.12.31 226ed6d409c39a00 URL_HIT YOUTUBE http://v23.lscache1.googlevideo.com/videoplayback?
id=226ed6d409c39a00&itag=34&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag&ip=60.52.125.176&signature=CBAF523E03590F813C504CF564EA2
2382E5CA124.249689606ABA90EA65541031220078EE8BE7EE5A&sver=3&expire=1239345793&key=yt1&ipbits=0
2009-04-10 08:43:32,153 26955 10.11.12.31 226ed6d409c39a00 CACHE_HIT YOUTUBE Video was served from cache.
2009-04-10 08:43:32,155 26955 10.11.12.31 226ed6d409c39a00 NEW_URL YOUTUBE 303:http://10.11.12.13/videocache/youtube/226ed6d4
09c39a00?id=226ed6d409c39a00&itag=34&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag&ip=60.52.125.176&signature=CBAF523E03590F813C50
4CF564EA22382E5CA124.249689606ABA90EA65541031220078EE8BE7EE5A&sver=3&expire=1239345793&key=yt1&ipbits=0
I used 'pkill python' again around 2009-04-10 08:43:00 to stop the high CPU utilization back to normal. When videocache is working normally, total system CPU utilization is usually around 20%.
Kulbir Saini,
After sometime I still have the same issues, many python instances until I run pkill python.
I tried editing /usr/share/videocache/videocache.py according to your instructions but the problem still persists.
I found that it only happens when squid loads during startup or restarts. Even though during the time of the bug when many python instance are eating away the CPU cycle, videocache is actually still caching video normally.
I've tried applying chown -R 777 to /var/log/videocache and /var/spool/videocache but it didn't helped.
LTitus210,
You mentioned in your wiki guide that /var/log/videocache/videocache.log is filling up to over 200MB in 5 minutes.
Whats the filling up with? Is it normal video caching log or are you getting any errors like 'RELOAD - videocache plugin was reloaded', similar to what I am getting?
Wait, it seems the chown -R 777 to /var/log/videocache and /var/spool/videocache is working.
I have restarted squid once and the stuck python processes didn't happen. Usually the python process will be stuck every midnight when lightsquid refreshes it's reports. This morning I was surprised to find the CPU cycle consumption normal.
I'll monitor for a few days to confirm that the issue is gone.
Just wanting to add that this doesnt work. Atleast not on 1.2.3 and since I basicly did the same before with 1.2.2 I doubt its working there either. Nothing can be seen with http://ip/videocache
Just wanting to add that this doesnt work. Atleast not on 1.2.3 and since I basicly did the same before with 1.2.2 I doubt its working there either. Nothing can be seen with http://ip/videocache
After a reboot it seems to be working. The problem I mentioned before involves lighthttpd settings. It doesnt allow any directories to be listed (so you can access the videos but you cant list them )
Hi every body.
I had installed PFsense and try to configure videocache for 2 day but it still don't work T_T
first, I try on this link "http://cachevideos.com/install-videocache-pfsense" but system show me (system->service) that squid had been stoped
One again I try follow this link "http://wiki.ecnet.org/wiki/PfSenseVideoCache".
After finish system show me that squid still start until I had reboot Pfsense then system show me that squid had stop again
Notice that I use pfsense 1.2.3 RC1 and somehow squidguard didn't work property (I can access my black list's web).
I think I setup squidguard correctly.
Does my problem was squidguard ?
PS. I had no exp. on proxy server (actually, I had used Pfsense since 2 week ago)
PS2. Sorry for my bad english.
Login to ftp.freeBSD.org using ftp (username = anonymous, password=email@address.com)
cd to pub/FreeBSD/ports/i386/packages-7-stable/Latest
then use:
get python26.tbz
get py26-bsddb.tbz
etc...
quit
then
pkg_add python26.tbz
pkg_add py26-bsddb.tbz
etc...
but there are days that I feel should not be on me all the commands:
$ pkg_add -r py25-gdbm
Error: FTP Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6.2-release/Latest/py25-gdbm.tbz: File unavailable (e.g., file not found, no access)
utilization pfsense:
1.2-RELEASE
built on Sun Feb 24 17:04:58 EST 2008
as I do. updated the topic. Thank you.
Hi,
I installed python and libraries using
"pkg_add -r ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-9-current/Latest/[FILE] {
File= python27,py27-bsddb,py27-gdbm,py27-sqlite3,py27-tkinter}
Then i type fetch command for URLGrabber from "http://urlgrabber.baseurl.org/download/urlgrabber-3.9.1.tar.gz"
Extact the .tzr, get into the file and when executed
"# python setup.py bdist_rpm" i get the following message:
/libexec/ld-elf.so.1: Shared object "libutil.so.9" not found, required by "python"
what im doing wrong?!?! i really want to try this.
Hi Anon,
Please mention the videocache version you are trying to install. Also, urlgrabber is not required to operate videocache any more. The install instructions are simple and can be viewed at installation page.
Thank You!
I have Pfsense and Im Following the steps from here:
http://doc.pfsense.org/index.php/Setup_VideoCache_with_Squid
I reinstalled everything again and im having the same problem
python
bsddb databases/py-bsddb
gdbm databases/py-gdbm
sqlite3 databases/py-sqlite3
tkinter x11-toolkits/py-tkinter
when i install the urlgraver and i do the:
python setup.py bdist_rpm
i get this message:
# python setup.py build
Traceback (most recent call last):
File "setup.py", line 3, in <module>
import urlgrabber as _urlgrabber
File "/root/urlgrabber-3.9.1/urlgrabber/"init".py", line 54, in <module>
from grabber import urlgrab, urlopen, urlread
File "/root/urlgrabber-3.9.1/urlgrabber/grabber.py", line 427, in <module>
import pycurl
ImportError: No module named pycurl
what im doing wrong?
Hi!
You didn't mention the videocache version you are trying to install. Please mention that.
Thank You!