[Resolved] multiple url_rewrite_program
by Anonymous on 17 Dec 2008Hello!
squid supports only one url_rewrite_program
adzap (addzapper) is a redirector program for squid that can chain many redirector programs.
Is it possible to use squid + adzap + squidguard + videocache ?
Here you have an article using two redirectors:
http://www.kernel-panic.it/openbsd/proxy/proxy6.html
I post this question because I have squid + squidguard working and I would like to add videocache to my installation.
Regards,
Josep Pujadas
20 Answers
Hi Josep,
First of all thanks a lot for considering videocache. I think videocache should work perfectly with the zapchain program because it just another url rewrite program. Install videocache from rpm or source.
Now try this
Change the permission of /usr/share/videocache/videocache.py to executable
chmod +x /usr/share/videocache/videocache.py
Now modify the /usr/local/bin/wrapzap so that it looks as follows (Only two lines need to changed)
#!/bin/sh
squidclamav=/usr/local/bin/squidclamav
zapper=/usr/local/bin/squid_redirect
# NOTICE THE CHANGE -- CHANGE No. 1
videocache=/usr/share/videocache/videocache.py
# Setting ZAP_MODE to "CLEAR" will cause the zapper to use transparent images,
# thus completely hiding ads. This may, however, hide useful markup.
ZAP_MODE=
# Base URL of the directory containing the replacement images
ZAP_BASE=http://www.kernel-panic.it/icons/zaps
ZAP_BASE_SSL=https://www.kernel-panic.it/icons/zaps
# The following variables contain the path to extra pattern files.
# ZAP_PREMATCH patterns are consulted before the main pattern list. Use it to
# prevent overzapping by some erroneous patterns in the main pattern file.
ZAP_PREMATCH=
# ZAP_POSTMATCH patterns are consulted after the main pattern list. Use it to
# add extra patterns
ZAP_POSTMATCH=
# ZAP_MATCH patterns are consulted instead of the main pattern list. Use it to
# fully customize AdZapper
ZAP_MATCH=
# Should you use Apache2 instead of Squid, set this to "NULL"
ZAP_NO_CHANGE=
# Placeholder images names. "Clear" versions have "-clear" appended to the root
# portion of the file name; e.g. "ad.gif" becomes "ad-clear.gif".
STUBURL_AD=$ZAP_BASE/ad.gif
STUBURL_ADSSL=$ZAP_BASE_SSL/ad.gif
STUBURL_ADBG=$ZAP_BASE/adbg.gif
STUBURL_ADJS=$ZAP_BASE/no-op.js
STUBURL_ADJSTEXT=
STUBURL_ADHTML=$ZAP_BASE/no-op.html
STUBURL_ADHTMLTEXT=
STUBURL_ADMP3=$ZAP_BASE/ad.mp3
STUBURL_ADPOPUP=$ZAP_BASE/closepopup.html
STUBURL_ADSWF=$ZAP_BASE/ad.swf
STUBURL_COUNTER=$ZAP_BASE/counter.gif
STUBURL_COUNTERJS=$ZAP_BASE/no-op-counter.js
STUBURL_COUNTERHTML=$ZAP_BASE/no-op-counter.html
STUBURL_WEBBUG=$ZAP_BASE/webbug.gif
STUBURL_WEBBUGJS=$ZAP_BASE/webbug.js
STUBURL_WEBBUGHTML=$ZAP_BASE/webbug.html
# Set this to "1" to use the rewrite facility to get the printer-friendly
# version of some pages
STUBURL_PRINT=
export ZAP_MODE ZAP_BASE ZAP_BASE_SSL ZAP_PREMATCH ZAP_POSTMATCH ZAP_MATCH ZAP_NO_CHANGE
export STUBURL_AD STUBURL_ADSSL STUBURL_ADJS STUBURL_ADHTML STUBURL_ADMP3 \\
STUBURL_ADPOPUP STUBURL_ADSWF STUBURL_COUNTER STUBURL_COUNTERJS \\
STUBURL_COUNTERHTML STUBURL_WEBBUG STUBURL_WEBBUGJS STUBURL_WEBBUGHTML \\
STUBURL_PRINT STUBURL_ADHTMLTEXT STUBURL_ADJSTEXT
# Exec the real zapper (chained with SquidClamav)
#exec /usr/local/bin/zapchain "$zapper" "$squidclamav"
# NOTICE THE CHANGE -- CHANGE NO. 2
exec /usr/local/bin/zapchain "$zapper" "$squidclamav" "$videocache"
You don't need to make changes anywhere else. Please try and let us know if it works :)
Ok, thanks! & one more question about ...
At squid.conf you allow the access to videocache for URL sites containing videos and bypass the url_rewrite_program for the other sites.
My situation is that I want to test all the URL with squidguard (first) and use videocache (second). It is for this that I want to chain more than one redirector.
In my case I think can't use acl at squid.conf to limit the videocache use ...
I'm wondering if adzap (addzapper) can manage this situation or it would be better to integrate other redirectors (like squidguard) with videocache.
Regards,
Josep Pujadas
Hi Bellera!
Allow all urls to be passed through wrapzap. Videocache will not do anything for urls that are not for video sites. Actually the urls passed to videocache are again checked against regular expression, so videocache itself can take care of the url filtering.
So you can go ahead with the configuration.
Thank You.
Hello! I'm working, but only /var/spool/videocache/tmp is used. So, I think I'm not caching the videos. I made the following change to my squid.conf:
url_rewrite_program /usr/local/libexec/zapchain "/usr/local/bin/squidGuard -c /usr/local/etc/squid/squidGuard.conf" "/usr/local/bin/python2.6 /usr/share/videocache/videocache.py"
zapchain is just a perl script to run many redirectors with squid.
squidguard and videocache are working together.
Any idea?
Thanks in advance,
Josep Pujadas
Hi!
The line in squid.conf should be
url_rewrite_program /usr/local/libexec/zapchain "/usr/local/bin/squidGuard -c /usr/local/etc/squid/squidGuard.conf"
Hello!
I can't see anything of your posted code.
More ...
Files at /var/spool/videocache/tmp are squid:squid owned with rw------- permissions and they have 0 bytes. Is it normal?
Apache process is owned by www user on FreeBSD, not the squid user. So, If I go to http://my_box/videoache I can browse the folders but I got access denied for the flv files.
Regards,
Josep Pujadas
Hum ...
You said to put (finally I could read with FireFox, with IExplorer not):
url_rewrite_program /usr/local/libexec/zapchain "/usr/local/bin/squidGuard -c /usr/local/etc/squid/squidGuard.conf"
Why do you ask to me to use only one redirector?
The two redirectors are working, squidguard and videocache. I can see the processes working with ps command and the logs for squid, squidguard and videocache seem to be good.
Regards,
Josep Pujadas
Hi Josep,
We want all the files to be owned by squid:squid so that squid process aka videocache can write to those files. But the files should have permissions rw-r--r-- and not rw------- . Make all of them 644 and Apache will serve the files happily.
Thank You.
Hi Josep,
I don't know when all browsers will start behaving the same way :( Anyways are the redirectors or rewriters working fine now?
Thank You.
Hi Kulbir,
I don't know how to change the videocache behavior for having tmp files at 644.
Thanks,
Josep Pujadas
Hello Josep,
Videocache by default should store files as 644. You change the permissions for existing files and videocache should take care of the rest.
Thank You.
Hello Kulbir,
I'm sorry but I don't understand your answer.
flv files at /var/spool/videocache/tmp are created automatically. You said files should be created with 644 permisions. But they are created with 600 permisions and they have 0 length (?). I can't go every time to change manually the permisions ...
On the other hand I saw at /var/log/videocache/videocache.log the following messages:
mutb7KgA9NM SCHEDULED YOUTUBE Video scheduled for download.
mutb7KgA9NM DOWNLOAD_ERR YOUTUBE An error occured while retrieving the video.
Is there any way to know the cause of the error?
I tried to change os.umask(077) to os.umask(0) but same error.
Thanks,
Josep Pujadas
Hi Josep,
Check your mail. I have sent you a modified version of videocache. See if it works.
Thank You.
Hi Kulbir,
Not needed. SOLVED! I'm caching!
I documented all my steps at:
http://www.bellera.cat/josep/videocache
Thank you very much!
Josep Pujadas
Hi Kulbir,
SOLVED! I'm caching!
I documented all my steps at:
http://www.bellera.cat/josep/videocache
Thank you very much!
Josep Pujadas
Hi Josep,
A big thanks to you for documenting the entire process :) Notice a link to your page in Hacks section in the left sidebar :)
Thank you for using videocache :)
hi admin ,
i am using squidguard for blocking porn site and banner.
in my server instal squid + squidguard and videocache.
i am using Centos 4.7 server.
My Problem is :
squidguard with videocache is not working ( not block porn site ).
when i use only squidguard without videocache , squidguard is working ( porn site was blocked )
in my squid.conf i add this script :
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/local/bin/zapchain "/usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf" "/usr/local/bin/python2.6 /usr/share/videocache/videocache.py"
url_rewrite_children 10
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.googlevideo\\.com\\/videoplayback \\.googlevideo\\.com\\/videoplay \\.googlevideo\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.google\\.com\\/videoplayback \\.google\\.com\\/videoplay \\.google\\.com\\/get_video\\?
acl videocache_allow_url url_regex -i \\.google\\.[a-z][a-z]\\/videoplayback \\.google\\.[a-z][a-z]\\/videoplay \\.google\\.[a-z][a-z]\\/get_video\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplayback\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplay\\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/get_video\\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\\-][a-z0-9][a-z0-9][a-z0-9]?\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i vid\\.akm\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\\.xtube\\.com\\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\\.vimeo\\.com\\/vimeo\\/videos\\/
acl videocache_allow_url url_regex -i va\\.wrzuta\\.pl\\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \\.files\\.youporn\\.com\\/(.*)\\/flv\\/
acl videocache_allow_url url_regex -i \\.msn\\.com\\.edgesuite\\.net\\/(.*)\\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.tube8\\.com\\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.tube8\\.com\\/
acl videocache_allow_url url_regex -i \\.mais\\.uol\\.com\\.br\\/(.*)\\.flv
acl videocache_allow_url url_regex -i \\.video[a-z0-9]?[a-z0-9]?\\.blip\\.tv\\/(.*)\\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\\.break\\.com\\/(.*)\\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid
Tell me what problem sir ?
thanks before
hi josep , i am not working. i use script same with u.
Why ? squidguard not blok porn site if running with vidoecache.
without videocache , squidguard is working .
help me !!!!
finally is WORKING . sorry....location of script was wrong.
Squid + squidguard + videocache So nice....Thanks For ALL
Hi admin,
I want to run two redirectors with squid named squidGuard and custom cpp executable.
Can you please tell me how to do this using zapchain?
Alone my custom executable is working with zapchain, but alone squidGuard is not working, so combination is not working!
Best Regards,
Sachin Patil.