when using your youtube cache, it is double download?
by Anonymous on 21 Nov 2008for example, i mean when i watching youtube, the browser will buffering/downloading, + youtube cache, also do another downloading the same video? that mean if the video is 10MB that mean browser download 10mb, youtube cache also do another download 10MB =total 20MB. that mean we r wasting download bandwidth?
34 Answers
Yes. Youtube Cache is designed for shared environments where popular videos are watched time and again by several clients. In single user mode, its an overhead if a video is watched only once. For example if I have 5-10 clients on my network or its a home network with 3-4 machines and I watch a video and forward the link to my parents/grandma/children and they watch the same video, then the bandwidth is saved.
PS : I am looking for some way to use the squid stream so that redundant downloading can be controlled. I'll update if I find some way of doing it.
thx Kulbir Saini, hope u will update using the squid stream soon :)
can we control the youtubecache downloader download tru aother dsl line? so the speed of the squid cache server will not affected?
Thank you too :) Yes. You can very easily control youtube cache download stream. Setup a very basic proxy server on another machine with a dsl line. And use the proxy option in youtube_cache.conf file. Specify proxy as
proxy = http://ip_of_another_machine:3128/
Enjoy !!
any word on this problem/issue... as we want to start using it in our office, but can't really afford the overhead and running a second proxy just to limit bandwidth consumption doesnt remove the fact it basically instantly doubles the traffic so basically if you save 100mb you have to calculate in that it might have used 90 megs or 300megs just downloading videos....
Also has anyone used this ... on a large network say i dont know 60mbit of youtube traffic ?
If he used store url rewrites as well, he could cache the whole thing in one hit. Not sure if he will get to it but I know for a fact it can be done. He should talk to Adri dev of cacheboy. It's not that hard to do.
Hi!
I am not using store url rewrite because its not available for 3.x and people will not be able to use videocache with squid 3.x
Thank You!
Really this question of the consumption of band of the VideoCache is preoccupying, same with the videos configured for only 1 schedule the consumed band the times is great. I thought about making a limit QOS for door 9100, plus this would go to also limit the band for the users who to have access the videos that are in cache.
Hi Josue,
9100 port doesn't need any QOS because its used for XMLRPC communication and that too locally on the machine. To limit bandwidth consumption by videocache you can use delay pools for the proxy machine itself in the squid.conf file. That would limit the bandwidth used by the videocache but will not affect the bandwidth for the videos being served from the cache to the users.
Thank you for using videocache :)
Very good, I read something on the Delay Pools, more never used in the practical one, use HTB and find very good. I go to search on the subject and rank again. Thanks.
Hi,
im using fedora 9 with squid 2.6 stable13. i install youtube cache & its working fine. but some time when i open youtube any new video its not cache when i repeat that video 3 times or much but its not caching. i repeat this by refresh page. any idea where im doin wrong. and in /etc/videocache.conf file i see a line where No of times a video should be requested before we start caching it.
hit_threshold = 2 can i convert into 1 ? means when video play 1 time its save in cache.
Thankx
Yeah. You can change that to one. There is no harm in doing that.
Thank you for using videocache.
thank you people for this great job , but i am still facing 2 problems, first am using a parent proxy server and when files are cached they can't be played again cause the redirection is using my parent proxy ip and this is a big problem , second am losing bandwidth and don't know how to enable pool for videocache service. any help would be highly appreciated.
Hi Voltaire,
Solution 1 : You can mount the cache directories from your parent proxy over nfs or smb to the child proxy and can run videocache on child proxy with those mounted directories as cache directories.
Solution 2 : There is an option called max parallel downloads in /etc/videocache.conf. Use that to control the bandwidth used for videocache.
Thank You!
the problem , is that when videocache redirect the http request it appends the proxy parent ip , and here is the problem , it saves the file but when it is requested again , it can't be played again and i get a message from youtube that this file doesn't exist again .
I don't know what it is the problem , i will provide you with the part of my squid.conf :
cache_peer x.x.x.x parent 8080 0 proxy-only no-digest
the videocache saves the file but if we look at the http request we find the ip address of the parent proxy server appended instead of my local squid server , and the parent proxy server is in the usa still my server is here in lebanon , everything is working fine but i can't get the videocache to play it again since it finds it in cache but the link has the parent proxy server ip and here is the problem.
Thank you for your help again.
Voltaire,
I think I have found your problem. Please check the option cache_host in your /etc/videocache.conf file. Change it to the ip of the child proxy server :) Let me know if the problem persists.
Thank you for using videocache :)
it is the same problem whether you put the local or child proxy server ip or the parent one, it always uses the parent ip when redirecting the http request, now on the other hand , if we use the acl you defined in your modifications of squid.conf and we used the always_direct for these acl , then the file will be cached and played with no problem at all, bu still one missing thing is that not all sites or films are redirected to the local ip , still some links are using the parent server, i think that many people will face the same problem if they are using parent proxy or even if their ISP are transparently redirecting them to a proxy or cache machine.
Example of failed links:
http://lax-v181.lax.youtube.com/get_video?video_id=zxZDSOSCxJw&ip=xxx.xxx.xxx.xxx
here xxx.xxx.xxx.xxx is the parent proxy ip .
Voltaire,
I am not really understanding what kind of parent proxy you are using. Many people are using videocache with parent child squid proxies and nobody reported any problem with redirections. Can you paste your /etc/videocache.conf somewhere like http://pastebin.com ?
i will describe the following problem dear admin ,
the videocache.conf file contains:
enable_video_cache = 1
cache_host = 194.126.x.x
proxy = http://194.126.x.x:8080/
and the rest is untouched
everything was working great before using a parent proxy.
Then when i used a parent proxy
in the squid.conf file
cache_peer 74.x.x.x parent 8080 0 no-query default
the file was downloaded but when it is requested another time , here is the problem , the youtube page says the file doesn't exist anymore , because in the redirection the parent proxy ip which is 74.x.x. is appended instead of my local proxy ip 194.126.x.x , so i had the following lines in my videocache log file:
2009-02-09 00:29:09,402 8464 194.126.x.x - REQUEST - http://v2.cache.googlevideo.com/get_video?origin=lax-v140.lax.youtube.com&video_id=FjiXavtVcVA&ip=74.x.x.x®ion=0&si$
2009-02-09 00:29:09,401 8464 - - STRAT_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
2009-02-09 00:29:11,228 8467 - - STRAT_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
2009-02-09 00:29:15,214 8465 - - STRAT_XMLRPC_SERVER_ERR - Cannot start XMLRPC Server - Exiting
and so on , until i told the squid to go direct for these addresses and it worked again , i don't what is the problem or even if it is my fault , but i did not do any changes.
The question is , why does the videocache redirector append the parent proxy ip instead of mine ??
and why when i tell the squid to go direct to these sites it works fine ??
Best Regards;
Dear Voltaire,
Your videocache is not working as it is supposed to be. Please let me know that how many children program of videocache you are running on your server? I mean what is the value of url_rewrite_children in your squid.conf file? Try to keep it at 5 and then restart the squid using the following command,
[root@localhost ~]# killall -9 squid python; service squid start
If the problem persists, get back here. We'll be happy to help.
Thank you for your efforts :)
dear admin
the value is 5 and i did not change anything in the squid.conf modification that needs to be pasted in the squid.conf file.
Best Regards;
Voltaire,
Please contact me using the contact tab on top. I'll need further details.
Thank You!
Dear Admin ;
Thank you for you patience with me and support , it worked. It was my ISP , they were transparently redirecting me to different proxy servers, at a time , due to lack of bandwidth , it worked , thaks a lot
Take care and Best Regards;
Voltaire
Hi Josué,
You can control youtube bandwidth (proxy => web) without affecting the internal bandwidth (proxy => local network). For this, you can use their own ACLs created by Kulbir. For this you can use ACLs used by videocache.
For example:
# --BEGIN-- videocache config for squid
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 10
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video
acl videocache_allow_url url_regex -i \\.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.googlevideo\\.com\\/videoplayback
acl videocache_allow_url url_regex -i \\.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\\.googlevideo\\.com\\/get_video
acl videocache_allow_url url_regex -i proxy\\-[0-9][0-9]\\.dailymotion\\.com\\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\\.xtube\\.com\\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\\.vimeo\\.com\\/vimeo\\/videos\\/
acl videocache_allow_url url_regex -i va\\.wrzuta\\.pl\\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \\.files\\.youporn\\.com\\/(.*)\\/flv\\/
acl videocache_allow_url url_regex -i \\.msn\\.com\\.edgesuite\\.net\\/(.*)\\.flv
acl videocache_allow_dom dstdomain v.mccont.com vp.video.google.com dl.redtube.com
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
# --END-- videocache config for squid
# --BEGIN-- delay_pool for youtube bandwidth control
delay_pools 2
delay_class 1 2
delay_class 2 2
delay_parameters 1 12000/12000 10000/10000 400/4000
delay_parameters 2 24000/24000 10000/10000 600/6000
delay_access 1 allow videocache_allow_url
delay_access 1 allow videocache_allow_dom
# --END-- delay_pool for youtube bandwidth control
The parameter to be changed is the "10000/10000".
100 kbps Down
100 kbps UP
I'm Brazilian too. Call me via e-mail for the user contact area.
Rafael,
Dont you think that if you use those acls, then the users browsing those websites will also suffer from delay pools. I think you can fine tune it using an additional acl for local IP. I mean to say combine your delay pools configuration with the configuration by sirkike.
Thank for the hack :D
I'm doing 1 gbit of youtube traffic.
Kulbir, I hope I'm not out of line here, but I think that people at small offices, which cares about the "double downloading", should check out the possibility to directly cache youtube via squid.
It is discussed and explained here:
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
and here:
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussion
I have tried this configuration and it works.
I have managed to optimize this configuration to allow it to support the quality of the video, range requests and collapsed forwarding.
I have pushed around 600~ mbit/s with this solution.
The bottleneck of this configuration is the squid and the fact it cannot utilize SMP machines.
Imriz,
The other problem I find with that solution is that with limited cache size, squid start throwing out cached videos very quickly and the cached videos are lost.
Plus playing around with refresh patterns turns out to be a dangerous job for people who are not really skilled at using squid.
Thank You!
Well, limited cache size is a problem videocache will suffer from too. With the right replacement policy, squid will keep the most popular videos in cache. Again, the main advantage with that configuration is that there is no "double download". But yes - maintaining that configuration requires A LOT more knowledge with squid internals, and if performance, ease of use and scalability are important, videocache is the right solution (and this comes after testing both configurations under high load).
Getting rid of the XMLRPC server and moving to UDP based communication will increase the stability and performance of videocache
keep up the good work :)
Imriz,
You are absolutely right my friend. Let me share my experience here.
Once I was crazy for caching youtube videos on my system. I searched on Google like hell but the only successful approach I could find were the links you pointed. I was a newbie with squid and could never manage to get the refresh patterns work in a desired manner.
Frustrated from the fact that I couldn't make youtube caching work, I thought why not write a god damn plugin for myself :D And then I started working on youtube cache which later became videocache :D
Thank You!
Kulbir & others,
I wrote a page about ... after a lot of testing ...
http://www.bellera.cat/josep/videocache/squid_videocache_youtube.html
Regards,
Josep Pujadas
cache youtube content is a must.. more than 40% of all traffic on our network is googlevideo transfer.
I think it should be using internal squid storage to avoid double downloads, and proxy/apache/videocache config mess.
Double downloads are a nonsense when typical cache hits are 30% of all traffic.