You are hereForums / Bugs / Videoache will download the same youtube movie twice

Videocache 1.9.5 is available now. This version improves support for logging and fixes video caching problems for various sites like Vimeo, Break, Tube8 etc due to change in URL patterns. Also, from this version onwards, UrlGrabber has been removed from dependencies. Check Changelog for details. If you have purchased version 1.9.x previously, you can claim free upgrade by contacting using Contact Link. Please mention the email address using which you purchased Videocache. New users proceed to download page right now, secure your copy and enjoy caching :-)

Videoache will download the same youtube movie twice


13 replies [Last post]
Offline
Joined: Dec 14 2008
Videoache will download the same youtube movie twice
Printer-friendly versionSend to friend

Hi,

Due to a recent change in youtube, the video_id in the first request (to the get_video script), and the videoid in the resulting redirect are different, which makes videocache download them both. I suggest changing the recommended regex in squid.conf to reject google's cache servers.

admin's picture
Offline
Joined: Nov 2 2008

Imriz,

Can you please cite an example. I didn't really get this.

Thank You!

Offline
Joined: Dec 14 2008

Consider the following get:
--13:29:54-- http://www.youtube.com/get_video?video_id=eMvDax3AuP4&t=vjVQa1PpcFN_VmUB...
Connecting to www.youtube.com|208.65.153.238|:80... connected.
HTTP request sent, awaiting response... 303 See Other
Location: http://v6.cache.googlevideo.com/videoplayback?id=78cbc36b1dc0b8fe&itag=3... [following]
--13:29:55-- http://v6.cache.googlevideo.com/videoplayback?id=78cbc36b1dc0b8fe&itag=3...
Resolving v6.cache.googlevideo.com... 74.125.99.223
Connecting to v6.cache.googlevideo.com|74.125.99.223|:80... connected.

Notice that the video id in the get_video request and in the videoplayback request are different. videocache will try to download them as seperated items.

admin's picture
Offline
Joined: Nov 2 2008

Imriz,

Ok. Then we should be caching the videos only from youtube and deny the videos from googlevideo servers. Can you try once that in your setup? Just deny them in squid.conf and they'll never reach videocache for download.

But here I am confused about the choice. We should deny youtube ones or the googlevideo ones. Whats your insight on this?

Thank You!

Offline
Joined: Dec 14 2008

Hi,

With transparent proxy in mind - I would suggest denying the googlevideo ones.

Offline
Joined: Dec 18 2008

Kulbir, Imriz, ...

I wrote a page about ... after a lot of testing ...

http://www.bellera.cat/josep/videocache/squid_videocache_youtube.html

Regards,

Josep Pujadas

Gandi (not verified)

Great stuff.

After a lot of testing i menaged to force youtube caching to work well.

This is my config:
# --BEGIN-- videocache config for squid
#url_rewrite_program /usr/bin/python /usr/local/videocache/videocache.py
url_rewrite_program /usr/local/squid/bin/zapchain "/usr/local/squid/bin/gg_rewrite" "/usr/bin/python /usr/local/videocache/videocache.py"
url_rewrite_children 5

acl videocache_allow_url url_regex -i www\.youtube\.com\/get_video\?
#acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/get_video\?
#acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/get_video\?
#acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/get_video\?
#acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\
#acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\
#acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
#acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
#acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
#acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
#acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
#acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
#acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
#acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
#acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
#acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)

#acl videocache_allow_dom dstdomain v.mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_allow_dom dstdomain dl.redtube.com

#acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com

url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
url_rewrite_access allow GG_banner

redirector_bypass on
# --END-- videocache config for squid

As You see i commented all googlevideo acls, and not used ones.
I changed youtube acl to match www.youtube.com

After that Youtube url's are requested only once but i had to change hit_threshold to 3 because some youtube url's reply with "we're sorry this video is no longer available" and users usually click refresh to fix this what causes another request and unnecessary caching. It happened before so it's not videocache issue.

Big thanks for this stuff. I searched for something like this for a very long time.

Gandi (not verified)

I found another bug.
When user tries to seek through youtube or redtube video, it makes another request.

Had to modify config to something like this:
[...]
acl videocache_allow_url url_regex -i www\.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i dl\.redtube\.com\/(.*)\.flv\?start=0

acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com www\.youtube\.com\/get_video\?video_id=.{11}&(start|begin)=
[...]

admin's picture
Offline
Joined: Nov 2 2008

Gandi,

Thats not the case actually. When user seeks, the video is requested again by the client. If the video is in queue (was requested previously), re-requesting will just increase its priority and will not force another download. So, it can be ignored.

Thank You!

Gandi (not verified)

Hi again!
I realize that it will not force another download, but it requests video another time.

Let's see example :)

hit_threshold = 2

User is starting to watch youtube video - it is first request.
User seeks through video - second request and videocache is starting download.
It's not desireble .. i think.
One user can start caching video just by seeking through file.

Sorry for off topic.

admin's picture
Offline
Joined: Nov 2 2008

Gandi,

Your argument is valid but the regex you proposed will create problems seeking in videos which have been cached by videocache. Moreover, when one sets some value for hit_threshold, I think it may be fine to go with +1 or -1.

Thank You!

Gandi (not verified)

Hi again!

You are right, it makes problems seeking in cached videos :( but theye are served from cache and loaded to browser vary fast.
Is there a possibility to add a feauture to avoid getting requests from single IP in particular time? I think it should fix it.

Big thanks again!

admin's picture
Offline
Joined: Nov 2 2008

Gandi,

Thats actually difficult to implement keeping in mind the complexity of the usage type. Imagine a cascading proxy case. The time limit will be an overkill.

Thank You!

admin's picture
Offline
Joined: Nov 2 2008

Gandi,

Thanks for the compliments and sharing your experience with videocache.

Keep caching :D

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <code>
  • Lines and paragraphs break automatically.
  • HTML tags will be transformed to conform to HTML standards.

More information about formatting options

CAPTCHA
Are you a human visitor ?
Image CAPTCHA
Enter the characters (without spaces) shown in the image.

Buy Now

Videocache Bundle (RPM and tar archive) is available. Secure your copy now by clicking the button below.
No. of Servers?
For details, check this page.

Recent comments

Spread it

Thank you for spreading the word!

Who's online

There are currently 0 users and 90 guests online.