youtube.com only REQUEST but not caching it
by Anonymous on 28 Jul 2009Hi Admin,
Till morning videocache was working fine but then its stop working as squid was not redirecting the youtube video url to it. after checking logs I found that video url has been change from http://v12.lscache5.googlevideo.com/ to http://v4.cache5.c.youtube.com/
after adding this below line in my squid.conf
acl videocache_allow_url url_regex -i \.c\.youtube\.com\/
But after squid is redirecting to videocache & Its only generating REQUEST in logs but it did not do any thing , like URL_HIT etc etc etc my log part is below.
same condition for the files which are already in cache of videocache.
thanks in advance.
my os is linux and distro is centos 5.3 + squid3.16 and it was running perfectly
TheOne Khan
2009-07-29 21:16:38,375 3749 10.101.10.1 - REQUEST - http://v4.cache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=D92DF83A04AAAE59C830D2BE333FBB56E9FBCDE6.B01ED2A0F7CA69D0C188F7BF50C74AF82C86E853&sver=3&expire=1248818400&key=yt1&factor=1.25&burst=40&id=80cb95dddfcdfeba&begin=68000&redirect_counter=1
2009-07-29 21:16:38,375 3749 10.101.10.3 - REQUEST - http://v14.cache6.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=D92DF83A04AAAE59C830D2BE333FBB56E9FBCDE6.B01ED2A0F7CA69D0C188F7BF50C74AF82C86E853&sver=3&expire=1248818400&key=yt1&factor=1.25&burst=40&id=80cb95dddfcdfeba&begin=68000&redirect_counter=1
19 Answers
Dear Sir,
i'm not that expert in videocache, i do the below steps and it works with me, i hope the admin can advice us also in squid.conf videocache section, i add the below line
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
in the /usr/share/videocache/videocache.py search for 'youtube' you will find lines like this
if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
comment it and change it to be
if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
video_id = dict['id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
i hope this solution works for you, currently it works for me :-)
For some requests still does not work.
Take a look:
2009-07-28 15:40:24,753 6274 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
I have this id 97feeb8478fe553b
# ls -l 97feeb8478fe553b
-rw-r--r-- 1 nobody nogroup 16953652 2009-07-27 14:40 97feeb8478fe553b
But, the VC not HIT in cache.
thanks for any help!
i copy and paste URL in my browse VC log said:
2009-07-29 00:46:44,224 10077 10.11.11.41 97feeb8478fe553b CACHE_MISS YOUTUBE Requested video was not found in cache.
2009-07-29 00:46:44,227 10077 10.11.11.41 97feeb8478fe553b NEW_URL YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:46:44,403 10085 10.11.11.41 97feeb8478fe553b SCHEDULED YOUTUBE Video scheduled for download.
2009-07-29 00:46:44,415 10077 192.168.113.140 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
after it complete download, it hit the cache as you can see
2009-07-29 00:50:19,652 10077 10.11.11.41 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b URL_HIT YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b CACHE_HIT YOUTUBE Video was served from cache.
is that happen with all videos ?? do you have any other acl related to youtube.com ?
Yes, in the order I have the following ACLs
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video\\?
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com
Remove the cached ID: 97feeb8478fe553b and the VC not download when I accessed again.
Thanks for your help!
could you send me the log generated by squid and videocache since request begin, also as i understand that the videocache already download the file but you can't hit it when you try to browse the same file again, also i want to know if that happen with all youtube videos or some of them ?
I accessed the video http://www.youtube.com/watch?v=l_7rhHj-VTs
Throughout the play the only logs were as follows
for the Squid:
1248888373.423 17942 189.*.*.* TCP_MISS/200 3383333 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv
1248888377.525 405 189.*.*.* TCP_MISS/200 18357 GET http://www.youtube.com/watch? - DIRECT/74.125.67.102 text/html
1248888380.042 0 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/0OzeLnfNe2w/default.jpg - NONE/- image/jpeg
1248888380.341 29 189.*.*.* TCP_IMS_HIT/304 296 GET http://s.ytimg.com/yt/swf/watch-vfl110367.swf - NONE/- application/x-shockwave-flash
1248888380.416 439 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/l_7rhHj-VTs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.543 345 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/Loy93Q6Dbvs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.945 436 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i2.ytimg.com/vi/5qBlNqKG2Bc/default.jpg - DIRECT/74.125.67.118 text/html
1248888381.730 262 189.*.*.* TCP_MISS/204 426 GET http://www.youtube.com/get_video? - DIRECT/74.125.67.102 text/html
1248888381.954 483 189.*.*.* TCP_MISS/304 209 GET http://www.youtube.com/crossdomain.xml - DIRECT/74.125.67.102 -
1248888382.616 38 189.*.*.* TCP_IMS_HIT/304 237 GET http://www.youtube.com/xlb/pt-br.xlb - NONE/- text/plain
1248888383.332 170 189.*.*.* TCP_MISS/204 251 GET http://video-stats.video.google.com/s? - DIRECT/74.125.67.127 text/html
1248888472.197 39 189.*.*.* TCP_HIT/200 4066 GET http://i2.ytimg.com/vi/-bFNKCH-AW4/default.jpg - NONE/- image/jpeg
1248888472.294 9 189.*.*.* TCP_HIT/200 2943 GET http://i1.ytimg.com/vi/pbM7pB7tg8s/default.jpg - NONE/- image/jpeg
1248888472.340 9 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/Tmo7rFuA3n0/default.jpg - NONE/- image/jpeg
1248888472.438 1 189.*.*.* TCP_HIT/200 3921 GET http://i4.ytimg.com/vi/gpiUUciN0Wk/default.jpg - NONE/- image/jpeg
1248888472.451 9 189.*.*.* TCP_HIT/200 3972 GET http://i4.ytimg.com/vi/ww7_c7a8gz0/default.jpg - NONE/- image/jpeg
1248888472.510 10 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/seM7pw90KPQ/default.jpg - NONE/- image/jpeg
1248888472.523 8 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/whYH4c7DvAQ/default.jpg - NONE/- image/jpeg
1248888472.552 10 189.*.*.* TCP_HIT/200 4051 GET http://i4.ytimg.com/vi/3zXR7m3fgNs/default.jpg - NONE/- image/jpeg
1248888472.582 25 189.*.*.* TCP_HIT/200 4022 GET http://i4.ytimg.com/vi/OC6EohGlZ8M/default.jpg - NONE/- image/jpeg
1248888472.934 1 189.*.*.* TCP_HIT/200 3946 GET http://i3.ytimg.com/vi/zcTMqTPw5bo/default.jpg - NONE/- image/jpeg
1248888472.950 0 189.*.*.* TCP_HIT/200 4015 GET http://i3.ytimg.com/vi/ZG9LKkNiP4U/default.jpg - NONE/- image/jpeg
1248888472.950 2 189.*.*.* TCP_IMS_HIT/304 276 GET http://i3.ytimg.com/vi/F-RP-JwR4Jc/default.jpg - NONE/- image/jpeg
1248888565.023 496 189.*.*.* TCP_MISS/200 662 GET http://ad.doubleclick.net/crossdomain.xml - DIRECT/216.73.86.153 text/xml
1248888565.582 289 189.*.*.* TCP_MISS/200 323 GET http://ad.doubleclick.net/activity;src=1185014;ytb=1;yatw=l_7rhHj-VTs; - DIRECT/216.73.86.153 image/gif
1248888565.954 184635 189.*.*.* TCP_MISS/200 16954055 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv
and for VideoCache:
2009-07-29 13:26:21,319 22211 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=9A2E35E0805536591F1652DBC480AAC7ACA26050.5A6203330959BA7FE6CFC1A39B07EBE213A5BBA1&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
VideoCache works when URL like this:
2009-07-29 13:33:12,780 22211 187.*.*.* - REQUEST - http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e
2009-07-29 13:33:12,780 22211 187.*.*.* d0a0fbf493c5e88e URL_HIT YOUTUBE http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e
Thanks for your help!
if it works when URL is like
http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e
i think this means that you match below acl first
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5] |2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplayback\\?
which handle by the below routine
# Google.com caching is handled here. if enable_google_cache: if (host.find('.google.com') > -1 or host.find('.googlevideo.com') > -1 or re.compile('\\.google\\.[a-z][a-z]').search(host) or re.compile('^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$').match(host)) and (path.find('videoplayback') > -1 or path.find('videoplay') > -1 or path.find('get_video') > -1) and path.find('get_video_info') < 0:
so make sure that the new acl is before the above google acl in the order or make it the first acl in your squid.conf, ofcourse in the videocache section
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
also check if you make the modification on the below section (as on my first post)
# Check if videocache plugin is on.
if enable_video_cache:
# Youtube.com caching is handled here.
if enable_youtube_cache:
if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:
type = 'YOUTUBE'
arglist = params.split('&')
dict = {}
for arg in arglist:
try:
dict[arg.split('=')[0]] = arg.split('=')[1]
except:
continue
if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
video_id = dict['id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
it's silly but let's make sure that in the /etc/videocache.conf, check if
enable_youtube_cache = 1 :-$
The ACL is in the other, see my squid.conf
url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 30
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
I change only this line:
if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:
it existed in his first post? But no matter, now work fine again! :D
My Squid is now crashing a lot, I do not know what it is. I think the version I'm using, squid-3.1.0.12.
Thanks a lot for your help! ;)
Hi guys;
I suggest that admin shld involve/agree in any changes that is needed for videocache . This can be in terms of patches instead we are making trial and error and eventually we will have a very different version form the original conf.
Hope admin will look at this threat very soon
Regards
Hi guys,
I insertes this changes in my /usr/share/videocache/videocache.py:
1) I change the line
if host.find('.youtube.com') > -1 and path.find('get_video') > -1 and path.find('get_video_info') < 0:
for
if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:
2) I change these lines:
if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new
For:
if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
video_id = dict['id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
In my Squid.conf I added in VideoCache's ACL:
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
Best regards,
Júlio Henrique
Thanks a lot for the code, I has the problem but now youtube cache work good
I modified my videocache.py file and I decided to upload it for those who cannot figure out how to correctly mod their files.
Just a question. is this for videocache 1.9.1 only? will it work for 1.9.0?
Hi Lferreira,
I don't know this, because I just test the code on videocache 1.9.1.
Best regards.
Hi all,
Please upgrade to version 1.9.2. You should be rushing to the download page :D
Thank You!