VideoCache

youtube.com only REQUEST but not caching it

by Anonymous on 28 Jul 2009

Hi Admin,

Till morning videocache was working fine but then its stop working as squid was not redirecting the youtube video url to it. after checking logs I found that video url has been change from http://v12.lscache5.googlevideo.com/ to http://v4.cache5.c.youtube.com/

after adding this below line in my squid.conf
acl videocache_allow_url url_regex -i \.c\.youtube\.com\/

But after squid is redirecting to videocache & Its only generating REQUEST in logs but it did not do any thing , like URL_HIT etc etc etc my log part is below.

same condition for the files which are already in cache of videocache.

thanks in advance.

my os is linux and distro is centos 5.3 + squid3.16 and it was running perfectly

TheOne Khan

2009-07-29 21:16:38,375 3749 10.101.10.1 - REQUEST - http://v4.cache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=D92DF83A04AAAE59C830D2BE333FBB56E9FBCDE6.B01ED2A0F7CA69D0C188F7BF50C74AF82C86E853&sver=3&expire=1248818400&key=yt1&factor=1.25&burst=40&id=80cb95dddfcdfeba&begin=68000&redirect_counter=1
2009-07-29 21:16:38,375 3749 10.101.10.3 - REQUEST - http://v14.cache6.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=D92DF83A04AAAE59C830D2BE333FBB56E9FBCDE6.B01ED2A0F7CA69D0C188F7BF50C74AF82C86E853&sver=3&expire=1248818400&key=yt1&factor=1.25&burst=40&id=80cb95dddfcdfeba&begin=68000&redirect_counter=1

19 Answers

by last1ghost1 on 28 Jul 2009

Dear Sir,

i'm not that expert in videocache, i do the below steps and it works with me, i hope the admin can advice us also in squid.conf videocache section, i add the below line

acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback 

in the /usr/share/videocache/videocache.py search for 'youtube' you will find lines like this

  if dict.has_key('video_id'):
    video_id = dict['video_id']
    new_url = submit_video(pid, client, type, url, video_id)
  else:
    log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

comment it and change it to be

if dict.has_key('video_id'):
  video_id = dict['video_id']
  new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
  video_id = dict['id']
  new_url = submit_video(pid, client, type, url, video_id)
else:
  log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

i hope this solution works for you, currently it works for me :-)

by lopan on 28 Jul 2009

For some requests still does not work.

Take a look:

2009-07-28 15:40:24,753 6274 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b

I have this id 97feeb8478fe553b

# ls -l 97feeb8478fe553b
-rw-r--r-- 1 nobody nogroup 16953652 2009-07-27 14:40 97feeb8478fe553b

But, the VC not HIT in cache.

thanks for any help!

by last1ghost1 on 28 Jul 2009

i copy and paste URL in my browse VC log said:

2009-07-29 00:46:44,224 10077 10.11.11.41 97feeb8478fe553b CACHE_MISS YOUTUBE Requested video was not found in cache.
2009-07-29 00:46:44,227 10077 10.11.11.41 97feeb8478fe553b NEW_URL YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:46:44,403 10085 10.11.11.41 97feeb8478fe553b SCHEDULED YOUTUBE Video scheduled for download.
2009-07-29 00:46:44,415 10077 192.168.113.140 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b

after it complete download, it hit the cache as you can see

2009-07-29 00:50:19,652 10077 10.11.11.41 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b URL_HIT YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=7CEA363A6609F77E694D0BF8E812051DBAC8207C.696CE6C827B99E72CCFE68AA4D82F409CA7239F0&sver=3&expire=1248829200&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b CACHE_HIT YOUTUBE Video was served from cache.

is that happen with all videos ?? do you have any other acl related to youtube.com ?

by lopan on 28 Jul 2009

Yes, in the order I have the following ACLs

acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
acl videocache_allow_url url_regex -i \\.youtube\\.com\\/get_video\\?
acl videocache_deny_url url_regex -i http:\\/\\/[a-z][a-z]\\.youtube\\.com http:\\/\\/www\\.youtube\\.com

Remove the cached ID: 97feeb8478fe553b and the VC not download when I accessed again.

Thanks for your help!

by sirkike on 29 Jul 2009

I have the same problem, this fix the problem?

by last1ghost1 on 29 Jul 2009

could you send me the log generated by squid and videocache since request begin, also as i understand that the videocache already download the file but you can't hit it when you try to browse the same file again, also i want to know if that happen with all youtube videos or some of them ?

by lopan on 29 Jul 2009

I accessed the video http://www.youtube.com/watch?v=l_7rhHj-VTs

Throughout the play the only logs were as follows

for the Squid:

1248888373.423  17942 189.*.*.* TCP_MISS/200 3383333 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv
1248888377.525    405 189.*.*.* TCP_MISS/200 18357 GET http://www.youtube.com/watch? - DIRECT/74.125.67.102 text/html
1248888380.042      0 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/0OzeLnfNe2w/default.jpg - NONE/- image/jpeg
1248888380.341     29 189.*.*.* TCP_IMS_HIT/304 296 GET http://s.ytimg.com/yt/swf/watch-vfl110367.swf - NONE/- application/x-shockwave-flash
1248888380.416    439 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/l_7rhHj-VTs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.543    345 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/Loy93Q6Dbvs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.945    436 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i2.ytimg.com/vi/5qBlNqKG2Bc/default.jpg - DIRECT/74.125.67.118 text/html
1248888381.730    262 189.*.*.* TCP_MISS/204 426 GET http://www.youtube.com/get_video? - DIRECT/74.125.67.102 text/html
1248888381.954    483 189.*.*.* TCP_MISS/304 209 GET http://www.youtube.com/crossdomain.xml - DIRECT/74.125.67.102 -
1248888382.616     38 189.*.*.* TCP_IMS_HIT/304 237 GET http://www.youtube.com/xlb/pt-br.xlb - NONE/- text/plain
1248888383.332    170 189.*.*.* TCP_MISS/204 251 GET http://video-stats.video.google.com/s? - DIRECT/74.125.67.127 text/html
1248888472.197     39 189.*.*.* TCP_HIT/200 4066 GET http://i2.ytimg.com/vi/-bFNKCH-AW4/default.jpg - NONE/- image/jpeg
1248888472.294      9 189.*.*.* TCP_HIT/200 2943 GET http://i1.ytimg.com/vi/pbM7pB7tg8s/default.jpg - NONE/- image/jpeg
1248888472.340      9 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/Tmo7rFuA3n0/default.jpg - NONE/- image/jpeg
1248888472.438      1 189.*.*.* TCP_HIT/200 3921 GET http://i4.ytimg.com/vi/gpiUUciN0Wk/default.jpg - NONE/- image/jpeg
1248888472.451      9 189.*.*.* TCP_HIT/200 3972 GET http://i4.ytimg.com/vi/ww7_c7a8gz0/default.jpg - NONE/- image/jpeg
1248888472.510     10 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/seM7pw90KPQ/default.jpg - NONE/- image/jpeg
1248888472.523      8 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/whYH4c7DvAQ/default.jpg - NONE/- image/jpeg
1248888472.552     10 189.*.*.* TCP_HIT/200 4051 GET http://i4.ytimg.com/vi/3zXR7m3fgNs/default.jpg - NONE/- image/jpeg
1248888472.582     25 189.*.*.* TCP_HIT/200 4022 GET http://i4.ytimg.com/vi/OC6EohGlZ8M/default.jpg - NONE/- image/jpeg
1248888472.934      1 189.*.*.* TCP_HIT/200 3946 GET http://i3.ytimg.com/vi/zcTMqTPw5bo/default.jpg - NONE/- image/jpeg
1248888472.950      0 189.*.*.* TCP_HIT/200 4015 GET http://i3.ytimg.com/vi/ZG9LKkNiP4U/default.jpg - NONE/- image/jpeg
1248888472.950      2 189.*.*.* TCP_IMS_HIT/304 276 GET http://i3.ytimg.com/vi/F-RP-JwR4Jc/default.jpg - NONE/- image/jpeg
1248888565.023    496 189.*.*.* TCP_MISS/200 662 GET http://ad.doubleclick.net/crossdomain.xml - DIRECT/216.73.86.153 text/xml
1248888565.582    289 189.*.*.* TCP_MISS/200 323 GET http://ad.doubleclick.net/activity;src=1185014;ytb=1;yatw=l_7rhHj-VTs; - DIRECT/216.73.86.153 image/gif
1248888565.954 184635 189.*.*.* TCP_MISS/200 16954055 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv

and for VideoCache:

2009-07-29 13:26:21,319 22211 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cburst%2Cfactor&itag=34&ipbits=0&signature=9A2E35E0805536591F1652DBC480AAC7ACA26050.5A6203330959BA7FE6CFC1A39B07EBE213A5BBA1&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=97feeb8478fe553b

VideoCache works when URL like this:

2009-07-29 13:33:12,780 22211 187.*.*.* - REQUEST - http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e
2009-07-29 13:33:12,780 22211 187.*.*.* d0a0fbf493c5e88e URL_HIT YOUTUBE http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e

Thanks for your help!

by last1ghost1 on 29 Jul 2009

if it works when URL is like

http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Cshardbypass%2Cburst%2Cfactor&shardbypass=yes&itag=34&ipbits=0&signature=06C5A35E4170ECFF7AA50C8DDDE3C870E0386F18.4A3F56E359A6C38B05C3E4FE86A52355B89D1B33&sver=3&expire=1248908400&key=yt1&factor=1.25&burst=40&id=d0a0fbf493c5e88e

i think this means that you match below acl first

acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5] |2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\/videoplayback\\?

which handle by the below routine

# Google.com caching is handled here. if enable_google_cache: if (host.find('.google.com') > -1 or host.find('.googlevideo.com') > -1 or re.compile('\\.google\\.[a-z][a-z]').search(host) or re.compile('^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$').match(host)) and (path.find('videoplayback') > -1 or path.find('videoplay') > -1 or path.find('get_video') > -1) and path.find('get_video_info') < 0:

so make sure that the new acl is before the above google acl in the order or make it the first acl in your squid.conf, ofcourse in the videocache section

acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback
by last1ghost1 on 29 Jul 2009

also check if you make the modification on the below section (as on my first post)

# Check if videocache plugin is on.
        if enable_video_cache:
            # Youtube.com caching is handled here.
            if enable_youtube_cache:
                if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:
                    type = 'YOUTUBE'
                    arglist = params.split('&')
                    dict = {}
                    for arg in arglist:
                        try:
                            dict[arg.split('=')[0]] = arg.split('=')[1]
                        except:
                            continue
                    if dict.has_key('video_id'):
                        video_id = dict['video_id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    elif dict.has_key('id'):
                        video_id = dict['id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    else:
                        log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
by last1ghost1 on 29 Jul 2009

it's silly but let's make sure that in the /etc/videocache.conf, check if
enable_youtube_cache = 1 :-$

by lopan on 29 Jul 2009

The ACL is in the other, see my squid.conf

url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 30
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on
by lopan on 29 Jul 2009

I change only this line:

if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:

it existed in his first post? But no matter, now work fine again! :D

My Squid is now crashing a lot, I do not know what it is. I think the version I'm using, squid-3.1.0.12.

Thanks a lot for your help! ;)

by GELL on 30 Jul 2009

Hi guys;

I suggest that admin shld involve/agree in any changes that is needed for videocache . This can be in terms of patches instead we are making trial and error and eventually we will have a very different version form the original conf.

Hope admin will look at this threat very soon

Regards

by Júlio Henrique de Oliveira on 10 Aug 2009

Hi guys,

I insertes this changes in my /usr/share/videocache/videocache.py:

1) I change the line

if host.find('.youtube.com') > -1 and path.find('get_video') > -1 and path.find('get_video_info') < 0:

for

if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:

2) I change these lines:

if dict.has_key('video_id'):
  video_id = dict['video_id']
  new_url = submit_video(pid, client, type, url, video_id)
else:
 log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new

For:

if dict.has_key('video_id'):
  video_id = dict['video_id']
  new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
  video_id = dict['id']
  new_url = submit_video(pid, client, type, url, video_id)
else:
  log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

In my Squid.conf I added in VideoCache's ACL:

acl videocache_allow_url url_regex -i \\.youtube\\.com\\/videoplayback

Best regards,
Júlio Henrique

by william on 13 Aug 2009

Thanks a lot for the code, I has the problem but now youtube cache work good

by Anonymous on 22 Aug 2009

I modified my videocache.py file and I decided to upload it for those who cannot figure out how to correctly mod their files.

http://www.megaupload.com/?d=TF8RGC99

by lferreira on 26 Aug 2009

Just a question. is this for videocache 1.9.1 only? will it work for 1.9.0?

by Júlio Henrique de Oliveira on 26 Aug 2009

Hi Lferreira,

I don't know this, because I just test the code on videocache 1.9.1.

Best regards.

by Kulbir Saini on 26 Aug 2009

Hi all,

Please upgrade to version 1.9.2. You should be rushing to the download page :D

Thank You!

You need to sign in. Please sign in to add answer to this question.