You are hereForums / Support / youtube.com only REQUEST but not caching it

Finally, the wait is over!!! Videocache 2.0.0 is here :-) Check changelog for changes and new features in this version. Contact Us for your free upgrade!!

youtube.com only REQUEST but not caching it

Post Reply

19 replies [Last post]

28 July, 2009 - 21:27

Anonymous

Hi Admin,

Till morning videocache was working fine but then its stop working as squid was not redirecting the youtube video url to it. after checking logs I found that video url has been change from http://v12.lscache5.googlevideo.com/ to http://v4.cache5.c.youtube.com/

after adding this below line in my squid.conf
acl videocache_allow_url url_regex -i \.c\.youtube\.com\/

But after squid is redirecting to videocache & Its only generating REQUEST in logs but it did not do any thing , like URL_HIT etc etc etc my log part is below.

same condition for the files which are already in cache of videocache.

thanks in advance.

my os is linux and distro is centos 5.3 + squid3.16 and it was running perfectly

TheOne Khan

2009-07-29 21:16:38,375 3749 10.101.10.1 - REQUEST - http://v4.cache5.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexp...

2009-07-29 21:16:38,375 3749 10.101.10.3 - REQUEST - http://v14.cache6.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cex...

28 July, 2009 - 23:36

last1ghost1

Offline

Joined: Jul 27 2009

Dear Sir, i'm not that expert in videocache, i do the below steps and it works with me, i hope the admin can advice us also
in squid.conf videocache section, i add the below line
#########################
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
#########################
in the /usr/share/videocache/videocache.py search for 'youtube' you will find lines like this

########################################################
                    if dict.has_key('video_id'):
                        video_id = dict['video_id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    else:
                        log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))
########################################################
comment it and change it to be
###############################################
if dict.has_key('video_id'):
                        video_id = dict['video_id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    elif dict.has_key('id'):
                        video_id = dict['id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    else:
                        log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

##############################################

i hope this solution works for you, currently it works for me :-)

29 July, 2009 - 00:14

lopan

Offline

Joined: Nov 5 2008

For some requests still does not work.

Take a look:

2009-07-28 15:40:24,753 6274 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...

I have this id 97feeb8478fe553b

# ls -l 97feeb8478fe553b
-rw-r--r-- 1 nobody nogroup 16953652 2009-07-27 14:40 97feeb8478fe553b

But, the VC not HIT in cache.

thanks for any help!

29 July, 2009 - 00:24

last1ghost1

Offline

Joined: Jul 27 2009

i copy and paste URL in my browse VC log said:
#####
2009-07-29 00:46:44,224 10077 10.11.11.41 97feeb8478fe553b CACHE_MISS YOUTUBE Requested video was not found in cache.
2009-07-29 00:46:44,227 10077 10.11.11.41 97feeb8478fe553b NEW_URL YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...
2009-07-29 00:46:44,403 10085 10.11.11.41 97feeb8478fe553b SCHEDULED YOUTUBE Video scheduled for download.
2009-07-29 00:46:44,415 10077 192.168.113.140 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...
#####

after it complete download, it hit the cache as you can see
#####
2009-07-29 00:50:19,652 10077 10.11.11.41 - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b URL_HIT YOUTUBE http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...
2009-07-29 00:50:19,653 10077 10.11.11.41 97feeb8478fe553b CACHE_HIT YOUTUBE Video was served from cache.
####
is that happen with all videos ?? do you have any other acl related to youtube.com ?

29 July, 2009 - 00:32

lopan

Offline

Joined: Nov 5 2008

Yes, in the order I have the following ACLs

acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com

Remove the cached ID: 97feeb8478fe553b and the VC not download when I accessed again.

Thanks for your help!

29 July, 2009 - 17:18

sirkike

Offline

Joined: Feb 6 2009

I have the same problem, this fix the problem?

29 July, 2009 - 20:05

last1ghost1

Offline

Joined: Jul 27 2009

could you send me the log generated by squid and videocache since request begin, also as i understand that the videocache already download the file but you can't hit it when you try to browse the same file again, also i want to know if that happen with all youtube videos or some of them ?

29 July, 2009 - 22:05

lopan

Offline

Joined: Nov 5 2008

I accessed the video http://www.youtube.com/watch?v=l_7rhHj-VTs

Throughout the play the only logs were as follows

for the Squid:
1248888373.423 17942 189.*.*.* TCP_MISS/200 3383333 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv
1248888377.525 405 189.*.*.* TCP_MISS/200 18357 GET http://www.youtube.com/watch? - DIRECT/74.125.67.102 text/html
1248888380.042 0 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/0OzeLnfNe2w/default.jpg - NONE/- image/jpeg
1248888380.341 29 189.*.*.* TCP_IMS_HIT/304 296 GET http://s.ytimg.com/yt/swf/watch-vfl110367.swf - NONE/- application/x-shockwave-flash
1248888380.416 439 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/l_7rhHj-VTs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.543 345 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i1.ytimg.com/vi/Loy93Q6Dbvs/default.jpg - DIRECT/74.125.67.133 text/html
1248888380.945 436 189.*.*.* TCP_REFRESH_UNMODIFIED/304 327 GET http://i2.ytimg.com/vi/5qBlNqKG2Bc/default.jpg - DIRECT/74.125.67.118 text/html
1248888381.730 262 189.*.*.* TCP_MISS/204 426 GET http://www.youtube.com/get_video? - DIRECT/74.125.67.102 text/html
1248888381.954 483 189.*.*.* TCP_MISS/304 209 GET http://www.youtube.com/crossdomain.xml - DIRECT/74.125.67.102 -
1248888382.616 38 189.*.*.* TCP_IMS_HIT/304 237 GET http://www.youtube.com/xlb/pt-br.xlb - NONE/- text/plain
1248888383.332 170 189.*.*.* TCP_MISS/204 251 GET http://video-stats.video.google.com/s? - DIRECT/74.125.67.127 text/html
1248888472.197 39 189.*.*.* TCP_HIT/200 4066 GET http://i2.ytimg.com/vi/-bFNKCH-AW4/default.jpg - NONE/- image/jpeg
1248888472.294 9 189.*.*.* TCP_HIT/200 2943 GET http://i1.ytimg.com/vi/pbM7pB7tg8s/default.jpg - NONE/- image/jpeg
1248888472.340 9 189.*.*.* TCP_IMS_HIT/304 276 GET http://i1.ytimg.com/vi/Tmo7rFuA3n0/default.jpg - NONE/- image/jpeg
1248888472.438 1 189.*.*.* TCP_HIT/200 3921 GET http://i4.ytimg.com/vi/gpiUUciN0Wk/default.jpg - NONE/- image/jpeg
1248888472.451 9 189.*.*.* TCP_HIT/200 3972 GET http://i4.ytimg.com/vi/ww7_c7a8gz0/default.jpg - NONE/- image/jpeg
1248888472.510 10 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/seM7pw90KPQ/default.jpg - NONE/- image/jpeg
1248888472.523 8 189.*.*.* TCP_IMS_HIT/304 276 GET http://i4.ytimg.com/vi/whYH4c7DvAQ/default.jpg - NONE/- image/jpeg
1248888472.552 10 189.*.*.* TCP_HIT/200 4051 GET http://i4.ytimg.com/vi/3zXR7m3fgNs/default.jpg - NONE/- image/jpeg
1248888472.582 25 189.*.*.* TCP_HIT/200 4022 GET http://i4.ytimg.com/vi/OC6EohGlZ8M/default.jpg - NONE/- image/jpeg
1248888472.934 1 189.*.*.* TCP_HIT/200 3946 GET http://i3.ytimg.com/vi/zcTMqTPw5bo/default.jpg - NONE/- image/jpeg
1248888472.950 0 189.*.*.* TCP_HIT/200 4015 GET http://i3.ytimg.com/vi/ZG9LKkNiP4U/default.jpg - NONE/- image/jpeg
1248888472.950 2 189.*.*.* TCP_IMS_HIT/304 276 GET http://i3.ytimg.com/vi/F-RP-JwR4Jc/default.jpg - NONE/- image/jpeg
1248888565.023 496 189.*.*.* TCP_MISS/200 662 GET http://ad.doubleclick.net/crossdomain.xml - DIRECT/216.73.86.153 text/xml
1248888565.582 289 189.*.*.* TCP_MISS/200 323 GET http://ad.doubleclick.net/activity;src=1185014;ytb=1;yatw=l_7rhHj-VTs; - DIRECT/216.73.86.153 image/gif
1248888565.954 184635 189.*.*.* TCP_MISS/200 16954055 GET http://v8.lscache3.c.youtube.com/videoplayback? - DIRECT/74.125.2.101 video/x-flv

and for VideoCache:
2009-07-29 13:26:21,319 22211 189.*.*.* - REQUEST - http://v8.lscache3.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Ce...

VideoCache works when URL like this:
2009-07-29 13:33:12,780 22211 187.*.*.* - REQUEST - http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2C...
2009-07-29 13:33:12,780 22211 187.*.*.* d0a0fbf493c5e88e URL_HIT YOUTUBE http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2C...

Thanks for your help!

29 July, 2009 - 22:36

last1ghost1

Offline

Joined: Jul 27 2009

if it works when URL is like
http://74.125.2.100/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2C...
i think this means that you match below acl first
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5] |2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
which handle by the below routine
# Google.com caching is handled here. if enable_google_cache: if (host.find('.google.com') > -1 or host.find('.googlevideo.com') > -1 or re.compile('\.google\.[a-z][a-z]').search(host) or re.compile('^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$').match(host)) and (path.find('videoplayback') > -1 or path.find('videoplay') > -1 or path.find('get_video') > -1) and path.find('get_video_info') < 0:
so make sure that the new acl is before the above google acl in the order or make it the first acl in your squid.conf, ofcourse in the videocache section
################
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
##################

29 July, 2009 - 22:47

lopan

Offline

Joined: Nov 5 2008

The ACL is in the other, see my squid.conf

url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
url_rewrite_children 30
acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback
acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl videocache_allow_url url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/ mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com .cdn.dailymotion.com
acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom
redirector_bypass on

29 July, 2009 - 22:41

#10

last1ghost1

Offline

Joined: Jul 27 2009

also check if you make the modification on the below section (as on my first post)
# Check if videocache plugin is on.
        if enable_video_cache:
            # Youtube.com caching is handled here.
            if enable_youtube_cache:
                if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:
                    type = 'YOUTUBE'
                    arglist = params.split('&')
                    dict = {}
                    for arg in arglist:
                        try:
                            dict[arg.split('=')[0]] = arg.split('=')[1]
                        except:
                            continue
                    if dict.has_key('video_id'):
                        video_id = dict['video_id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    elif dict.has_key('id'):
                        video_id = dict['id']
                        new_url = submit_video(pid, client, type, url, video_id)
                    else:
                        log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

29 July, 2009 - 23:13

#11

lopan

Offline

Joined: Nov 5 2008

I change only this line:

if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:

it existed in his first post? But no matter, now work fine again! :D

My Squid is now crashing a lot, I do not know what it is. I think the version I'm using, squid-3.1.0.12.

Thanks a lot for your help! ;)

29 July, 2009 - 22:43

#12

last1ghost1

Offline

Joined: Jul 27 2009

it's silly but let's make sure that in the /etc/videocache.conf, check if
enable_youtube_cache = 1 :-$

30 July, 2009 - 07:27

#13

Gell (not verified)

Hi guys;

I suggest that admin shld involve/agree in any changes that is needed for videocache . This can be in terms of patches instead we are making trial and error and eventually we will have a very different version form the original conf.

Hope admin will look at this threat very soon

Regards

11 August, 2009 - 02:20

#14

Julio Henrique (not verified)

Hi guys,

I insertes this changes in my /usr/share/videocache/videocache.py:

1) I change the line "if host.find('.youtube.com') > -1 and path.find('get_video') > -1 and path.find('get_video_info') < 0:" for "if host.find('.youtube.com') > -1 and path.find('get_video') > -1 or path.find('videoplayback') and path.find('get_video_info') < 0:"

2) I change these lines:

if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new

For:

if dict.has_key('video_id'):
video_id = dict['video_id']
new_url = submit_video(pid, client, type, url, video_id)
elif dict.has_key('id'):
video_id = dict['id']
new_url = submit_video(pid, client, type, url, video_id)
else:
log(format%(pid, client, '-', 'URL_ERROR', type, 'video_id not found in ' + new_url))

In my Squid.conf I added in VideoCache's ACL:

acl videocache_allow_url url_regex -i \.youtube\.com\/videoplayback

Best regards,

Júlio Henrique

14 August, 2009 - 03:09

#15

William (not verified)

Thanks a lot for the code, I has the problem but now youtube cache work good

22 August, 2009 - 09:07

#16

Anonymous (not verified)

I modified my videocache.py file and I decided to upload it for those who cannot figure out how to correctly mod their files.

http://www.megaupload.com/?d=TF8RGC99

26 August, 2009 - 16:48

#17

lferreira

Offline

Joined: Feb 16 2009

Just a question. is this for videocache 1.9.1 only? will it work for 1.9.0?

27 August, 2009 - 04:06

#18

Julio Henrique (not verified)

Hi Lferreira,

I don't know this, because I just test the code on videocache 1.9.1.

Best regards.

27 August, 2009 - 04:24

#19

admin

Offline

Joined: Nov 2 2008

Hi all,

Please upgrade to version 1.9.2. You should be rushing to the download page :D

Thank You!

Post new comment

Your name: *

E-mail: *

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account associated with the e-mail address you provide, it will be used to display your avatar.

Homepage:

Comment: *

Web page addresses and e-mail addresses turn into links automatically.
Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <code>
Lines and paragraphs break automatically.
HTML tags will be transformed to conform to HTML standards.
Search Engines will index and follow ONLY links to allowed domains.

More information about formatting options

Notify me when new comments are posted

All comments

Replies to my comment

CAPTCHA

Are you a human visitor ?

What code is in the image?: *

Enter the characters shown in the image.

Introduction

VideoCache Hacks

youtube.com only REQUEST but not caching it

Post new comment

My Book on Squid

Purchase/Renew

New forum topics

Recent comments

Who's online

Introduction

User login

VideoCache Hacks

youtube.com only REQUEST but not caching it

Post new comment

My Book on Squid

Purchase/Renew

New forum topics

Recent comments

Who's online