Request a website to cache!
by Kulbir Saini on 27 Jan 2009Dear all,
I am creating this topic for capturing your requests for caching new websites. Please reply to this topic in order you want a website to be included in videocache and I'll add it to the Todo List. Please provide as much details as possible about the website and the videos!
Thank you for using videocache!
61 Answers
Hello!
My users are viewing a lot of videos from www.3cat24.cat
Trey are also flash videos.
This is a public news channel in my country (Barcelona - Catalonia - Spain).
It would be interesting to have these videos cached in a new version.
Thanks,
Josep Pujadas
Hello!
One more for me:
This is a public educational network with a lot of videos in my country (Barcelona - Catalonia - Spain).
The videos are in Real Player format and are served from http://video.xtec.cat:8080
For example, the page
http://www.edu3.cat/Edu3tv/Fitxa?p_id=37475
contains the video
http://video.xtec.cat:8080/ramgen/edu3tv/video/tvc/qkicom/1154428.rm
Regards,
Josep Pujadas
Dear Josep,
Added both to the todo list.
Thank you for the active participation :)
you can add:
pornhub.com
xvideo.com
This site are using alot on my network. :P,
Max
Dear Kulbir,
Could you add :
[root@proxy 3.17]# tail -10000 /var/log/squid/access.log | grep videoplay | grep -v PARENT
1233714651.119 22864 202.182.55.122 TCP_MISS/200 273504 GET http://208.117.252.91/videoplayback? - DIRECT/208.117.252.91 video/x-flv
1233714654.004 2242 202.182.53.36 TCP_MISS/200 10881 GET http://208.117.252.85/videoplayback? - DIRECT/208.117.252.85 video/x-flv
1233714677.950 210861 202.182.54.50 TCP_MISS/200 16824205 GET http://208.117.252.81/videoplayback? - DIRECT/208.117.252.81 video/x-flv
1233714867.658 221168 202.182.49.160 TCP_MISS/200 3712771 GET http://208.117.252.91/videoplayback? - DIRECT/208.117.252.91 video/x-flv
Seems some youtube / google video site aren't using
domain but using ip.
And the pattern is they always use videoplayback?
Could you add this to your to-do list ?
Thank's
//Budi
Budi,
Thanks for the insight. Included in Todo List now.
Update : This task has been completed.
Thiago,
My primary concern is to cover more and more audio and video portals/websites. After we have significant number of audio/video websites supported, we can go for image caching as well :)
Thank You!
1234132171.450 178212 202.182.49.136 TCP_MISS/206 512842 GET http://mobile01.tube8.com/flv/151677c042260db9a66621847767fde7/498eeca1/0812/11/4940f69a0ca6c/4940f69a0ca6c.3gp - DIRECT/208.88.224.77 video/mp4
1234132171.454 39782 202.182.49.136 TCP_MISS/206 17226 GET http://mobile01.tube8.com/flv/151677c042260db9a66621847767fde7/498eeca1/0812/11/4940f69a0ca6c/4940f69a0ca6c.3gp - DIRECT/208.88.224.77 video/mp4
Dear Kulbir, Seems content from mobilexx.tube8.com aren't get
caching yet.
Could you add to your todo list ?
Thx
//Budi
Please...
i need cachevideos rewrite the orkut.com img servers url to static server 4 squid index.
ex:
img3.orkut.com/xxxx/1.jpg
to
img1.orkut.com/xxxx/1.jpg
in php works here:
else if (preg_match("/^http:\\/\\/img[2-9]\\.orkut\\.com.*/", $url, $result)) {
// ############################### ORKUT
$url = preg_replace("/\\/img[0-9]\\./", "/img1.", $url);
print "$url\\n";
and antivirus too
else if (preg_match("/^http:\\/\\/update\\.avg\\.com.*/", $url, $resultado)) {
$url = preg_replace("/\\/update\\./", "/guru.", $url);
print "$url\\n";
many tks
Hey Thunder,
I appreciate your efforts in writing the above script but we are not really into image caching right now. We'll try to include this part as soon as we finish the videos part.
Thank you for your efforts.
Budi,
Thanks for the details. Included in Todo List.
Update : This task has been completed.
Josep,
Videos from this website are served via RTSP protocol and I don't really have any knowledge of how that protocol works. I tried few tricks but they didn't work. So its unlikely that I'll be able to support it.
Thank You!
Josep,
Videos from this site are search via RTMP protocol which I was unable to follow. I think I wont be able to support this site.
Thank You!
Heya,
Thanks for a great resource, it is saving us a lot of work here! Would it be possible to add teachertube.com to the list of supported websites? The relevant video entry bit in the squid log is
1234532453.084 7140 w.x.y.z TCP_MISS/200 4892170 GET http://www.teachertube.com/flvideo2/80159.flv - CD_PARENT_HIT/x.y.com text/plain
Thanks in advance
Wes
Schmindie,
It seems teacher tube serves videos in a manner so that they can be cached by squid itself. I mean the urls are static in nature. So, videocache need not support this website.
Thank you for your interest and compliments :)
Kulbir,
Thanks for your interest. I found a solution for this case because there is also a http static URL. Thanks,
Josep Pujadas
Kulbir,
Thanks for your interest. I found a solution for this case because there is also a http static URL. Thanks,
Josep Pujadas
Kulbir,
Looking at my logs I found the following networks for google/youtube:
208.65.152.0/22 YOUTUBE 1
75.35.233.64/29 YOUTUBE 2
75.31.180.240/29 YOUTUBE 3
64.15.112.0/20 YOUTUBE2
208.117.224.0/19 YOUTUBE3
195.59.171.0/24 GOOGLE/YOUTUBE EUROPE
213.146.171.0/24 GOOGLE/YOUTUBE EUROPE
194.221.68.0/24 GOOGLE/YOUTUBE EUROPE
74.125.0.0/16 GOOGLE (googlevideo.com)
204.246.232.0/24 GOOGLE (googlevideo.com)
206.132.73.0/24 GOOGLE (googlevideo.com)
I searched these networks at ARIN and RIPE because I needed to redirect my video queries to a second ADSL connection.
Perhaps it helps also for videoplayback by IP address.
Regards,
Josep Pujadas
Josep,
The network list from your logs may not be exhaustive for two reasons,
- They have world wide localization. Different countries may have different IP ranges.
- Google/Youtube is ever expanding network.
Anyways videocache handles the videos served via IPs properly, so we need not worry about it.
Thank you for the efforts.
Kulbir,
"The network list from your logs may not be exhaustive"
Yes, I know. You're right.
I saw at your git (http://fedorapeople.org/gitweb?p=kulbirsaini/public_git/videocache.git) you include the IP feature at 1.9 version ...
Thanks for your work! Now I have to migrate from 1.9 to 1.9!
Regards,
Josep Pujadas
Josep,
I have setup a new code repository which is located at http://git.cachevideos.com/videocache.git and easy to remember link :D Please bookmark this one.
Thank You!
Anon,
xvideos uses a really strange mechanism to serve videos and I am not really able to get the hang of their algorithm. If you have any insight about it, please post it here.
Thank You!
looking in my logs, I've found that the request url seems to be the same every time, but the squid don't serve the file.
Kimble,
I didn't really get you. Can you please cite an example.
Thank You!
Sorry for my bad English.
I have 1900 users and ~ 300-500 MB Traffic/D (request) on youtube.
Thanks VideoCache only 50MB Cache (multiple connections) Great Work!
On Sevenload.com I have ~ 300-600 MB traffic per day and no idea. (regexp)
Sample Url: http://www.bildschirmarbeiter.com/video/pakettransport_in_china/
Header:
+++GET 2378609+++
Using Proxy - 192.168.0.5:8080
GET http://datal3.sevenload.com/data59.sevenload.com/slcom/rr/ze/hgfimld/zlwslmnlcimf.flv HTTP/1.1
Host: datal3.sevenload.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 700
Cookie: s_vi=[CS]v1|497E3142000079F5-A02083A00001F2B[CE]; session=17e09a48918aa03b250672baf2b9bf16; slbrowserid=70df8e722d0e3e29308d73773c8489c337e51e4d; s_cc=true; s_sq=%5B%5BB%5D%5D
Connection: keep-alive
+++RESP 2378609+++
HTTP/1.1 200 OK
Cache-Control: max-age=3600
Date: Sun, 15 Mar 2009 19:02:37 GMT
Content-Length: 1406413
Content-Type: video/x-flv
ETag: "-2027975073"
Expires: Fri, 13 Mar 2009 11:52:56 GMT
Last-Modified: Thu, 12 Mar 2009 19:17:37 GMT
Accept-Ranges: bytes
Server: lighttpd
Age: 193
X-Cache: HIT from proxy.gate #<-- no no hit ;)
X-Cache-Lookup: HIT from proxy.gate:3128
Via: 1.0 proxy.gate (squid/3.0.STABLE8)
Proxy-Connection: keep-alive
+++Close 2378609+++
Anon,
Thanks a lot for providing the header information for request. This actually helps in speeding up the support process for a particular websites.
I'll try to add the support for this website as soon as possible.
Thank You!
http://www.myvideo.de/
(in particular http://www.myvideo.de/musik-tv - but that should be handled the same way)
first video
GET http://myvideo-610.vo.llnwd.net/d4/movie15/7f/5451611.flv HTTP/1.1
Host: myvideo-610.vo.llnwd.net
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.1.15) Gecko/20080612 Iceweasel/2.0.0.15 (Debian-2.0.0.15-0etch1)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Proxy-Connection: keep-alive
HTTP/1.0 200 OK
Server: Apache
Accept-Ranges: bytes
Content-Length: 26097844
Cache-Control: s-maxage=3600
X-Permitted-Cross-Domain-Policies: all
Content-Type: text/plain
Age: 41932
Date: Tue, 24 Mar 2009 19:46:20 GMT
Last-Modified: Thu, 13 Nov 2008 08:26:36 GMT
Expires: Tue, 24 Mar 2009 20:07:28 GMT
X-Cache: MISS from proxy.lnet
X-Cache-Lookup: MISS from proxy.lnet:3128
Via: 1.0 proxy.lnet:3128 (squid/2.6.STABLE5)
Proxy-Connection: keep-alive
same video again (reloaded page and used search function to replay the video)
GET http://myvideo-610.vo.llnwd.net/d4/movie15/7f/5451611.flv HTTP/1.1
Host: myvideo-610.vo.llnwd.net
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.1.15) Gecko/20080612 Iceweasel/2.0.0.15 (Debian-2.0.0.15-0etch1)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Proxy-Connection: keep-alive
HTTP/1.0 200 OK
Server: Apache
Accept-Ranges: bytes
Content-Length: 26097844
Cache-Control: s-maxage=3600
X-Permitted-Cross-Domain-Policies: all
Content-Type: text/plain
Age: 41970
Date: Tue, 24 Mar 2009 19:46:58 GMT
Last-Modified: Thu, 13 Nov 2008 08:26:36 GMT
Expires: Tue, 24 Mar 2009 20:07:28 GMT
X-Cache: MISS from proxy.lnet
X-Cache-Lookup: MISS from proxy.lnet:3128
Via: 1.0 proxy.lnet:3128 (squid/2.6.STABLE5)
Proxy-Connection: keep-alive
other video
GET http://myvideo-510.vo.llnwd.net/d4/movie9/84/5897511.flv HTTP/1.1
Host: myvideo-510.vo.llnwd.net
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.8.1.15) Gecko/20080612 Iceweasel/2.0.0.15 (Debian-2.0.0.15-0etch1)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Proxy-Connection: keep-alive
HTTP/1.0 200 OK
Server: Apache
Accept-Ranges: bytes
Content-Length: 21174498
Cache-Control: s-maxage=43200
X-Permitted-Cross-Domain-Policies: all
Content-Type: text/plain
Age: 8067
Date: Tue, 24 Mar 2009 19:46:52 GMT
Last-Modified: Wed, 11 Mar 2009 12:37:24 GMT
Expires: Wed, 25 Mar 2009 05:32:24 GMT
X-Cache: MISS from proxy.lnet
X-Cache-Lookup: MISS from proxy.lnet:3128
Via: 1.0 proxy.lnet:3128 (squid/2.6.STABLE5)
Proxy-Connection: keep-alive
I would like to request pandora.com, cnn.com, npr.org, dilbert.com, myspace, and facebook. This is a most impressive add-on to squid. Much bandwidth saved :-D Thanks for all of yalls efforts on this.
Also if yall get into image caching deviantart.com and google images.
Hello,
I request all site in http://www.ovguide.com/adult.html
Thank You,
Dear Kulbir,
I would like to request http://www.daum.net and http://tvpot.daum.net
Thank you
Anank
Hello Admin !
I am new to your website. I liked your software alot personally. I have not much knowledge about the linux, but i let my friends know about your software. They have good knowledge about linux OS. They helped me already in configuring FBSD for me, and i am very much happy that today they configured Video Cache Software for me. Thanks for making such a great tool.
I would like to suggest that kindly add the site www.defloration.com to the To-Do-List.
I am having a little bit problem, like Youtube, Metacafe, RedTube sites are working properly but other sites like Tube8, Porntube, dailymotion is not caching. And also the folders size is not displaying.
Kindly help me out regarding this.
Major music databases like song.pk, cooltoad & beemp3 releated sites.
Thanks alot