You are hereForums / Support / Adding throttle [Solved]

Version 1.9.9 is out. We have added support for Youtube mobile videos, Facebook, AOL and MySpace videos as well. Fixed caching issues for some already supported websites. Options to control cache directory selection algorithm, bandwidth usage by videocache, etc. More robust videocache scheduler with a lot more options and features. Have a look at changelog for more details. If you have purchased Videocache in last one year, Contact Us to claim your free upgrade. Please mention the email address using which you purchased Videocache. New users proceed to Pricing Page right now, secure your copy and enjoy caching :-) Also, check our new and simplified upgrade policy :-)

Adding throttle [Solved]


16 replies [Last post]
Offline
Joined: Feb 16 2009
Adding throttle [Solved]
Printer-friendly versionSend to friend

Is it possible to change videocache.py to minimize the amount of traffic requested. For now it's using all available bandwidth, but i only want it use half for downloading objects to cache.
I saw that it's possible to throttle urlgrabber to the speed that i want, but i don't know for sure how to do it.

admin's picture
Offline
Joined: Nov 2 2008

Hello,

First of all, let me tell you that you are 50th user on our forum :D

You can control the amount of bandwidth used by videocache, using max_parallel_downloads option in /etc/videocache.conf. We have not implemented throttling using url grabber yet. We may consider as a feature for the future.

Thank you for considering videocache :)

Offline
Joined: Feb 16 2009

well, but with max_parallel_downloads the problem persist, even if i define only 1 download max. the connection will attempt to retrieve the file at full speed, witch will consume all the bandwidth. This is being a big issue because we cannot allow that if a person request a 100Mb file, the internet for everyone get slow for a couple of minutes.
What i've seen is that it's possible to implement it. with your permission, i would like to starting to mess with the code to see if i can do something about it. (i never programmed in python, but what the heck... :)   )

admin's picture
Offline
Joined: Nov 2 2008

Hi!

If you really want fine grained control over bandwidth used by videocache for downloading, you can look-up delay pools for squid. First explore delay pools.

Now, considering you know delay pools well, create a delay pool for your proxy server itself (because videocache is using your proxy server to download videos) and limit the bandwidth there. With delay pools, you can limit bandwidth using every possible acl offered by squid :D

So, instead of messing up with python code, invest time in delay pools.

But if you still feel you want scratch videocache code, here is the code repository :D

Thank you for using videocache :D

Offline
Joined: Feb 16 2009

Hi,
After spending a couple of hours seing all the code, and thinking on how little i knew on python :) , i found how to control it. I will post it here the how to. if you wish, you could add it to the instalation / configuration manual

First, find the path of the urlgrabber in you system.
for me (debian etch x86_64)

/var/lib/python-support/python2.4/urlgrabber
on other systems, find the directory by typing
find / -name urlgrabber
Edit the file grabber.py and in the "throttle = 0", place the speed for each download.
 

admin's picture
Offline
Joined: Nov 2 2008

Hi!

That was a quick hack!! Anyways you got it working now. I'll try to get into core code in next version if possible :) Thank you digging that out for us :D

sirkikee (not verified)

I solved this with delay_pools at 127.0.0.1 and works fine for me...

admin's picture
Offline
Joined: Nov 2 2008

Sirkikee,

Can you please post your delay pools configuration on forum, so that other users can benefit from it?

Thanks in advance :)

sirkike's picture
Offline
Joined: Feb 6 2009

256k of bandwidth for download and global bursts of 512k
acl squidlocal src 127.0.0.1
delay_pools 1
delay_class 1 1
delay_parameters 1 32768/65536
delay_access 1 allow squidlocal
delay_access 1 deny all

sirkike's picture
Offline
Joined: Feb 6 2009

checking the delay_pool, i found this... is very strange...
http://www.elblogmio.com.ar/wp-content/uploads/2009/02/squidlocal_pool.jpg

the delay_pool works fine, but... the cached files are accessed again through squid, but not all files, i try another files that are cached and do not are accessed through squid.

perhaps I should use always_direct for public IP of cache_host ?

admin's picture
Offline
Joined: Nov 2 2008

Sirkike,

I think always direct would be a nice hack for this. Please try and let us know.

Thanks in advance :)

sirkike's picture
Offline
Joined: Feb 6 2009

Don't works, is still looking for the videos through squid cache. I think i need to put the cache_host to the local lan IP of server, and not the public IP.

admin's picture
Offline
Joined: Nov 2 2008

Sirkike,

I think you should try going this way.

1. Set cache_host to your local lan IP (and not 127.0.0.1).
2. Use always_direct option for local lan IP.
2. Now allow 127.0.0.1 for internet access via your proxy.
3. Set delay_pools for 127.0.0.1 and not the lan IP.

I think this setup should work fine.

Thank You!

sirkike's picture
Offline
Joined: Feb 6 2009

Yes... that setup works fine, but... i need put cache_host at public IP. :P

admin's picture
Offline
Joined: Nov 2 2008

Sirkike,

Well, then follow the same procedure replacing lan IP with your wan IP and using always_direct for your wan IP.

Thank You!

sirkike's picture
Offline
Joined: Feb 6 2009

Yes, yes it did. But like I said earlier, even if you use always_direct anyway when you order the video on the public address 200.XXX/videocache/0/etc passes through the cache 127.0.0.1 as pictured. Still, no matter ... not all the videos, so no problem.

Let me tell you've done a great job, this utility is very useful.

admin's picture
Offline
Joined: Nov 2 2008

Sirkike,

Well, then it still remains as a problem. We'll see if someone else figure it out and write here.

Thanks for the compliments.

Thank you for using videocache :)

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <code>
  • Lines and paragraphs break automatically.
  • HTML tags will be transformed to conform to HTML standards.

More information about formatting options

CAPTCHA
Are you a human visitor ?
Image CAPTCHA
Enter the characters (without spaces) shown in the image.

My Book on Squid

Purchase/Renew

One License - USD $399
One License - USD $349*
Validity - One Year
Need more? Contact Us
*Includes free upgrades and support for one year.



For details, check pricing.

Recent comments

Who's online

There are currently 0 users and 100 guests online.