You are hereForums / Support / Multiple caches

Videocache 1.9.5 is available now. This version improves support for logging and fixes video caching problems for various sites like Vimeo, Break, Tube8 etc due to change in URL patterns. Also, from this version onwards, UrlGrabber has been removed from dependencies. Check Changelog for details. If you have purchased version 1.9.x previously, you can claim free upgrade by contacting using Contact Link. Please mention the email address using which you purchased Videocache. New users proceed to download page right now, secure your copy and enjoy caching :-)

Multiple caches


10 replies [Last post]
Anonymous
Multiple caches
Printer-friendly versionSend to friend

Right now videocache will only save to one location.

Since I have many disks in my cache I would like to spread the videos across multiple disks. Is this possible?
acai berry

admin's picture
Online
Joined: Nov 2 2008

Hey,

Its already on the todo list.

Thanks anyways!

Offline
Joined: Nov 6 2008

Hi

I have 4 HDD each is 1TB, my OS is fedora 9 .
Fedora is seeing them all as one HDD which is 3.6TB

What OS are you using?

10x

admin's picture
Online
Joined: Nov 2 2008

Hi Anon,

This feature has been implemented in the latest code. Please check the code repository for the latest code.

The feature will be available in next version.

Thank You!

Offline
Joined: Feb 15 2009

I installed VideoCache 1.9 and its been running fine for a few days now. I installed it with 3 separate disks for caching. Since it was started it only saved videos to one of the disks. ( /var/cache/squid3/ ) Now that the disk is full it isn't trying to save videos to the other 2 disks. How do I get it to use the other 2 disks?

My base_dir config is this,

base_dir = /var/cache/squid1/ | /var/cache/squid2/ | /var/cache/squid3/

admin's picture
Online
Joined: Nov 2 2008

Hey,

I think thats a bug. For the time being you can get rid of the problem by setting up youtube_cache_size to half the size of disk1. In the mean time I'll try to fix it and will issue a patch for the same.

Thank you for reporting :)

admin's picture
Online
Joined: Nov 2 2008

Hello,

This bug has been fixed in the latest code upstream. Check the code repository. The next version will not have this bug.

Thank You for reporting the bug.

Offline
Joined: Feb 15 2009

I installed the latest snapshot. There was a problem with videocache logging every file that was in the cache.

My log was filling up with messages like,

2009-02-19 16:08:51,165 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/4648b3442e666584 Not a directory.
2009-02-19 16:08:51,165 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/sYtTmJXo-dg Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/0682626280c6e126 Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/53bc2d293388655a Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/EJhWRugstwg Not a directory.

I ended up just commenting out the part of the code that was doing the logging. Since doing that videocache has been running great :) It recognizes that one drive is full and is now saving videos to another drive. Thanks for adding the new disk_avail_threshold feature.

Should I have reported this in the Bug section? I wasn't sure since its snapshot code.

--- /usr/local/src/videocache/videocache/videocache.py 2009-02-19 13:31:25.000000000 -0600
+++ /usr/share/videocache/videocache.py 2009-02-19 18:30:29.000000000 -0600
@@ -320,8 +320,8 @@ def get_cache_size(cache_dir):
if os.path.isdir(dir):
for file in os.listdir(dir):
size += int(os.stat(os.path.join(dir, file))[6])
- else:
- log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
+ # else:
+ # log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
except:
log(format%(os.getpid(), '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
return -1

admin's picture
Online
Joined: Nov 2 2008

Pyromania,

Can you please share the value of your base_dir variable and the directory layout for your cache directories. This looks like a bug.

Thank You!

Offline
Joined: Feb 15 2009

This is my base_dir line,

base_dir = /var/cache/squid1/:133120 | /var/cache/squid2/:133120 | /var/cache/squid3/:133120

Its 3 150GB raptor drives. Each drive mounted on /var/cache/squid1, /var/cache/squid2 and /var/cache/squid3

Each directory just has the standard videocache directories in it,

# ls -l /var/cache/squid3
total 380
drwxr-xr-x 2 squid squid 4096 Feb 15 00:17 bliptv
drwxr-xr-x 2 squid squid 4096 Feb 14 23:33 break
drwxr-xr-x 2 squid squid 4096 Feb 14 23:11 dailymotion
drwxr-xr-x 2 squid squid 4096 Feb 12 18:23 google
drwx------ 2 squid root 16384 Jan 22 18:11 lost+found
drwxr-xr-x 2 squid squid 4096 Feb 12 18:23 metacafe
drwxr-xr-x 2 squid squid 4096 Feb 12 18:23 redtube
drwxr-xr-x 2 squid squid 4096 Feb 13 21:16 soapbox
drwxr-xr-x 2 squid squid 8192 Feb 15 03:34 tmp
drwxr-xr-x 2 squid squid 4096 Feb 15 00:18 tube8
drwxr-xr-x 2 squid squid 4096 Feb 12 18:23 tvuol
drwxr-xr-x 2 squid squid 4096 Feb 14 21:03 vimeo
drwxr-xr-x 2 squid squid 4096 Feb 12 18:23 wrzuta
drwxr-xr-x 2 squid squid 4096 Feb 15 00:25 xtube
drwxr-xr-x 2 squid squid 4096 Feb 14 23:19 youporn
drwxr-xr-x 2 squid squid 307200 Feb 15 00:26 youtube

When I first started up the snapshot code this was the start of the log,

2009-02-19 16:08:47,225 10899 - - SCHEDULEDER - Download Scheduler starting.
2009-02-19 16:08:47,225 10899 - - XMLRPCSERVER - Starting XMLRPCServer on port 9100.
2009-02-19 16:08:51,153 10896 XXX.XXX.XXX.XXX - REQUEST - http://v14.cache.googlevideo.com/videoplayback?id=f40b413d86814f54&itag=...
2009-02-19 16:08:51,157 10896 XXX.XXX.XXX.XXX f40b413d86814f54 URL_HIT YOUTUBE http://v14.cache.googlevideo.com/videoplayback?id=f40b413d86814f54&itag=...
2009-02-19 16:08:51,160 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/0f9510cdd48edfc2 Not a directory.
2009-02-19 16:08:51,160 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/-7036110663353354505 Not a directory.
2009-02-19 16:08:51,161 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/bb0f7cf7ddf72baa Not a directory.

Then the CACHE_DIR_ERR messages continued for every file in /var/cache/squid2/youtube/ and /var/cache/squid3/youtube/. The /var/cache/squid1/ directory has no videos in it so that must be why it started with /var/cache/squid2

admin's picture
Online
Joined: Nov 2 2008

Pyromania,

That was a minor bug. Please apply this patch to your videocache.py file and it'll be fine.

Please undo the changes that you have done to videocache.py first.

diff --git a/videocache/videocache.py b/videocache/videocache.py
index b13309b..f2cab26 100755
--- a/videocache/videocache.py
+++ b/videocache/videocache.py
@@ -311,8 +311,9 @@ def get_cache_size(cache_dir):
     """
     # Initialize with 4096bytes as the size of an empty dir is 4096bytes.
     size = 4096
+    pid = os.getpid()
     if not os.path.isdir(cache_dir):
-        log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', cache_dir, 'Not a directory.'))
+        log(format%(pid, '-', '-', 'CACHE_DIR_ERR', cache_dir, 'Not a directory.'))
         return -1
     try:
         for dir in os.listdir(cache_dir):
@@ -321,9 +322,9 @@ def get_cache_size(cache_dir):
                 for file in os.listdir(dir):
                     size += int(os.stat(os.path.join(dir, file))[6])
             else:
-                log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
+                log(format%(pid, '-', '-', 'CACHE_DIR_ERR', dir, 'Not a directory (2).'))
     except:
-        log(format%(os.getpid(), '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
+        log(format%(pid, '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
         return -1
     return size / (1024*1024)
 
@@ -383,7 +384,7 @@ def download_from_source(args):
         return
 
     # Check if we have enough disk space left to download more videos. 
-    if cache_size != 0 and get_cache_size(cache_dir) >= cache_size:
+    if cache_size != 0 and get_cache_size(os.path.dirname(cache_dir.rstrip('/'))) >= cache_size:
         log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \'' + cache_dir + '\' has exceeded the maximum size allowed.'))
         remove(video_id)
         return
@@ -491,13 +492,13 @@ def cache_video(client, url, type, video_id):
                 disk_stat = os.statvfs(cache_dir)
                 disk_available = disk_stat[statvfs.F_BSIZE] * disk_stat[statvfs.F_BAVAIL] / (1024*1024.0)
                 # If cache_size is not 0 and the cache directory size is more than cache_size, we are done with this cache directory.
-                if cache_size != 0 and get_cache_size(cache_dir) >= cache_size:
-                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \'' + cache_dir + '\' has exceeded the maximum size allowed.'))
+                if cache_size != 0 and get_cache_size(base_tup[0]) >= cache_size:
+                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \'' + base_tup[0] + '\' has exceeded the maximum size allowed.'))
                     # Check next cache directory
                     continue
                 # If disk availability reached disk_avail_threshold, then we can't use this cache anymore.
                 elif disk_available < disk_avail_threshold:
-                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \'' + cache_dir + '\' has reached the disk availability threshold.'))
+                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \'' + base_tup[0] + '\' has reached the disk availability threshold.'))
                     # Check next cache directory
                     continue
                 else:

Thank you for reporting the bug :)

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <img> <code>
  • Lines and paragraphs break automatically.
  • HTML tags will be transformed to conform to HTML standards.

More information about formatting options

CAPTCHA
Are you a human visitor ?
Image CAPTCHA
Enter the characters (without spaces) shown in the image.

Buy Now

Videocache Bundle (RPM and tar archive) is available. Secure your copy now by clicking the button below.
No. of Servers?
For details, check this page.

Recent comments

Spread it

Thank you for spreading the word!

Who's online

There are currently 1 user and 77 guests online.