Support Queries

Multiple caches

Right now videocache will only save to one location.

Since I have many disks in my cache I would like to spread the videos across multiple disks. Is this possible?

acai berry

10 Answers

Hey,

Its already on the todo list.

Thanks anyways!

Hi

I have 4 HDD each is 1TB, my OS is fedora 9 .
Fedora is seeing them all as one HDD which is 3.6TB

What OS are you using?

10x

Hi Anon,

This feature has been implemented in the latest code. Please check the code repository for the latest code.

The feature will be available in next version.

Thank You!

I installed VideoCache 1.9 and its been running fine for a few days now. I installed it with 3 separate disks for caching. Since it was started it only saved videos to one of the disks. ( /var/cache/squid3/ ) Now that the disk is full it isn't trying to save videos to the other 2 disks. How do I get it to use the other 2 disks?

My base_dir config is this,

base_dir = /var/cache/squid1/ | /var/cache/squid2/ | /var/cache/squid3/

Hey,

I think thats a bug. For the time being you can get rid of the problem by setting up youtube_cache_size to half the size of disk1. In the mean time I'll try to fix it and will issue a patch for the same.

Thank you for reporting :)

Hello,

This bug has been fixed in the latest code upstream. Check the code repository. The next version will not have this bug.

Thank You for reporting the bug.

I installed the latest snapshot. There was a problem with videocache logging every file that was in the cache.

My log was filling up with messages like,

2009-02-19 16:08:51,165 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/4648b3442e666584 Not a directory.
2009-02-19 16:08:51,165 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/sYtTmJXo-dg Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/0682626280c6e126 Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/53bc2d293388655a Not a directory.
2009-02-19 16:08:51,166 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/EJhWRugstwg Not a directory.

I ended up just commenting out the part of the code that was doing the logging. Since doing that videocache has been running great :) It recognizes that one drive is full and is now saving videos to another drive. Thanks for adding the new disk_avail_threshold feature.

Should I have reported this in the Bug section? I wasn't sure since its snapshot code.

--- /usr/local/src/videocache/videocache/videocache.py  2009-02-19 13:31:25.000000000 -0600
+++ /usr/share/videocache/videocache.py 2009-02-19 18:30:29.000000000 -0600
@@ -320,8 +320,8 @@ def get_cache_size(cache_dir):
             if os.path.isdir(dir):
                 for file in os.listdir(dir):
                     size += int(os.stat(os.path.join(dir, file))[6])
-            else:
-                log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
+         #   else:
+         #       log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
     except:
         log(format%(os.getpid(), '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
         return -1

Pyromania,

Can you please share the value of your base_dir variable and the directory layout for your cache directories. This looks like a bug.

Thank You!

This is my base_dir line,

base_dir = /var/cache/squid1/:133120 | /var/cache/squid2/:133120 | /var/cache/squid3/:133120

Its 3 150GB raptor drives. Each drive mounted on /var/cache/squid1, /var/cache/squid2 and /var/cache/squid3

Each directory just has the standard videocache directories in it,

# ls -l /var/cache/squid3
total 380
drwxr-xr-x 2 squid squid   4096 Feb 15 00:17 bliptv
drwxr-xr-x 2 squid squid   4096 Feb 14 23:33 break
drwxr-xr-x 2 squid squid   4096 Feb 14 23:11 dailymotion
drwxr-xr-x 2 squid squid   4096 Feb 12 18:23 google
drwx------ 2 squid root   16384 Jan 22 18:11 lost+found
drwxr-xr-x 2 squid squid   4096 Feb 12 18:23 metacafe
drwxr-xr-x 2 squid squid   4096 Feb 12 18:23 redtube
drwxr-xr-x 2 squid squid   4096 Feb 13 21:16 soapbox
drwxr-xr-x 2 squid squid   8192 Feb 15 03:34 tmp
drwxr-xr-x 2 squid squid   4096 Feb 15 00:18 tube8
drwxr-xr-x 2 squid squid   4096 Feb 12 18:23 tvuol
drwxr-xr-x 2 squid squid   4096 Feb 14 21:03 vimeo
drwxr-xr-x 2 squid squid   4096 Feb 12 18:23 wrzuta
drwxr-xr-x 2 squid squid   4096 Feb 15 00:25 xtube
drwxr-xr-x 2 squid squid   4096 Feb 14 23:19 youporn
drwxr-xr-x 2 squid squid 307200 Feb 15 00:26 youtube

When I first started up the snapshot code this was the start of the log,

2009-02-19 16:08:47,225 10899 - - SCHEDULEDER - Download Scheduler starting.
2009-02-19 16:08:47,225 10899 - - XMLRPCSERVER - Starting XMLRPCServer on port 9100.
2009-02-19 16:08:51,153 10896 XXX.XXX.XXX.XXX - REQUEST - http://v14.cache.googlevideo.com/videoplayback?id=f40b413d86814f54&itag=34&ip=XXX.XXX.XXX.XXX&region=0&signature=763AF9C2DE878D696C04A8B3E74FD2847C32E952.BB18953521A625C0427E2B7DCC5D0CD6845012CD&sver=2&expire=1235102930&key=yt1&ipbits=0
2009-02-19 16:08:51,157 10896 XXX.XXX.XXX.XXX f40b413d86814f54 URL_HIT YOUTUBE http://v14.cache.googlevideo.com/videoplayback?id=f40b413d86814f54&itag=34&ip=XXX.XXX.XXX.XXX&region=0&signature=763AF9C2DE878D696C04A8B3E74FD2847C32E952.BB18953521A625C0427E2B7DCC5D0CD6845012CD&sver=2&expire=1235102930&key=yt1&ipbits=0
2009-02-19 16:08:51,160 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/0f9510cdd48edfc2 Not a directory.
2009-02-19 16:08:51,160 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/-7036110663353354505 Not a directory.
2009-02-19 16:08:51,161 10896 - - CACHE_DIR_ERR /var/cache/squid2/youtube/bb0f7cf7ddf72baa Not a directory.

Then the CACHE_DIR_ERR messages continued for every file in /var/cache/squid2/youtube/ and /var/cache/squid3/youtube/. The /var/cache/squid1/ directory has no videos in it so that must be why it started with /var/cache/squid2

Pyromania,

That was a minor bug. Please apply this patch to your videocache.py file and it'll be fine.

Please undo the changes that you have done to videocache.py first.

diff --git a/videocache/videocache.py b/videocache/videocache.py
index b13309b..f2cab26 100755
--- a/videocache/videocache.py
+++ b/videocache/videocache.py
@@ -311,8 +311,9 @@ def get_cache_size(cache_dir):
     """
     # Initialize with 4096bytes as the size of an empty dir is 4096bytes.
     size = 4096
+    pid = os.getpid()
     if not os.path.isdir(cache_dir):
-        log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', cache_dir, 'Not a directory.'))
+        log(format%(pid, '-', '-', 'CACHE_DIR_ERR', cache_dir, 'Not a directory.'))
         return -1
     try:
         for dir in os.listdir(cache_dir):
@@ -321,9 +322,9 @@ def get_cache_size(cache_dir):
                 for file in os.listdir(dir):
                     size += int(os.stat(os.path.join(dir, file))[6])
             else:
-                log(format%(os.getpid(), '-', '-', 'CACHE_DIR_ERR', os.path.join(cache_dir, dir), 'Not a directory.'))
+                log(format%(pid, '-', '-', 'CACHE_DIR_ERR', dir, 'Not a directory (2).'))
     except:
-        log(format%(os.getpid(), '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
+        log(format%(pid, '-', '-', 'CACHE_SIZE_ERR', cache_dir, 'Error occurred while calculating the size of directory.'))
         return -1
     return size / (1024*1024)

@@ -383,7 +384,7 @@ def download_from_source(args):
         return

     # Check if we have enough disk space left to download more videos. 
-    if cache_size != 0 and get_cache_size(cache_dir) >= cache_size:
+    if cache_size != 0 and get_cache_size(os.path.dirname(cache_dir.rstrip('/'))) >= cache_size:
         log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \\'' + cache_dir + '\\' has exceeded the maximum size allowed.'))
         remove(video_id)
         return
@@ -491,13 +492,13 @@ def cache_video(client, url, type, video_id):
                 disk_stat = os.statvfs(cache_dir)
                 disk_available = disk_stat[statvfs.F_BSIZE] * disk_stat[statvfs.F_BAVAIL] / (1024*1024.0)
                 # If cache_size is not 0 and the cache directory size is more than cache_size, we are done with this cache directory.
-                if cache_size != 0 and get_cache_size(cache_dir) >= cache_size:
-                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \\'' + cache_dir + '\\' has exceeded the maximum size allowed.'))
+                if cache_size != 0 and get_cache_size(base_tup[0]) >= cache_size:
+                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \\'' + base_tup[0] + '\\' has exceeded the maximum size allowed.'))
                     # Check next cache directory
                     continue
                 # If disk availability reached disk_avail_threshold, then we can't use this cache anymore.
                 elif disk_available < disk_avail_threshold:
-                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \\'' + cache_dir + '\\' has reached the disk availability threshold.'))
+                    log(format%(pid, client, video_id, 'CACHE_FULL', type, 'Cache directory \\'' + base_tup[0] + '\\' has reached the disk availability threshold.'))
                     # Check next cache directory
                     continue
                 else:

Thank you for reporting the bug :)