Videocache User Manual

Website : cachevideos.com

(Version 2.0.0)


Contents

0. Too Long; Don’t Want to Read

1. What is Videocache?

2. Support Websites

3. Required Software to run Videocache

4. Supported Platforms and Operating Systems

5. Installing or Upgrading Videocache

6. Configuring Videocache

6.a Configuring Squid Proxy Server

6.b Configuring Videocache

7. Running Videocache

7.a Update Script (vc-update)

7.b Apache Web Server

7.c Videocache Scheduler (vc-scheduler)

7.d Squid Proxy Server

8. Monitoring Videocache

9. Getting Help or Technical Support

10. Want to show some Videocache love!


0. Too Long; Don’t Want to Read

If this document is too long for you and you don’t want to read all the things and still want to get Videocache working in minimal time. Follow the steps below strictly!

  1. Decompress the videocache bundle!
    [root@white-magnet ~]$ tar -xvzf videocache-2.0.0.tar.gz
  2. Switch to the directory videocache-2.0.0
    [root@white-magnet ~]$ cd videocache-2.0.0
  3. Run install script
    [root@white-magnet videocache-2.0.0]$ bash install.sh
    If you run into any issues, Videocache will log proper error messages and will suggest a solution. Follow the steps and run install script again.
  4. Open VIdeocache configuration file located at /etc/videocache.conf and set critical options like client_email, cache_host, base_dir. These options are generally located at the top of file for your convenience.
  5. Run update script (vc-update) to update your installation in accordance with the changes you made to Videocache configuration file. Run the command as below.
    [root@white-magnet ~]$ vc-update (or /usr/sbin/vc-update if /usr/sbin/ not in path)
  6. Start or restart Apache Web Server with the following command.
    [root@white-magnet ~]$ apachectl -k restart
  7. Start or restart Videocache scheduler using the following command.
    [root@white-magnet ~]$ vc-scheduler -s restart
  8. Check status of Videocache scheduler with the following command.
    [root@white-magnet ~]$ vc-scheduler -s status
    If Videocache scheduler is not running, try restarting it. If it still doesn’t work, go ahead and check /var/log/videocache/scheduler.log for any error messages and fix them. Restart scheduler again after fixing the errors.
  9. Restart Squid Proxy Server daemon using the following command.
    [root@white-magnet ~]$ squid -k restart
  10. Navigate to Videocache log directory located at /var/log/videocache/ and check various log files to check Videocache’s activity.

TADA!!! We are up and running! Now spare some time and read about Videocache to optimize your savings!


1. What is Videocache?

Videocache (http://cachevideos.com/) is a Squid URL rewriter plugin to facilitate caching dynamic audio/video content from various websites like YouTube, Dailymotion, Metacafe etc. It can cache videos in a separate directory (other than Squid’s cache directories) in a browsable fashion and can serve the subsequent requests from the cache. Videocache helps in saving bandwidth and reducing load time of the videos. Videocache is currently used by ISPs, Universities, Schools, Cyber Cafes and Companies in various parts of the world.

NOTE: If you are new to Squid or you are willing to explore Squid in details, please check

my book Squid Proxy Server 3.1: Beginner's Guide at http://tinyurl.com/squidbook.

2. Support Websites

Below is an exhaustive list of websites partially or wholly supported by Videocache. These websites doesn’t normally contain any pornographic content.

  1. YouTube - www.youtube.com
  2. AOL - www.aol.com
  3. Bing - www.bing.com/videos/
  4. Blip - www.blip.tv
  5. Break - www.break.com
  6. CNN - www.cnn.com/video/
  7. Dailymotion - www.dailymotion.com
  8. Facebook - www.facebook.com[1]
  9. Metacafe - www.metacafe.com
  10. MySpace - www.myspace.com/video/
  11. Vimeo - www.vimeo.com
  12. Weather - www.weather.com
  13. Wrzuta - www.wrzuta.pl[2]
  14. Youku - www.youku.com

Videocache also supports caching of videos from pornographic websites mentioned in the list below. Visit these sites only if you are above 18 years (or permitted age in your country).

  1. ExtremeTube - www.extremetube.com
  2. HardSexTube - www.hardsextube.com
  3. KeezMovies - www.keezmovies.com
  4. PornHub - www.pornhub.com
  5. RedTube - www.redtube.com
  6. SlutLoad - www.slutload.com
  7. SpankWire - www.spankwire.com
  8. Tube8 - www.tube8.com
  9. Xhamster - www.xhamster.com
  10. XTube - www.xtube.com
  11. XVideos - www.xvideos.com
  12. YouPorn - www.youporn.com

3. Required Software to run Videocache

Videocache depends on a bunch of softwares. While some to these software packages are required during installation others are required to run Videocache successfully. Below is a list of packages required by Videocache.

  1. During Installation
  1. Bash shell www.gnu.org/software/bash/
  2. Wget - www.gnu.org/software/wget/
  3. Tar - www.gnu.org/software/tar/
  4. Gcc (C Compiler) - gcc.gnu.org
  1. Runtime
  1. Python (version 2.4.3 or later) - www.python.org
  2. Squid Proxy Server (version 2.6.STABLE21 or later) - www.squid-cache.org
  3. Apache Web Server - httpd.apache.org

4. Supported Platforms and Operating Systems

Videocache work on both 32bit and 64bit platforms. Videocache will run on most Linux/Unix based operating systems where the required software packages mentioned above are available. Below is a list of officially supported operating systems.

  1. Fedora
  2. CentOS
  3. Red Hat Enterprise Linux
  4. Ubuntu
  5. Debian
  6. OpenSuSE
  7. Mandriva
  8. Gentoo
  9. FreeBSD/NetBSD
  10. Slackware

Videocache may also work on other Linux/Unix distributions provided the required software packages are present.

5. Installing or Upgrading Videocache

The installation and upgrade process are same for Videocache. You just need to decompress the software bundle you have received from us and run the installer file.

NOTE: If you are upgrading your existing Videocache installation, please make sure that you take a backup of your Videocache configuration file located at /etc/videocache.conf.


Please follow the three simple steps below to install or upgrade Videocache. Before you can install or upgrade, you must be logged in as root or superuser.

  1. Decompress the Videocache software bundle videocache-2.0.0.tar.gz
    [root@white-magnet ~]$ tar -xvzf videocache-2.0.0.tar.gz
  2. Navigate to the directory videocache-2.0.0
    [root@white-magnet ~]$ cd videocache-2.0.0
  3. Run the installer file (install.sh) to install or upgrade Videocache.
    [root@white-magnet videocache-2.0.0]$ bash install.sh

That’s all you need to do to install Videocache. If there is any error during installation, Videocache will print error message and the process to diagnose it in the terminal. Please resolve the issue and rerun the installer script.

6. Configuring Videocache

Configuring Videocache involves configuring Squid and Videocache. First, we will have a look at steps to configure Squid so that it can use Videocache as a plugin.

6.a Configuring Squid Proxy Server

You need to configure your Squid Proxy Server whether you installed a fresh copy of Videocache or you are upgrading from an older version of Videocache. Videocache bundle contains two files named vc_squid_2.conf and vc_squid_3.conf. Based on the Squid version, you’ll be using one of the files mentioned above (vc_squid_2.conf for Squid 2.x and vc_squid_3.conf for Squid 3.x).


Open your Squid configuration file (generally located at
/etc/squid/squid.conf) and remove any Videocache specific configuration it may have previously. Now open the vc_squid_x.conf file from Videocache bundle and copy all its contents. Now open the Squid configuration file and paste at the top of the file. Please make sure that you paste the contents of vc_squid_x.conf at the top of your Squid configuration file squid.conf.

6.b Configuring Videocache

Videocache and its various components are managed by a single configuration file located at /etc/videocache.conf. After installing or upgrading Videocache, we need to set options to appropriate values so that Videocache can run without any issues. Videocache configuration file contains sufficient inline documentation and examples for understanding. We are providing a brief description for the options below. The options marked red must be set for Videocache to work properly.

  1. client_email
    Set this option to the email address using which you purchased Videocache license.
    Default:
    NOT SET

  1. cache_host
    The hostname or the IP address of this server. Please make sure all your clients can directly reach this server using this IP address. This IP address is used for serving the cached responses to your clients over HTTP.
    If you have Apache listening on any port other than 80, then suffix the port as well.
    Default : NOT SET
    Please don’t use http:// or slashes (/).
    Examples:
  1. cache_host = IP_ADDRESS:HTTP_PORT
  2. cache_host = 192.168.36.204
  3. cache_host = proxy.example.com
  4. cache_host = proxy.example.com:81

  1. videocache_user
    Use this option to set the user who runs Squid daemon. Normally, Videocache will be able to guess this automatically. But if it can’t guess, you should set this yourself. It's
    squid by default on RHEL/Fedora/CentOS/SuSE and proxy on Debian/Ubuntu/BSD
    Default : NOT SET
    Example:
  1. videocache_user = squid

  1. base_dir
    This option is used to define the cache directories where Videocache will store cached videos. We can specify multiple caching directories separated by ‘|’.
    Default : /var/spool/videocache/
    Examples:
  1. base_dir = /videocache/
  2. base_dir = /cache/videocache/ | /hdd2/videocache/ | /home/videocache/

  1. disk_avail_threshold
    If free space in a cache directory goes below this option’s value (in Megabytes), then Videocache will start purging old videos to make space for the new ones.
    Default: 15000
    Please don’t append MB or MegaBytes.

  1. disk_cleanup_strategy
    When all the cache directories are full, Videocache will start removing old videos to make space for the new ones. We can define Videocache’s strategy to remove old videos using this option.
    Default : 1
    Available Strategies
  1. 1 : Delete least recently used videos (preserves videos with most hits)
  2. 2 : Delete videos which are larger in size. (frees maximum disk space)

  1. enable_videocache
    This option controls the global behaviour of Videocache. If it is 0, Videocache will stop caching or serving anything. This option's value can be either 0 or 1.
    Default : 1

  1. offline_mode
    When offline mode is enabled, Videocache will serve the videos already in cache and will skip caching the new videos. If set to 0, Videocache will cache new videos and serve the already cached videos. When set to 1, Videocache will serve the already cached videos and will not cache any new videos.
    Default : 0

  1. base_dir_selection
    This option can be used to specify the algorithm which Videocache will use to select a cache directory to store the videos when more than one cache directories are used. Please select one of the values as described below.
    Default : 2
    Available Algorithms
  1. 1 : Sequential - Fill the first cache dir, then second and so on.
  2. 2 : Round Robin - Round robin among cache directories to save videos.
  3. 3 : Disk Space - Save videos to a directory with max free space at that time.

  1. enable_store_log_monitoring
    This option enables the monitoring of Squid's cache store log (
    store.log). Videocache will try to cache the videos permanently at a different location without using any upstream bandwidth. Enable this option to optimize videocache. This option’s value can be either 0 or 1.
    Please don’t forget to configure Squid to enable store.log using cache_store_log directive in Squid configuration file.
    Default: 1

  1. squid_store_log
    Full path to Squid's
    store.log determined by cache_store_log directive in Squid configuration file (squid.conf). Please make sure that you enable Squid's store log to optimize the caching capabilities of videocache. Note that videocache will use this file in read-only mode.
    Default : NOT SET

  1. this_proxy
    The proxy server setup on the machine where Videocache is being installed. This proxy server is used to transfer temporarily cached videos in Squid's cache directory to Videocache's cache directory.
    Not necessary to specify this option if you are not using enable_store_log_monitoring option.
    Default : NOT SET
    Examples:
  1. this_proxy = IP_ADDRESS:SQUID_PORT
  2. this_proxy = 127.0.0.1:3128
  3. this_proxy = proxy.example.com:3128

  1. hit_threshold
    Number of times a video should be requested before we start caching it.
    Default : 1

  1. max_cache_processes
    The maximum number of parallel cache processes allowed. If all connections are consumed, videos will be queued for caching. Don't set it too high.
    Default : 10

  1. max_cache_speed
    The maximum bandwidth allocated to a cache process. For example, when
    max_cache_speed is set to 100, a cache process can cache a video at a maximum speed of 100 kilobytes per second. Set this to zero (0) if you want a cache process to use unlimited bandwidth.
    The maximum bandwidth used by Videocache at any time will not exceed [max_cache_processes * max_cache_speed] kilobytes per second. So, you can configure these options depending on bandwidth availability.
    Examples:
  1. max_cache_speed = 100 (Please don't append KB or MB)

  1. max_cache_queue_size
    The maximum number of videos the Videocache scheduler can keep in queue for caching. Scheduler consumes some main memory or RAM (~256 bytes per video) for storing video metadata information. Please don't set
    max_cache_queue_size too high otherwise scheduler can consume significant amount of main memory.
    Default : 100000

  1. cache_period
    This option specifies the time interval when the Videocache scheduler is allowed to cache videos. You can use this option to configure Videocache to cache videos in off-peak hours so that you can provide maximum possible bandwidth to your clients in peak hours. The format for specifying
    cache_period is
  1. HH1:MM1-HH2:MM2 OR 
  2. HH1:MM1-HH2:MM2, HH3:MM3-HH4:MM4, HH5:MM5-HH6-MM6
  1. Time must be specified in 24 hour format. Also, HH1:MM1 must be less that HH2:MM2. Multiple time intervals can be specified by using comma (,) as a separator.
    Example:
  1. cache_period = 04:00-06:00, 14:00-15:00
    This will force videocache to cache videos only from 4AM to 6AM and from 2PM to 3PM.
  2. If you want videocache to cache videos only during night from 11PM to 7AM, then you'll have to specify two time intervals 23:00-23:59 and 00:00-07:00 to meet the condition that start time must be less than end time.

  1. proxy
    If requests from this machine must pass through another proxy server, then specify the proxy server using this option.
    Please specify this option when requests must pass through another proxy server.
    Default : NOT SET
    Examples:
  1. proxy = 192.168.1.2:3128
  2. proxy = proxy.example.com:3128

  1. proxy_username
    If the proxy server specified using proxy option requires authentication, please specify the username using this option.
    Default : NOT SET
    Example:
  1. proxy_username = admin

  1. proxy_passowrd
    If the proxy server specified using proxy option requires authentication, please specify the username using this option.
    Default : NOT SET
    Example:
  1. proxy_password = crypt1c

  1. max_video_size
    Videos of size more than max_video_size (MegaBytes) will not be cached. Set to 0 to disable this option. Please don’t append KB or MB.
    Default : 0

  1. min_video_size
    Videos of size less than min_video_size (MegaBytes) will not be cached. Set to 0 to disable this option. Please don’t append KB or MB.
    Default : 0

  1. force_video_size
    This option forces the
    max_video_size and min_video_size options strictly. If enabled and we try to cache a video for which Content-Length HTTP header is not provided by the web server, then videocache will not cache it. Set to 0 to disable it.
    Default : 1

  1. logdir
    Directory where videocache logs will be stored.
    Default : /var/log/videocache/

  1. logformat
    Logformat allows you to get log messages in your preferred format. The options logformat, scheduler_logformat, cleaner_logformat are applicable to main videocache log, scheduler log and cleaner log respectively.
    Default : %tl %p %s %i %w %c %v %m %d
    Use the format codes described below.
  1. %  - A literal % character
  2. ts - Seconds since epoch
  3. tu - Time in millisecond
  4. tl - Local Time
  5. tg - GMT Time
  6. p  - Process ID of the process logging the message
  7. s  - Severity level of the log message
  8. i  - Client's IP address
  9. w  - Website ID (eg. YOUTUBE/FACEBOOK/VIMEO etc.)
  10. c  - Status Code (CACHE_HIT/CACHE_MISS etc.)
  11. v  - Video ID of current video
  12. b  - Size of the video in bytes
  13. m  - Additional Message (for verbose logs)
  14. d  - Debug message (for debugging purpose)
  1. Similar options:
  1. scheduler_logformat (Default : %tl %p %s %i %w %c %v %m %d)
  2. cleaner_logformat (Default : %tl %p %s %w %c %v %m %d)

  1. timeformat
    You can use a custom format for displaying time in log messages. Use the format codes described below.
    Default: %d/%b/%Y:%H:%M:%S
  1. %a    Abbreviated weekday name (Sun, Mon, Tue, Wed, Thu, Fri, Sat)
  2. %A    Full weekday name (Sunday, Monday, ...)
  3. %b    Abbreviated month name (Jan, Feb, Mar, ...)
  4. %B    Full month name (January, February, ...)
  5. %d    Day of the month as a decimal number [01..31]
  6. %H    Hour (24-hour clock) as a decimal number [00..23]
  7. %I    Hour (12-hour clock) as a decimal number [01..12]
  8. %j    Day of the year as a decimal number [001..366]
  9. %m    Month as a decimal number [01..12]
  10. %M    Minute as a decimal number [00..59]
  11. %p    Either AM or PM
  12. %S    Second as a decimal number [00..59]
  13. %y    Year without century as a decimal number [00..99]
  14. %Y    Year with century as a decimal number

  1. enable_videocache_log
    Using this option, you can control the logging activity of the main Videocache process. When this option are set to 0, Videocache will not log anything to logfile specified using the
    logfile option. This option’s value can either be 0 or 1.
    Default : 1
    Similar options:
  1. enable_scheduler_log (Default : 1)
  2. enable_cleaner_log (Default : 1)
  3. enable_trace_log (Default : 1)

  1. logfile
    The name of main Videocache log file can be specified using logfile option.
    Please don’t specify the full path. Only specify the name of the file.
    Default : videocache.log
    Similar options:
  1. scheduler_logfile (Default : scheduler.log)
  2. cleaner_logfile (Default : cleaner.log)
  3. tracefile (Default : trace.log)

  1. max_logfile_size
    Maximum size of logfile specified using
    logfile option. The size is in MegaBytes.
    Default : 10 (Please don’t append KB or MB)
    Similar options:
  1. max_scheduler_logfile_size (Default: 10)
  2. max_cleaner_logfile_size (Default: 10)
  3. max_tracefile_size (Default: 10)

  1. max_logfile_backups
    The log files are automatically rotated once they have exceeded the
    max_logfile_size. This option specifies the number of backup files you want to keep. For example, max_logfile_backups = 2 will keep videocache.log and videocache.log.1 and videocache.log.2 as logfiles.
    Default : 10
    Similar options:
  1. max_scheduler_logfile_backups (Default : 5)
  2. max_cleaner_logfile_backups (Default : 1)
  3. max_tracefile_backups (Default : 1)

  1. enable_youtube_cache
    This option enables the caching of YouTube videos. This option's value can be either 0 or 1.
    Default : 1
    Each supported website have an option to enable or disable caching of its videos in the form enable_website_cache. You can opt to cache the websites you want by disabling the caching for other websites.

  1. enable_youtube_format_support (Option available for only YouTube)
    This options determines if Videocache will cache different YouTube video formats separately. Please select an appropriate algorithm from the listed below.
    Default : 3
    Available strategies:
  1. 1 : (disabled) Don't check for YouTube video formats. Cache one of the formats and serve it for requests for all kinds of formats.
  2. 2 : (strict) Strictly check for YouTube formats and cache all formats separately. Consumes maximum bandwidth.
  3. 3 : (approximate) Check YouTube formats but with approximation. For example, if a client asked for a video in 480p format and we already have 360p format of the same video in cache, then serve 360p format and vice-versa.

  1. enable_youtube_html5_videos (Option available for only YouTube)
    This option enables the caching of HTML5 videos from YouTube. This option's value can be 0 or 1.
    Default : 1

  1. enable_youtube_3d_videos (Option available for only YouTube)
    This option enables the caching of 3D videos from YouTube. This option's value can either be 0 or 1.
    Default : 1

  1. enable_youtube_partial_caching (Option available for only YouTube)
    This option enables the caching of several video segments used by YouTube to serve a single video. This option works only when
    enable_store_log_monitoring option is enabled. This option's value can either be 0 or 1.
    Default : 1

  1. max_youtube_video_quality (Option available for only YouTube)
    This option enforces the maximum video quality from Youtube. If a user browses a video in higher quality format, Videocache will still cache and serve the video in the format specified below or a lower quality format depending on the availability.
    Default : 720p
    Example:
  1. max_youtube_video_quality = 720p (Please don't use quotes)
  1. Available formats:
  1. 480p, 720p, 1080p, 2304p

  1. min_youtube_views (Option available for only YouTube)
    This option will help in enhancing the performance of Videocache. If
    min_youtube_views is set to 1000, then Videocache will cache a video only if it has received at least 1000 views on Youtube. Otherwise, video will not be cached. Set this to 0 to disable this option.
    Default : 100

  1. rpc_host
    XMLRPC is used for memory sharing across different instances of videocache.
    Leave these settings as it is if you don't have a fair idea of XMLRPC. This will be 127.0.0.1 in most cases.
    Default : 127.0.0.1

  1. rpc_port
    Please make sure this port is not currently in use. If it is in use by some other program, change this to some value above 1024 which is not in use by any other program.
    Default : 9100

7. Running Videocache

One we are done with configuring Videocache using the configuration file located at /etc/videocache.conf, we are all set to start Videocache. Running Videocache is running the update script to update system, starting your Apache Web Server, start your Videocache scheduler and finally starting Squid daemon. Let’s go over these steps one by one.

7.a Update Script (vc-update)

Before we go ahead and start using Videocache, we need to update our Videocache installation because we just changed some options in Videocache configuration file. To run update script, use the following command as root or superuser.

[root@white-magnet ~]$ vc-update (or /usr/sbin/vc-update if /usr/sbin/ is not in your path)

You must run update script every time you make changes to Videocache configuration file. The update script should be run before starting Videocache, Apache or Squid.

7.b Apache Web Server

One we have run update script successfully, we need to start or restart our Apache Web Server to serve  cache directories. Start or restart Apache using the following command.

[root@white-magnet ~]$ apachectl -k restart

7.c Videocache Scheduler (vc-scheduler)

Videocache scheduler schedules videos for caching at a later stage and also act as a coordinator among several Videocache processes. This is an important component and must be running all the time. To start or restart Videocache scheduler, use the following commands (use /usr/sbin/vc-scheduler command if /usr/sbin/ is not in your path)

[root@white-magnet ~]$ vc-scheduler -s start (or vc-scheduler -s restart)

Once you have started Videocache scheduler, just check the status to make sure it’s running properly using the following command.

[root@white-magnet ~]$ vc-update -s status

7.d Squid Proxy Server

At last, we will run our Squid proxy server. To run Squid we’ll use the command squid (or squid3 on some systems for Squid version 3.x). If squid executable is not in your path, please use the full path to squid command to run Squid daemon. The command is

[root@white-magnet ~]$ squid

If you want to restart Squid daemon, use the command

[root@white-magnet ~]$ squid -k restart

That’s all. We now have Videocache running on our server.


8. Monitoring Videocache

Monitoring Videocache is pretty easy. Navigate to the Videocache’s log directory which is /var/log/videocache/ by default. Check the value of logdir option in /etc/videocache.conf for the current log directory being used by Videocache. Once you are in the log directory, there are following log files for various components of Videocache.

  1. videocache.log
    The main Videocache plugin logs its activity to the
    videocache.log files. Here you’ll find the messages for clients requesting videos and Videocache try to find and serve the same from cache.
  2. scheduler.log
    Videocache scheduler which schedules and manage queue of the videos we still need to cache logs its activity to
    scheduler.log. Here you’ll find messages about what videos are currently being cached.
  3. cleaner.log
    When there is not enough space for new videos in the cache directories, Videocache starts deleting older/unused videos from caches. Videocache will log all the information with respect to these videos to the
    cleaner.log log file.
  4. trace.log
    At times, when Videocache is not lucky or encounters a problem where it doesn’t know the way out, it’ll probably crash. But we have made Videocache strong enough that it’ll not give up. It’ll just log the error messages and traceback to
    trace.log log file and will proceed to handle the next request in the queue.

9. Getting Help or Technical Support

Though we trying our best to make Videocache as self-healing and bug free as possible, there may be some issues we may have missed. If you face any issue, have any doubts, or just want to get more information about Videocache, please feel free to visit about website http://cachevideos.com/ and contact the support team. Alternatively, you can send us an email directly at support@whitemagnet.com.

10. Want to show some Videocache love!

If you want to share some suggestions, comments or criticism about Videocache or you just want to say a quick thanks for the wonders Videocache have done for you, you can use any of the methods listed below

  1. Send us an email at support@whitemagnet.com
  2. Visit our website http://cachevideos.com/ and use the contact link.
  3. Visit our forums http://cachevideos.com/forum/ and create a new thread.

Happy Caching!!! <3


[1] Does not support caching of Videos served via HTTPS.

[2] Supports caching of Audio and Video both.