#!/usr/bin/perl
# -*- coding: ascii -*-
###########################################################################
# clive, command line video extraction utility.
# Copyright 2007, 2008, 2009 Toni Gundogdu.
#
# This file is part of clive.
#
# clive is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# clive is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
###########################################################################
use warnings;
use strict;

use clive::App;

clive::App->main;


__END__

=head1 NAME

clive - command line video extraction utility

=head1 SYNOPSIS

clive [options]... [URL]...

=head1 DESCRIPTION

clive is a command line video extraction utility for Youtube and other
video-sharing websites. It was written to work around the Adobe Flash
plugin requirement.

clive is not an universal video extraction utility. It only works with
the video hosts that it supports.

=head1 OPTIONS

 -h, --help                     print help and exit
 -v, --version                  print version and exit
     --hosts                    print supported hosts and exit
     --home-dir=HOMEDIR         use HOMEDIR instead of $HOME
Output Options:
     --emit-csv                 emit video details in CSV to stdout
     --debug                    print cURL debug messages
 -q, --quiet                    turn off all output
 -l, --recall                   recall last input
     --recall-file=FILE         use FILE instead of default ~/.clivelast
     --stderr                   redirect all output to stderr
HTTP Options:
     --agent=STRING             identify as STRING to http server
     --connect-timeout=SECS     max time allowed connection to take
     --connect-timeout-socks=S  same as above, tries to workaround SOCKS
     --proxy=ADDR               use ADDR for http proxy
     --no-proxy                 do not use http proxy
Cache Options:
     --cache-file=FILE          use FILE instead of default ~/.clivecache
 -r, --cache-read               enable reading from cache
 -d, --cache-dump               dump cache records to stdout
     --cache-dump-format=STRING print format for dumping cache records
     --cache-grep=PATTERN       grep cache records for PATTERN
 -i, --cache-ignore-case        ignore case while matching records
 -D, --cache-remove-record      remove matched records from cache
     --cache-clear              truncate cache records
     --no-cache                 disable cache all (read and write) use
Download Options:
 -f, --format=FORMAT            extract FORMAT of video
 -O, --output-file=FNAME        write video to file
 -n, --no-extract               do not extract any videos
 -c, --continue                 continue partially downloaded file
     --save-dir=DIR             save video files to DIR
     --cclass=CLASS             use character CLASS to filter titles
 -C, --no-cclass                do not apply character class
     --filename-format=STRING   use STRING to format output filename
     --exec=CMD                 run CMD subsequently after file transfer
     --stream=PERCENT           run stream command (below) at PERCENT
     --stream-exec=CMD          stream command to run
     --limit-rate=AMOUNT        limit transfer rate to AMOUNT (KB/s)
     --stop-after=SIZE|PERCENT  stop file transfer after SIZE or PERCENT
 -R, --raw                      process video page html on as-is basis

See clive manual page for EXAMPLES.

=head1 OPTION SYNTAX

You may freely mix different option styles and specify options after
the command line arguments, e.g.:
  % clive -c URL --format=best

You may also put several options together that do not require arguments:
  % clive -cnrf best URL

Note that the "dashed" options have aliases available which are not
documented in the L</OPTIONS> section. For example:
  % clive --no-extract --no_extract --noextract
  % clive --cache-read --cache_read --cacheread

=head1 OPTION DESCRIPTIONS

=over 4

=item B<-h, --help>

Print help and exit.

=item B<-v, --version>

Print version and exit.

=item B<--hosts>

Print supported hosts, with precompiled regexps used to identify links,
and exit.

=item B<--home-dir>=I<homedir>

Use I<homedir> instead of the default $HOME (and $CLIVE_HOME) environment
variable.

=back

B<Output options>

=over 4

=item B<--emit-csv>

Print (emit) video details in CSV format to standard output.
Implies --no-extract.

=item B<--debug>

Print cURL debug (or verbose) messages to standard error.

=item B<-q, --quiet>

Turn off all output to standard output and error.

=item B<-l, --recall>

Recall (or reuse) the last input.

=item B<--recall-file>=I<file>

Use I<file> instead of I<homedir>/.clivelast.

=item B<--stderr>

Direct all output to standard error.

=back

B<HTTP Options>

=over 4

=item B<--agent>=I<string>

Identify clive as I<string> to HTTP servers. Defaults to "Mozilla/5.0".

=item B<--connect-timeout>=I<seconds>

Maximum time in I<seconds> allowed for connection to take. Defaults to 30.

=item B<--connect-timeout-socks>=I<seconds>

Same as above but tries to workaround the SOCKS proxy bug in cURL. Defaults
to 30.

=item B<--proxy>=I<address>

Use I<address> for HTTP proxy. Example: "http://foo:1234".

=item B<--no-proxy>

Disable HTTP proxy use even if http_proxy environment variable is set.

=back

B<Cache Options>

=over 4

=item B<--cache-file>=I<file>

Use I<file> instead of I<homedir>/.clivecache.

=item B<-r, --cache-read>

Read video details from cache record if it exists. Allows clive to
skip fetching the video page again. See L</CACHE> section for more
on this.

=item B<-d, --cache-dump>

Dump cache records to standard output.

=item B<--cache-dump-format>=I<format-string>

Used to format the output of the above. Defaults to "%n: %t [%f, %mMB]".

Example:
  % clive --cache-dump --cache-dump-format="%d: %t"

Supported format specifiers:
  %t .. video page title
  %i .. video id
  %h .. video host
  %l .. video file length (bytes)
  %m .. video file length (MB)
  %d .. date (last update)
  %T .. time (last update)
  %s .. time stamp (same as "%d %T")
  %f .. video file format
  %n .. index

=item B<--cache-grep>=I<pattern>

Grep stored cache records for I<pattern>. See also L</EXAMPLES - ADVANCED USE>.

=item B<-i, --cache-ignore-case>

Ignore case-differences while matching records.

=item B<-D, --cache-remove-record>

Remove matched records from cache.

=item B<--cache-clear>

Truncate cache records.

=item B<--no-cache>

Disable all (read and write) cache use.

=back

B<Download Options>

=over 4

=item B<-f, --format>=I<format>

Extract I<format> of the video. If I<format> is set to I<best>, clive
will attempt to extract the best video quality.

Note that the I<format> is strictly host specific. See the L</FORMATS>
section for more on this.

=item B<-O, --output-file>=I<filename>

Write video to I<filename>. Not to be used when extracting
multiple videos.

=item B<-n, --no-extract>

Do not extract the video.

=item B<-c, --continue>

Continue partially downloaded video file.

=item B<--sav-dir>=I<dir>

Save extracted videos to I<dir>.

=item B<--cclass>=I<class>

Use character-I<class> to filter video page titles. Defaults to "\w".
Refer to the Perl regular expressions (character classes) for more on
this.

=item B<-C, --no-cclass>

Negates the use of the character-class.

=item B<filename-format>=<format-string>

Use <format-string> to format output video filenames. Defaults to "%t.%s".

Supported format specifiers:
  %t .. video page title (after applying character-class filter)
  %s .. video file suffix (e.g. "flv")
  %i .. video id
  %h .. video host

=item B<--exec>=I<command>;

Execute I<command> subsequently after video file transfer. Optional
arguments may be passed to the command. The expression must be
terminated by a semicolon (";"). If the specifier "%i" appears
anywhere in the I<command>, it is replaced by the pathname of the
extracted video file.

=item B<--exec>=I<command>+

Same as above but "%i" is replaced with as many pathnames as
possible for the invocation of I<command>.

=item B<--stream>=I<percent>

Execute --stream-exec=I<command> when file transfer reaches
I<percent>.

=item B<--stream-exec>=I<command>

Execute (fork a child process) I<command> while transferring
video file. This "simulates" streaming the media but does so
without checking for buffer underruns so make sure you set
--stream=I<percent> high enough and that you have a fast
internet connection.

clive will not attempt to re-execute the command if it
terminates before the file transfer finishes.

clive will wait that the child process terminates before
it moves on to extract another file or exits if there
are not any left.

Note that some video file formats (namely Google mp4) are
known to B<not> to work with this feature.

=item B<--limit-rate>=I<amount>

Limit transfer rate to I<amount> KB/s.

=item B<--stop-after>=I<size|percent>

Stop file transfer after I<size> or I<percent>. The value must
be terminated by either '%' or 'M'.

=item <-R, --raw>

Process video page HTML as it is. Technically, this means that
clive will not try to decode the HTML using Encode::decode_utf8.
This may be useful with some of the supported host such as
Cctv which uses GBK as its HTML encoding.

=back

=head1 EXAMPLES - BASIC USE

=over 4

=item clive "http://youtube.com/watch?v=3HD220e0bx4"

Extracts video (flv) from the above video page link. You
can then play the flv video file in a media player.

=item cat E<gt> url.lst

  http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips
  http://youtube.com/watch?v=3HD220e0bx4
  http://break.com/index/beach-tackle-whip-lash.html
  http://www.liveleak.com/view?i=704_1228511265

=item cat url.lst | clive

You can feed clive multiple video page links like this
or via command line:

=item clive URL URL2 URL3 URL4

When you are using the pipes, be sure to separate each link
with a newline.

=item xclip -o | clive

Many X clipboard utilities exist. This example uses C<xclip(1)>
and a pipe to paste the contents of the clipboard to clive.

=item clive -l

Recall last video page link input.

=back

=head1 EXAMPLES - ADVANCED USE

=over 4

=item clive -f best "http://youtube.com/watch?v=3HD220e0bx4"

Extract the best format of the video.

=item clive -r -f best "http://youtube.com/watch?v=3HD220e0bx4"

Causes clive to skip re-fetching the video page and parsing it.
Instead it reuses the record saved in the cache from the previous
time.

=item clive --cache-dump

Dump cache records to stdout. You can use --cache-dump-format
to format the output.

=item clive -ig 3hd2

Grep for "3hd2" pattern in cache records. If pattern
matches, clive continues to extract the matched videos.
Note the use of "-i" (--cache-ignore-case).

=item clive -ig 3hd2 -D

Same as above but removes the record from cache instead of
extracting the video.

=item clive --exec="ffmpeg -i %i %i.mp3;" URL

Extract video and use C<ffmpeg(1)> to copy audio from it
to mp3 file.

=item clive --stream-exec="mplayer -really-quiet %i"
    --stream=25 URL

Start playing the video being extracted when the transfer
reaches 25% complete.

=back

=head1 FORMATS

clive extracts flv (typically 320x240) by default from all
supported websites. Some of them support also other formats.

=over 4

=item B<youtube.com>

=item B<last.fm>

Format: (flv|fmt17|fmt18|fmt22|fmt35)

flv (fmt34) and fmt18 (mp4) are usually available. Others may be
available. At the time of writing this, the following formats
are recognized by both clive and Youtube:

  fmt22 .. mp4    (1280x720) (HD)
  fmt35 .. flv     (640x380) (HQ)
  fmt18 .. mp4     (480x360)
  flv   .. fmt34   (320x180)
  fmt17 .. 3gp     (176x144)

Some of the videos available at last.fm are actually Youtube
videos. clive can handle such video links.

=item B<video.google.com>

Format: (flv|mp4)

mp4 format is available for a limited number of videos.

=item B<dailymotion.com>

Format: (flv|spak-mini|vp6-hq|vp6-hd|vp6|h264)

The HD and HQ videos may not always be available.

  ON2-1280x720 (vp6-hd)
  ON2-848x480  (vp6-hq)
  H264-512x384 (h264)
  ON2-320x240  (vp6)
  FLV-320x240  (flv/spark)
  FLV-80x60    (spak-mini)

=item B<vimeo.com>

Format: (flv|hd)

HD should be available for the vimeo.com/hd channel videos at least.
Note that "flv" only means "default" here, as some of the hosted
videos are encoded (as default) in other video formats such as
"mp4" rather than "flv".

For further reading:
  http://vimeo.com/help/hd

=item B<evisor.tv>

=item B<liveleak.com>

=item B<tv.cctv.com>

=item B<sevenload.com>

=item B<break.com>

=item B<redtube.com>

Format: flv

=back

=head1 FILES

=over 4

=item ~/.cliverc

User configuration file. You can append the command line options to
the config file, e.g.:
  % cat >> ~/.cliverc
  -f best
  --proxy=http://foo:1234

=item ~/.clivelast

File containing last user input.

=item ~/.clivecache

BerkeleyDB based cache containing the records of fetched and
parsed video pages.

=item Notes

The default path is $HOME (or ~). clive will also respect the CLIVE_HOME
environment variable.

=back

=head1 CACHE


The purpose of the cache is to allow clive to skip fetching
and parsing the video page again. It does not contain any
actual video data so one should not expect to recover a deleted
video file from the cache. Only some of the parsed details
are stored as records to the cache.

By now, it is should be a well known fact that the cache fails
with some of the supported hosts. For example Youtube video links
expire after some time, this causes the re-extraction to fail if
the cached video link is used later again.

This was the main reason why in 2.2.0 reading from cache was
disabled by default. Many users reported the reuse of expired
video links as a bug previously even though it was well documented
in the manual page explaining that most of the HTTP 403/404 errors
were actually caused by expired video links.

It is, of course, still possible to read from cache. You can
enable this by invoking the --cache-read option. This causes
clive to look up a saved cache record and reuse the stored
video details if they are found instead of fetching the video
page.

The use of the cache can be disabled with the --no-cache option.
This disables both read and write. Note that if the BerkeleyDB Perl
module is not installed, clive will not use the cache.

See also the --cache-grep option.

=head1 BUGS

You are welcome to submit bug reports via the Issue tracker at:
  <http://code.google.com/p/clive/issues/>

=head1 EXIT STATUS

clive exits 0 on success, and E<gt>0 if an error occurs.

 CLIVE_OK          = 0
 CLIVE_NOTHINGTODO = 1    # file already retrieved
 CLIVE_NOSUPPORT   = 2    # host not supported
 CLIVE_READ        = 3    # file open/read error
 CLIVE_GREP        = 4    # grep: nothing matched in cache
 CLIVE_OPTARG      = 5    # invalid option argument
 CLIVE_SYSTEM      = 6    # system call failed (e.g. fork)
 CLIVE_REGEXP      = 7    # regexp pattern matching failed
 CLIVE_FORMAT      = 8    # requested format unavailable
 CLIVE_NET         = 9    # network error
 CLIVE_STOP        = 10   # --stop-after

=head1 OTHER

Project page:
  <http://code.google.com/p/clive/>

Front-end (GUI):
  <http://code.google.com/p/abby/>

Additional utilities (clive-utils):
  <http://code.google.com/p/clive-utils/>

Development repository can be cloned with:
  % git clone git://repo.or.cz/clive.git

Release announcements:
  <http://freshmeat.net/projects/clive/>

=head1 HISTORY

clive was originally written in Python but later (2.0.0)
rewritten in Perl. The project started as a workaround
to Youtube Adobe flash-plugin requirement which was
poorly supported on Unix-like systems.

=head1 AUTHOR

Toni Gundogdu <legatvs@gmail.com>

Thanks to all those who have contributed to the project over the
years by sending patches, reporting bugs and writing feedback.
You know who you are.

=cut
