
Rubicon - a testsuite for Ruby
==============================

This is **Rubicon**, a test suite for Ruby.

Rubicon was originally written by Dave Thomas & Andy Hunt (for more
info see the History_ section below). Now Rubicon is maintained as
the project rubytests_ at RubyForge_.

Rubicon provides testcases for the core Ruby language and for the
built-in classes. We try to track the evolution of Ruby as it is
implicitly specified by the C-implementation of the language.
Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite with
very few errors (no errors at all on FreeBSD). There are still some
errors on other platforms, but we are working on resolving these
issues (see the Issues_ section below).

Rubicon is far from complete: more tests need to be written, and
existing tests need to be refined. If you want to help improve
Rubicon, see the Contributing_ section below.

Rubicon is also used by the team working on a Java implementation of
Ruby (see http://jruby.sourceforge.net/).


Requirements for running Rubicon
--------------------------------

In order to use Rubicon you need:

- a Ruby interpreter to test, e.g. Ruby_ 1.8.2 or JRuby_ version 0.8.0.

- the `Test::Unit`_ framework. Recent versions of Ruby include
  Test::Unit, but if you want to test an older version of Ruby or a
  new fancy Ruby interpreter written in Haskell :) you have to make
  sure you have a working copy of Test::Unit.

- a copy of this Rubicon testsuite.

- a C compiler and a "make" program. The C compiler is used to build a
  couple of "utilities" invoked by the Rubicon tests. The "make"
  program is not strictly necessary: it is just used to build the
  utilities (located in the ``util`` subdirectory).

Rubicon is written to be able to test different versions of
Ruby. Recently much focus has been on getting Ruby 1.8.2 to work. Some
older versions should also work (maybe with more errors). But
maintaining a testsuite for a "moving target" like Ruby can be a bit
tricky sometimes.

Rubicon can be run on different OS-es. It has been used to test Ruby
1.8.2 on at least Linux, FreeBSD, Solaris, Mac OS-X and Windows-XP.
Some of the tests are of course OS-dependent, so the testsuite give
slightly different results on the different platforms.

Getting Rubicon
---------------

You can get Rubicon in a number of ways:

- via CVS from the archives at rubyforge.org::

      $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests login
      $ cvs -d :pserver:anonymous@rubyforge.org:/var/cvs/rubytests co rubicon

  If you need more details about how to setup CVS access, see the "CVS
  Repository" link on the rubytests_ page.

- fetching a release from the "Files" tab at rubytests_.  The latest
  release may be a bit behind the "bleeding edge" in the CVS archives,
  but if you want a well-defined version, this may be the right choice.

- getting a tar-file snapshot via the ViewCvs interface to the CVS
  archives. Look at the bottom of the page at:
  http://rubyforge.org/cgi-bin/viewcvs.cgi/rubicon/?cvsroot=rubytests

Installation
------------

Once you have downloaded and unpacked the Rubicon files, you don't need
to do anything more. Just start using it.


Structure
---------

The tests for the builtin classes/modules are in the subdirectory
``builtin``, and those for the language are in ``language``. Over time,
different directories may be added with additional sets of tests
(networking, for example).

In order to run the tests, we need a small amount of support code,
written in C. This is in the ``util`` directory.

Running
-------

There are two makefiles in Rubicon: one named ``GNUMakefile`` and
another named ``Makefile``. The ``GNUMakefile`` should work on Linux
and other UN*X variants. If your ``make`` program is not GNU Make (for
example on Solaris or FreeBSD), you have to explicitly specify the
name of the makefile. The ``Makefile`` is for Windows. In summary::

    $ make ARGUMENTS                      # UN*X & GNU make

    $ make -f GNUMakefile ARGUMENTS       # UN*X & non-GNU make

    $ nmake ARGUMENTS                     # Windows & Visual Studio



In order to run the Rubicon tests, you have do do:

1) build the "utilities" in the ``util`` directory::

       $ make util
       cc     util/checkstat.c   -o util/checkstat
       cc     util/test_touch.c   -o util/test_touch


2) choose one of the possible "entry points" to the tests::

      # all Rubicon tests
      $ ruby AllTests.rb

      # all in the "builtin" directory
      $ cd builtin
      $ ruby AllBuiltinTests.rb

      # all in the "language" directory
      $ cd language
      $ ruby AllLanguageTests.rb

      # individual testcase file
      $ cd builtin
      $ ruby Test<classname>.rb       # e.g. TestArray.rb

      # individual testcase file
      $ cd language
      $ ruby Test<feature>.rb

3) choose a Ruby interpreter to test. The simplest way is to make sure
   the desired version of Ruby is in your ``PATH`` (as in the example
   above). But but you can also specify the Ruby interpreter with an
   explicit path, like::

       $ /somewhere/ruby-1.8.5/bin/ruby AllTests.rb
       ...
       $ /somewhere/ruby-1.8.6/bin/ruby AllTests.rb

   This way you will be able to easily test different Ruby versions
   without having to adjust the PATH. The testcases that spawn another
   Ruby instance makes sure that they start **the same** Ruby
   interpreter as the one used at the top level.

4) You can also run individual tests in a testcase file by giving the
   name of the test method as an argument on the command line::

       $ ruby TestBignum.rb test_OR


Mailing lists
-------------

There are two mailing lists for Rubicon at RubyForge:

- "rubytests-devel" is for general discussions about Rubicon.

- "rubytests-commit" is a readonly list. For each CVS commit to the
  Rubicon archives, a mail is sent to this list.

Look at the "Mailing Lists" link on the rubytests_ page for more
info about these mailing lists.


Contributing
------------

If you want to help improve the tests in Rubicon, join the mailing
list where discussions about Rubicon take place. If you want to submit
changes, we would appreciate context diffs (e.g. produced with
``cvs diff -u``).

If you are (or will become) a regular contributor you should also
become a member of the rubytests_ project at RubyForge_.
That way you can get write access to the CVS archives, and commit
changes yourself.

There are many things that can be done to improve Rubicon:

- add tests for methods that are not currently tested

- strengthen existing tests

- run the tests on more platforms, and adjust the testcases if
  necessary

- ...

Environmental Tests
-------------------

*TODO: this section need to be updated, and expanded ...*

Some tests depend on the environment in which they run. Guard
these with appropriate conditions:

- if your test depends on the Ruby version, using the Version class 
  methods. Always passing in either a string or a range of strings in 
  the form "1.6.2".   Version is defined in ``rubicon.rb``. ::

      Version.in("1.6.3".."1.6.6") do
        # stuff that is meant to work in Ruby versions
        # 1.6.3 to 1.6.6
      end

      Version.greater_than("1.6.2") do
        # code that depends on ruby version greater than 1.6.2
      end
      
- if your test depends on the operating system type, use the
  comparison classes. There's a class defined for each target
  operating system and environment (for example, Linux has one
  class, and Windows has MsWin32, Cygwin, and MinGW). These
  classes define class methods to allow you to include and exclude
  blocks of code depending on the OS that Rubicon is running
  under. ::

      MsWin32.only do
        # stuff for MsWin only
      end

      Windows.or_variant do
        # all Windows variants
      end

      Linux.dont do
        # don't run this under Linux
      end

  Have a look near the top of ``rubicon.rb`` for details.


Issues
------

- what is correct floating point behaviour?
  Linux and FreeBSD differ. Should this "shine through" at the Ruby
  level, or should Ruby attempt to work the same way on different
  platforms?

- some time-related tests now depend on the timezone of the used
  machine. The test fails if you are in (at least) some parts of the
  US.

- *TODO: describe platform issues ...*

- *TODO: need to add more here ...*


Other Ruby testsuites
---------------------

We believe that Rubicon still is the most comprehensive testsuite for
the Ruby language and core libraries. But there some other tests too:

- in the CVS archive for Ruby itself, there is a file
  ``sample/test.rb``. This is the file that is used when the ``make
  test`` command is used.

- in the CVS archive for Ruby itself, there is a ``test`` subdirectory
  with a lot of tests. It is invoked by the ``make test-all`` command
  at the top level.


History
-------

Rubicon was originally written by Dave Thomas & Andy Hunt
(http://www.pragmaticprogrammer.com) and was for a long time the
authoritative test suite for the Ruby language. Rubicon was then
maintained at ruby-lang.org alongside Ruby itself. These archives
still exist has not been touched for a couple of years now.

On a page (still accessible) at www.rubycentral.com an explanation is
given for the name:

    *Rubicon is a set of tests for Ruby itself. (The name comes either
    from Ruby Confidence tests, or from the fact that once you're
    passed the Rubicon, the world is at your feet.)*

    November, 2000

They also said:

    *Rubicon is a set of unit tests for the Ruby interpreter and its
    libraries. Written using RubyUnit, the tests run automatically,
    allowing you to verify the correct installation of new versions of
    Ruby. We found this process to be invaluable while we were writing
    Programming Ruby, as it allowed us to check out changes in the CVS
    releases of Ruby with very little effort.*

The current project rubytests_ at RubyForge_ is an attempt to revive
Rubicon. When the project was started at RubyForge, Chad Fowler wrote
the following in the README file:

    *We are now starting a renewed effort to bring rubicon up to
    date with Ruby 1.8+,* [...]

Having Rubicon as a project at RubyForge has several advantages. We
get the whole infrastructure at RubyForge_ for free: CVS-archives,
user-accounts, mailing-lists, web-pages, etc.

After Rubicon was moved to RubyForge in mars 2004, the tests were
ported to `Test::Unit`_. Rubicon has also been updated to work with Ruby
1.8.2. Currently (2005-01-18) Ruby 1.8.2 passes the Rubicon testsuite
with very few errors (no errors at all on FreeBSD).

.. _Ruby:  http://www.ruby-lang.org/
.. _JRuby: http://jruby.sourceforge.net/

.. _rubytests: http://rubyforge.org/projects/rubytests/
.. _RubyForge: http://rubyforge.org/

.. _`Test::Unit`: http://www.ruby-doc.org/stdlib/libdoc/test/unit/rdoc/index.html

.. meta::
    :description: Rubicon - a testsuite for Ruby
    :keywords: ruby, rubytests, testsuite

