Jenkins Setup Instructions and Usage Information

This file describes how to setup the Jenkins automation server for
continuous integration of the SUNDIALS Git repository. The steps below
are based on installing Jenkins 2.60.2 on a desktop machine running
RHEL6 following the instructions available at:
https://pkg.jenkins.io/redhat-stable/

=======================================================================

Installing Jenkins:

1) Install Java 8

   > sudo yum install java-1.8.0-openjdk.x86_64

2) Install the stable version of Jenkins
  
   > sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
   > sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
   > sudo yum install jenkins

3) Start Jenkins

   > sudo service jenkins start

4) Open a web browser and navigate to http://system-address:8080/
   where system-address is the network address of the machine Jenkins
   was installed on e.g., http://tux369.llnl.gov:8080/

5) Click "select plugins to install"

6) Add the following plugins to the default selections:

   * Dashboard view
   * build_name_setter
   * embeddable_build_status
   * Rebuilder
   * SSH Agent
   * Throttle Concurrent Build
   * Email Extension Template

7) Click "install"

8) When the installation is complete, fill in the information needed
   to create the first admin user then click "save and finish"

9) Click "start using jenkins"

=======================================================================

Additional Plugins:

The "Bitbucket Branch Source Plugin" provides better integration with
the local Bitbucket server, Git branches, and pull requests. 

The "Collapsing Console Sections Plugin" provides better readability
of the Jenkins console output by enabling collapsible sections.

To install these plugins:

1) Navigate to Manage Jenkins > Manage Plugins

2) Click the "Available" tab

3) Search for the plugin name

4) Check the box next to plugin

5) Click "install without restart"

=======================================================================

Using the Collapsing Console Sections Plugin:

The Collapsing Console Sections Plugin works by parsing the Jenkins
console output to find lines, defined by Java regular expression, that
mark the start and end of a collapsible section. 

1) Navigate to Manage Jenkins > Configure System

2) Under "Collapsing Console Sections" check the box next to Enable
   sections numbering. 

3) To create a Jenkins Setup section, click on "Add Console Section"
   and enter/do the following: 

   Section name:        Jenkins Setup
   Section starts with: ((Branch indexing)|(Started by user .*))
   Section ends with:   END JENKINS SETUP
   Check the box next to "Collapse Sections by default"

4) To create a CMAKE section, click on "Add Console Section" and
   enter/do the following: 

   Section name:        CMAKE
   Section starts with: START CMAKE
   Section ends with:   (cmake returned .*)
   Check the box next to "Collapse Sections by default"

5) To create a BUILDING section, click on "Add Console Section" and
   enter/do the following: 

   Section name:        BUILDING
   Section starts with: START MAKE
   Section ends with:   (make returned .*)
   Check the box next to "Collapse Sections by default"

6) To create a TESTING section, click on "Add Console Section" and
   enter/do the following:

   Section name:        TESTING
   Section starts with: START TEST
   Section ends with:   (make test returned .*)
   Check the box next to "Collapse Sections by default"

7) To create a INSTALLING section, click on "Add Console Section" and
   enter/do the following:

   Section name:        INSTALLING
   Section starts with: START INSTALL
   Section ends with:   (make install returned .*)
   Check the box next to "Collapse Sections by default"

8) Click "Save"

=======================================================================

Adding Credentials:

For Jenkins to use the Bitbucket server API we need to provide it with
a user's credentials.

1) Navigate to Credentials > global 

2) Click "Add Credentials"

3) The Kind should be "Username and password" and the Scope should be
   "Global (Jenkins, nodes, items, all child items, etc)". Fill in the
   username and password then click OK.

   Note: The saved login information in Jenkins will need to be
   updated whenever the user changes their Bitbucket password.

To clone repositories Jenkins will need a pair of SSH keys.

1) Create new SSH Keys for Jenkins:

   > cd ~/.ssh
   > mkdir key_backup
   > cp id_rsa* key_backup
   > mkdir jenkins
   > ssh-keygen -t rsa -C jenkins
   Press <Enter>
   Press <Enter>
   Press <Enter>
   > cp id_rsa* jenkins
   > cp key_backup/id_rsa* .

2) Navigate to Credentials > global 

3) Click "Add Credentials"

4) The Kind should be "SSH Username with private key", Scope
   should be "Global (Jenkins, nodes, items, all child items,
   etc)", and fill in the Username as Jenkins. For the Private Key
   field:
   
   a) If you are changing the JENKINS_USER (see below) then enter the
      path to the private key created above

      /home/username/.ssh/jenkins/id_rsa

      where username is the user name set for JENKINS_USER below.

   b) If you are not changing the JENKINS_USER (see below) then you
      you will need to either use the "Enter directly" option or
      transfer ownership of the keys to the local user jenkins, move
      them a directory the local jenkins user can access, and enter the
      path to as in a) above.

   Then click OK.

=======================================================================

Adding a Bitbucket Server:

Because the repository is located in a local Bitbucket server we need to
give this information to Jenkins:

1) Navigate to Manage Jenkins > Configure System

2) Under Bitbucket Endpoints, click on "Add" and select "Bitbucket
   Server" 

3) Enter a name for the server (e.g. mybitbucket llnl) and for the
   server URL enter https://mybitbucket.llnl.gov

4) Click "Save"

=======================================================================

Changing the Default Git:

If you are changing the JENKINS_USER (see below), then you may need to
change which installation of Git Jenkins uses. When Jenkins is run
as a specific user it uses the .gitconfig file of that user. Depending
on the options set in .gitconfig Jenkins may need to use a different
(newer) install of Git than it finds by default.

NOTE: This setting only effects what version of Git Jenkins will used
to interact with the repo. If any regression test scripts use Git
commands they will use the default install of Git unless a newer
version is added to the path or called explicitly. See below for
updating the PATH environment variable:

1) Navigate to Manage Jenkins > Global Tool Configuration

2) Under Git, change "Path to Git executable" to the desired
   installation of Git (e.g. https://mybitbucket.llnl.gov)

3) Click "Save"

=======================================================================

Updating the PATH environment variable:

If regression testing scripts require that certain path are set in the
PATH environment variable, the following steps detail how to modify
PATH for each Jenkins testing node.

1) Navigate to Manage Jenkins > Manage Nodes

2) Click the gear icon next to the node you want to configure

3) Under "Node Properties" check the box next to "Environment
   variables" 

4) Click "Add" and enter the PATH+SOMENAME in the Name field and the
   desired path in the Value field. For example to add the path for
   a newer version of git to PATH do enter the following:

   Name:  PATH+NFS_GIT
   Value: /usr/apps/git/2.9.4/bin

5) Click "Save"

=======================================================================

Adjusting the number of build executors:

Depending on the machine Jenkins is installed on it may or may not be
desirable to allow multiple regression tests to run in parallel on a
node (machine). The number of jobs Jenkins can run in parallel can be
adjusted on a per node basis. If there are more jobs to run than idle
nodes Jenkins will place the jobs in a first in first out queue and
run them as resources become available.

1) Navigate to Manage Jenkins > Manage Nodes 

2) Click the gear icon next to the desired node

3) Change "# of executors" to the desired number of test jobs that can
   run in parallel

4) Click "Save"

=======================================================================

Enabling Anonymous Read Access:

For users to view the build status without needing to logging in to
Jenkins will enable anonymous read access.

1) Navigate to Manage Jenkins > Configure Global Security

2) Under Access Control > Authorization, check the box in front of
   "Allow anonymous read access"

3) Click "Save"

=======================================================================

Changing JENKINS_USER:

Regression testing in SUNDIALS utilizes external libraries (e.g. MPI,
LAPACK, hypre, etc.) that are installed in a shared location
(/usr/casc/apps) that is accessible to members of the nsde Active
Directory group. By default Jenkins is installed and runs as the local
user jenkins. As a local jenkins user can not be added to the nsde
group and so is not able to access the shared directory. If the
libraries used in regression testing are installed in a local
directory accessible to the jenkins user (e.g. /usr/lib) then the
following steps are not necessary.

NOTE: When changing the JENKINS_USER so Jenkins runs under a different
user account Jenkins will utilize the git configuration (~/.gitconfig)
file of that user. Depending on the settings in the .gitconfig file
this can cause unexpected behavior in Jenkins. For example, if the
pull.ff is set to 'only' this can break merges performed in pull
request testing. A possible workaround for the user is to setup an
alias with the desired behavior.

The following instructions describe how to change the user Jenkins is
run under. 

1) Stop Jenkins

   > sudo service jenkins stop

2) Open the Jenkins configuration file, change the JENKINS_USER
   variable to the desired user name, and save the revised file

   > sudo nano /etc/sysconfig/jenkins
   JENKINS_USER="username"  

3) Change the ownership of the Jenkins home, webroot, and logs

   > sudo chown -R username:username /var/lib/jenkins
   > sudo chown -R username:username /var/cache/jenkins
   > sudo chown -R username:username /var/log/jenkins

4) Start Jenkins

   > sudo service jenkins start

=======================================================================

Adding a Jenkins Pipeline:

A Jenkins Pipeline is domain specific language for implementing a
project's entire build/test/deploy pipeline in a Jenkinsfile that is
stored in the project's code repo. The following steps setup Jenkins
pipeline for a multibranch Bitbucket server repository.  

1) Click on "New Item"

2) Enter an item name e.g. SUNDIALS

3) Click "Multibranch Pipeline" then click "OK"

4) Under Branch Source, click "Add source" and select "Bitbucket"

5) Under Bitbucket set the following:

   a) Server: select the server name entered when adding the Bitbucket
      server above (e.g. mybitbucket llnl)

   b) Credentials: select the user name entered for accessing the
      Bitbucket server API 

   c) Owner: sundials

   d) Repository Name: select the desired repo name (e.g. sunrepo)

   e) Discover branches: select the strategy "All branches" 

6) Under Bitbucket click "Add" and select "Checkout over SSH." Click
   "Credentials" select  and and select Jenkins.

7) Under Scan Multibranch Pipeline Trigger, check the box next to
   "Periodically if not otherwise run." Leave the Interval as 1
   minute. 

   Note: The hook to notify Jenkins of new pushes to the Bitbucket
   server does not currently work with the Bitbucket Branch Source
   Plugin. A plugin is available that works with this Plugin but it
   needs to be install by a Bitbucket administrator. 

8) Click "Save"

Jenkins will now scan the repository for branches and pull requests
that contain a Jenkinsfile and run the pipeline for any branches or
pull requests with a Jenkinsfile.

=======================================================================
