Changes to SUNDIALS in release 4.1.0

An additional N_Vector implementation was added for Tpetra vector from
Trilinos library to facilitate interoperability between SUNDIALS and Trilinos.
This implementation is accompanied by additions to user documentation and
SUNDIALS examples.

A bug was fixed where a nonlinear solver object could be freed twice in some use
cases.

The EXAMPLES_ENABLE_RAJA CMake option has been removed. The option EXAMPLES_ENABLE_CUDA
enables all examples that use CUDA including the RAJA examples with a CUDA back end (if the RAJA
NVECTOR is enabled).

The implementation header files (e.g. arkode_impl.h) are no longer installed. This means users
who are directly manipulating package memory structures will need to update their code
to use the package's public API.

Python is no longer required to run make test and make test_install.

Fixed a bug in ARKodeButcherTable_Write when printing a Butcher table
without an embedding.

Changes to SUNDIALS in release 4.0.2

Added information on how to contribute to SUNDIALS and a contributing agreement.

Moved definitions of DLS and SPILS backwards compatibility functions to a source file.
The symbols are now included in the appropriate package library, e.g. libsundials_cvode.lib.

Changes to SUNDIALS in release 4.0.1

A bug in ARKODE where single precision builds would fail to compile has been fixed.

Changes to SUNDIALS in release 4.0.0

The direct and iterative linear solver interfaces in all SUNDIALS packages have
been merged into a single unified linear solver interface to support any valid
SUNLINSOL module. This includes the DIRECT and ITERATIVE types as well as the
new MATRIX_ITERATIVE type. Details regarding how SUNDIALS packages utilize linear
solvers of each type as well as discussion regarding intended use cases for
user-supplied SUNLINSOL implementations are included in the SUNLINSOL chapter of
the user guides. All example programs have been updated to use the new unified
interfaces.

The unified interface is very similar to the previous DLS and SPILS interfaces.
To minimize challenges in user migration to the unified linear solver interface,
the previous DLS and SPILS routines for all packages may still be used; these
will be deprecated in future releases, so we recommend that users migrate to the
new names soon. Additionally, we note that Fortran users will need to enlarge
their iout array of optional integer outputs, and update the indices that they
query for certain linear-solver-related statistics.

The names of all constructor routines for SUNDIALS-provided SUNLinSol
implementations have been updated to follow the naming convention SUNLinSol_*
where * is the name of the linear solver e.g., Dense, KLU, SPGMR, PCG, etc.
Solver-specific ``set'' routine names have been similarly standardized. To
minimize challenges in user migration to the new names, the previous routine
names may still be used; these will be deprecated in future releases, so we
recommend that users migrate to the new names soon. All example programs have
been updated to used the new naming convention.

The SUNBandMatrix constructor has been simplified to remove the storage upper
bandwidth argument.

SUNDIALS integrators (ARKode, CVODE, CVODES, IDA, and IDAS) have been updated to
utilize generic nonlinear solver modules through the SUNNONLINSOL API. This API
will ease the addition of new nonlinear solver options and allow for external or
user-supplied nonlinear solvers. The SUNNONLINSOL API and provided SUNNONLINSOL
modules are described in a new user guide chapter and follow the same object
oriented design and implementation used by the NVECTOR, SUNMATRIX, and
SUNLINSOL modules. All integrator example programs have also been updated to
used the new nonlinear solver API.

Three fused vector operations and seven vector array operations have been added
to the NVECTOR API. These optional operations are disabled by default and may be
activated by calling vector specific routines after creating an NVECTOR. See the
NVECTOR chapter in the user guides for more information on the new operations.

Added a new NVECTOR (NVECTOR_OPENMPDEV) which leverages OpenMP 4.5+ device offloading.

Multiple updates to the CUDA NVECTOR were made:

  * Changed the N_VMake_Cuda function to take a host data pointer and a device
    data pointer instead of an N_VectorContent_Cuda object.
    
  * Changed N_VGetLength_Cuda to return the global vector length instead of
    the local vector length.

  * Added N_VGetLocalLength_Cuda to return the local vector length.
  
  * Added N_VGetMPIComm_Cuda to return the MPI communicator used.

  * Removed the accessor functions in the namespace suncudavec.
  
  * Added the ability to set the cudaStream_t used for execution of the CUDA
    NVECTOR kernels. See the function N_VSetCudaStreams_Cuda.

  * Added N_VNewManaged_Cuda, N_VMakeManaged_Cuda, and N_VIsManagedMemory_Cuda
    functions to accommodate using managed memory with the CUDA NVECTOR.

Multiple updates to the RAJA NVECTOR were made:

  * Changed N_VGetLength_Raja to return the global vector length instead of
    the local vector length.

  * Added N_VGetLocalLength_Raja to return the local vector length.

  * Added N_VGetMPIComm_Raja to return the MPI communicator used.
 
  * Removed the accessor functions in the namespace sunrajavec.

Two changes were made in the CVODE/CVODES/ARKODE initial step size algorithm:

  1. Fixed an efficiency bug where an extra call to the RHS function was made.
  2. Changed the behavior of the algorithm if the max-iterations case is hit.
     Before the algorithm would exit with the step size calculated on the 
     penultimate iteration. Now it will exit with the step size calculated
     on the final iteration.

Fortran 2003 interfaces to CVODE, the fixed-point and Newton nonlinear solvers,
the dense, band, KLU, PCG, SPBCGS, SPFGMR, SPGMR, and SPTFQMR linear solvers,
and the serial, PThreads, and OpenMP NVECTORs have been added.

The ARKode library has been entirely rewritten to support a modular approach to
one-step methods, which should allow for rapid research and development of novel
integration methods without affecting existing solver functionality.

A new ARKode stepper, MRIStep, has been added for two rate explicit-explicit
multirate infinitesimal step methods.

ARKode's dense output infrastructure has been improved to support higher-degree
Hermite polynomial interpolants (up to degree 5) over the last successful time step.


Changes to SUNDIALS in release 3.2.1

* Fixed a bug in the CUDA NVECTOR where the N_VInvTest operation could write
  beyond the allocated vector data.

* Fixed library installation path for multiarch systems. This fix changes the default
  library installation path to CMAKE_INSTALL_PREFIX/CMAKE_INSTALL_LIBDIR
  from CMAKE_INSTALL_PREFIX/lib. CMAKE_INSTALL_LIBDIR is automatically set, but is
  available as a CMAKE option that can modified.

Changes to SUNDIALS in release 3.2.0

* Fixed problem with index types which would occur with some compilers
  (e.g. armclang) that did not define __STDC_VERSION__. The fix includes
  a depcrecation of the current behavior of the SUNDIALS_INDEX_TYPE
  CMake option.

* Fixed a thread-safety issue in CVODES and IDAS when using adjoint sensitivity
  analysis.

* Added hybrid MPI/CUDA and MPI/RAJA vectors to allow use of more than one 
  MPI rank when using a GPU system.  The vectors assume one GPU device per 
  MPI rank.

* Changed the name of the RAJA nvector library to libsundials_nveccudaraja.lib
  from libsundials_nvecraja.lib to better reflect that we only support
  CUDA as a backend for RAJA currently.

* Increased CMake minimum version to 3.1.3

* Add constraint handling feature to CVODE and CVODES.

* Fixed a bug in IDAS where the saved residual value used in the nonlinear
  solve for consistent initial conditions was passed as temporary workspace and
  could be overwritten.

* Several changes were made to the build system. If MPI is enabled and MPI  
  compiler wrappers are not set, the build system will check if
  CMAKE_<language>_COMPILER can compile MPI programs before trying to locate and
  use an MPI installation. The native CMake FindMPI module is now used to locate
  an MPI installation. The options for setting MPI compiler wrappers and the
  executable for running MPI programs have been updated to align with those in
  native CMake FindMPI module. This included changing MPI_MPICC to MPI_C_COMPILER,
  MPI_MPICXX to MPI_CXX_COMPILER combining MPI_MPIF77 and MPI_MPIF90 to
  MPI_Fortran_COMPILER, and changing MPI_RUN_COMMAND to MPIEXEC_EXECUTABLE. When a
  Fortran name-mangling scheme is needed (e.g., LAPACK_ENABLE is ON) the build system
  will infer the scheme from the Fortran compiler. If a Fortran compiler is not
  available or the inferred or default scheme needs to be overridden, the advanced
  options SUNDIALS_F77_FUNC_CASE and SUNDIALS_F77_FUNC_UNDERSCORES can be used to
  manually set the name-mangling scheme and bypass trying to infer the scheme.
  Additionally, parts of the main CMakeLists.txt file were moved to new files in
  the src and example directories to make the CMake configuration file structure
  more modular.

Changes in SUNDIALS in release 3.1.2

* Fixed Windows specific problem where sunindextype was not correctly 
  defined when using 64-bit integers for the sundials index type. On Windows
  sunindextype is now defined as the MSVC basic type __int64.

* Changed LICENSE install path to instdir/include/sundials

* Updated the minimum required version of CMake to 2.8.12 and enabled
  using rpath by default to locate shared libraries on OSX.

* The misnamed function CVSpilsSetJacTimesSetupFnBS in cvodes has been deprecated
  and replaced by CVSpilsSetJacTimesBS. The deprecated function
  CVSpilsSetJacTimesSetupFnBS will be removed in the next major release.

* Added and updated usage-notes examples from the SUNDIALS website to work with
  SUNDIALS 3.x. The new examples are cvode/cvDisc_dns.c cvode/cvRoberts_dns_negsol.c
  and cvodes/cvsRoberts_FSA_dns_Switch.c. 

* Added sparse SUNMatrix "Reallocate" routine to allow specification
  of the nonzero storage.

* Updated the KLU SUNLinearSolver module to set constants for the two
  reinitialization types, and fixed a bug in the full reinitialization
  approach where the sparse SUNMatrix pointer would go out of scope on
  some architectures.
  
* Updated the "ScaleAdd" and "ScaleAddI" implementations in the sparse
  SUNMatrix module to more optimally handle the case where the target
  matrix contained sufficient storage for the sum, but had the wrong
  sparsity pattern.  The sum now occurs in-place, by performing the
  sum backwards in the existing storage.  However, it is still more
  efficient if the user-supplied Jacobian routine allocates storage
  for the sum $I+\gamma J$ or $M+\gamma J$ manually (with zero entries
  if needed).

Changes to SUNDIALS in release 3.1.1

* Fixed a minor bug in the CVODE and CVODES cvSLdet routine, where a
  return was missing in the error check for three inconsistent roots.

* Fixed a potential memory leak in the SPGMR and SPFGMR linear
  solvers: if "Initialize" was called multiple times then the solver
  memory was reallocated (without being freed). 
  
* Fixed a minor bug in the ARKReInit routine, where a flag was
  incorrectly set to indicate that the problem had been resized
  (instead of just re-initialized). 

* Fixed C++11 compiler errors/warnings about incompatible use of
  string literals.

* Updated KLU SUNLinearSolver module to use a typedef for the
  precision-specific solve function to be used (to avoid compiler
  warnings). 

* Added missing typecasts for some (void*) pointers (again, to avoid
  compiler warnings).

* Bugfix in sunmatrix_sparse.c where we had used 'int' instead of
  'sunindextype' in one location.

* Fixed a minor bug in KINPrintInfo where a case was missing for
  KIN_REPTD_SYSFUNC_ERR leading to an undefined info message.

* Added missing #include <stdio.h> in NVECTOR and SUNMATRIX
  header files.

* Added missing prototypes for ARKSpilsGetNumMTSetups in ARKode and
  IDASpilsGetNumJTSetupEvals in IDA and IDAS.

* Fixed an indexing bug in the CUDA NVECTOR implementation of
  N_VWrmsNormMask and revised the RAJA NVECTOR implementation of
  N_VWrmsNormMask to work with mask arrays using values other than
  zero or one. Replaced doubles with realtypes in the RAJA vector
  test functions.

* Fixed compilation issue with GCC 7.3.0 and Fortran programs that do
  not require a SUNMatrix or SUNLinearSolver module (e.g. iterative
  linear solvers, explicit methods in ARKode, functional iteration
  in CVODE, etc.).


Changes to SUNDIALS in release 3.1.0

* Added NVECTOR print functions that write vector data to a specified
  file (e.g., N_VPrintFile_Serial).

* Added 'make test' and 'make test_install' options to the build system for
  testing SUNDIALS after building with 'make' and installing with 'make install'
  respectively.

* Added "Changes in ..." (latest version) to Intro. in all User Guides.



Changes to SUNDIALS in release 3.0.0

* New linear solver API and interfaces for all SUNDIALS packages and 
  linear solvers
 - Refactored all matrix and linear solver interfaces in SUNDIALS
 - Implemented new interfaces in all 6 SUNDIALS Packages
 - Corresponding updates to all example programs released with SUNDIALS
The goal of the redesign of these interfaces was to provide more encapsulation
and ease in interfacing custom linear solvers and interoperability 
with linear solver libraries.
 - Specific changes include:
      - Added generic SUNMATRIX module with three provided implementations:
        dense, banded and sparse.  These replicate previous SUNDIALS Dls and
        Sls matrix structures in a single object-oriented API.
      - Added example problems demonstrating use of generic SUNMATRIX modules.
      - Added generic SUNLINEARSOLVER module with eleven provided
        implementations: dense, banded, LAPACK dense, LAPACK band, KLU,
        SuperLU_MT, SPGMR, SPBCGS, SPTFQMR, SPFGMR, PCG.  These replicate
        previous SUNDIALS generic linear solvers in a single object-oriented
        API.
      - Added example problems demonstrating use of generic SUNLINEARSOLVER
        modules.
      - Expanded package-provided direct linear solver (Dls) interfaces and
        scaled, preconditioned, iterative linear solver (Spils) interfaces
        to utilize generic SUNMATRIX and SUNLINEARSOLVER objects.
      - Removed package-specific, linear solver-specific, solver modules
        (e.g. CVDENSE, KINBAND, IDAKLU, ARKSPGMR) since their functionality
        is entirely replicated by the generic Dls/Spils interfaces and
        SUNLINEARSOLVER/SUNMATRIX modules.  The exception is CVDIAG, a
        diagonal approximate Jacobian solver available to CVODE and CVODES.
      - Converted all SUNDIALS example problems to utilize new generic
        SUNMATRIX and SUNLINEARSOLVER objects, along with updated Dls and
        Spils linear solver interfaces.
      - Added Spils interface routines to ARKode, CVODE, CVODES, IDA and
        IDAS to allow specification of a user-provided "JTSetup" routine.
        This change supports users who wish to set up data structures for
        the user-provided Jacobian-times-vector ("JTimes") routine, and
        where the cost of one JTSetup setup per Newton iteration can be
        amortized between multiple JTimes calls.

* Two new NVECTOR modules added: for CUDA and RAJA support for GPU systems
(Information on RAJA: https://software.llnl.gov/RAJA/ )
These vectors are supplied to provide very basic support for running
on GPU architectures.  Users are advised that these vectors both move all data
to the GPU device upon construction, and speedup will only be realized if the
user also conducts the right-hand-side function evaluation on the device.
In addition, these vectors assume the problem fits on one GPU.
For further information about RAJA, users are referred to the web site, 
https://software.llnl.gov/RAJA/.

* Addition of sunindextype option for 32- or 64-bit integer data index types 
within all SUNDIALS structures
 - sunindextype is defined to be int32_t or int64_t when portable types are
   supported, otherwise it is defined as int or long int.
 - The Fortran interfaces continue to use long\_int for indices, except for 
   their sparse matrix interface that now uses the new sunindextype.
 - Includes interfaces to PETSc, hypre, SuperLU_MT, and KLU with 
   either 32-bit or 64-bit capabilities depending how the user configures 
   SUNDIALS.

* To avoid potential namespace conflicts, the macros defining booleantype
values TRUE and FALSE have been changed to SUNTRUE and SUNFALSE respectively.

* Temporary vectors were removed from preconditioner setup and solve
routines for all packages.  It is assumed that all necessary data
for user-provided preconditioner operations will be allocated and
stored in user-provided data structures.

* The file include/sundials\_fconfig.h was added.  This file contains 
SUNDIALS type information for use in Fortran programs.

* Added support for many xSDK-compliant build system keys
(Information on xSDK compliance: https://xsdk.info/policies/ )
The xSDK is a movement in scientific software to provide a foundation for the
rapid and efficient production of high-quality, 
sustainable extreme-scale scientific applications.  More information can
be found at https://xsdk.info.

* Added functions SUNDIALSGetVersion and SUNDIALSGetVersionNumber to
get SUNDIALS release version information at runtime.

* In build system:
 - Added separate BLAS_ENABLE and BLAS_LIBRARIES CMake variables
 - Additional error checking during CMake configuration
 - Fixed minor CMake bugs
 - Renamed CMake options to enable/disable examples for greater clarity 
   and added option to enable/disable Fortran 77 examples
    - Changed EXAMPLES_ENABLE to EXAMPLES_ENABLE_C
    - Changed CXX_ENABLE to EXAMPLES_ENABLE_CXX
    - Changed F90_ENABLE to EXAMPLES_ENABLE_F90
    - Added EXAMPLES_ENABLE_F77 option

* Corrections and additions to all User Guides.

* Added "Changes in ..." (latest version) to Intro. in all User Guides.


ARKode:
* Added comments to arkode_butcher.c regarding which methods should have 
  coefficients accurate enough for use in quad precision
* Bug Fix: Fixed a bug in arkode_butcher.c in use of RCONST
* Bug Fix: Fixed a bug in in the arkInitialSetup utility routine in the 
  order of operations when setting up and using mass matrices
  to ensure the mass matrix vector product is set up before the "msetup" 
  routine is called.
* Fixed ARKode printf-related compiler warnings when building SUNDIALS
  with extended precision.


CVODE and CVODES:
* Bug Fix: In CVodeFree, now call lfree() unconditionally (if non-NULL).

IDA and IDAS:
* Bug Fix: Added missing prototype for IDASetMaxBacksIC in ida.h and idas.h


KINSOL:
* Bug Fix: Corrected KINSOL fcmix name translation for FKIN_SPFGMR
* Renamed KINLocalFn and KINCommFn to KINBBDLocalFn and KINBBDCommFn
  respectively in the BBD preconditioner module for consistency with
  other SUNDIALS solvers.

*------------------------------------------------------------------------------*
Changes to SUNDIALS since Release of March 30, 2015.

* Two new NVECTOR modules added: for parallel Hypre and PETSC.

* In vector API, added new required function, N_VGetVectorID.

* Upgrades to sparse solver interfaces; now support CSR matrix type with KLU solver.

* Updated BiCGStab solver to remove redundant dot product.

* In all packages, example codes were changed from using NV_DATA macro to 
  using N_VGetArrayPointer_* when using the native vectors shipped with
  SUNDIALS

* Fixed minor bug when linear solver performance counters were used uninitialized. 

* In all packages, fixed memory leak in banded preconditioner interface.

* Fixed potential memory leak in KLU ReInit functions in all solvers.

* Fixed some examples w.r.t. switch to new macro/function names SUNRexp etc.

* Various minor fixes to installation-related files.

* Corrected name N_VCloneEmptyVectorArray to N_VCloneVectorArrayEmpty in
  all documentation files.

* Updated all packages to return integers from linear solver and preconditioner 'free' functions.

* Revised all usage skeletons to remove references to names specific to
  any particular NVECTOR module, using more general instructions instead.

* Removed Matlab interface from distribution as it has not been updated
  since 2009.  We expect to update this interface soon.

* Minor corrections and additions to all User Guides.



* In KINSOL, minor bug fix in Picard iteration.

* In KINSOL, minor bug fix in line search.

* In FKINSOL, added FKINCREATE and FKININIT routines to split FKINMALLOC routine
  into two pieces.  FKINMALLOC remains for backward compatibility, but 
  documentation for it has been removed.

* In KINSOL, added kinFoodWeb_kry_omp.c openMP example.



* In ARKode updated linear and mass matrix solvers so that 'free' routines 
  return integer instead of void; updated documentation accordingly.

* In ARKODE, fixed a bug in one Butcher table.

* In ARKODE, fixed error in arkDoErrorTest in recovery after failure.

* In ARKODE, fixed initialization of linear solver performance counters.
* Method and embedding for Billington and TRBDF2 explicit Runge-Kutta methods were swapped. 
* Fix for user specification of absolute tolerance array along with vector Resize() functionality.
* Fix for user-supplied Butcher tables without embeddings (if fixed time steps or manual adaptivity are employed).
* Multiple documentation updates.
* Added missing ARKSpilsGetNumMtimesEvals() function.
 
* Two new NVECTOR modules added: for parallel Hypre and PETSc.
* Enhanced Bi-CGstab by removing redundant dot product.
* Upgrades to sparse solver interfaces; now support CSR matrix type with KLU solver.
* Implicit predictor algorithms were updated: methods 2 and 3 were improved, a new predictor approach was added, and the default choice was modified.
* Revised handling of integer codes for specifying built-in Butcher tables: a global numbering system is still used, but methods now have #defined names to simplify the user interface. 
* Maximum number of Butcher table stages was increased from 8 to 15 to accommodate very high order methods, and an 8th-order adaptive ERK method was added.
* Added support for the explicit and implicit methods in an additive Runge-Kutta method to utilize different stage times, solution and embedding coefficients, to support new SSP-ARK methods.
* Extended FARKODE interface to include a routine to set scalar/array-valued residual tolerances, to support Fortran applications with non-identity mass-matrices. 
* Enhanced Anderson acceleration using QR updating



* In IDA, added idaFoodWeb_kry.c, idaFoodWeb_bnd_omp.c, and idaFoodWeb_kry_omp.c
  examples using openMP.

* Corrected example idaFoodWeb_bnd.c in PrintOutput (wrong component printed).

* In IDA and IDAS, added optional input function IDASetMaxBacksIC to limit
  number of linesearch backtrack operations in IDACalcIC.  User guides
  amended accordingly.



* In IDAS, added idasRoberts_FSA_klu.c, idasRoberts_FSA_sps.c, 
  idasRoberts_ASAi_klu.c, and idasRoberts_ASAi_sps.c sensitivity analysis
  examples using sparse direct solvers.  Also added idasFoodWeb_bnd_omp.c and 
  idasFoobWeb_kry_omp.c openMp examples.

* In IDAS, made SuperLUMT call for backward problem consistent with CVODES.

* In IDAS, added missing backward problem support functions: IDALapackDenseB,
IDALapackDenseFreeB, IDALapackBandB, IDALapackBandFreeB.

* In IDAS, fixed for-loop bugs in IDAAckpntAllocVectors.



* In CVODE, added cvAdvDiff_bnd_omp.c example using openMP.

* In FCVODE, added fcvRoberts_klu.f and fcvRoberts_sps.f fortran
  sparse direct solver examples.

* In FCVODE, fixed argument order bugs in FCVKLU and FCVSUPERLUMT linear 
  solver interfaces.



* In CVODES, aded cvsRoberts_FSA_klu.c, cvsRoberts_FSA_sps.c, 
  cvsRoberts_ASAi_klu.c, cvsRoberts_ASAi_sps.c sparse direct solver examples.
  Also, added cvsAdvDiff_bnd_omp.c openMP example. 
 
* In CVODES, added CVKLUB prototype and corrected CVSuperLUMTB prototype.



* In FKINSOL, FCVODE, and FIDA, added missing Fortran interface routines 
  so that users can supply the sparse Jacobian routine. 

* In CVODES and IDAS changed each **FreeB() to type int; added return(0) 
  to each.


* In CVODES and IDAS header files, corrected documentation of backward
integration  functions, esp. the 'which' argument.

* In KINSOL and ARKODE, updated Anderson acceleration implementation with QR updating.

* In CVODES and IDAS, added ReInit and SetOrdering  wrappers for backward problems.

* In CVODE, IDA, and ARKODE, fixed Fortran interfaces to enable calls to
  *GetErrWeights, *GetEstLocalErrors, and *GetDky within a time step.

* In CVODES and IDAS, in interpolation routines for backward problems,
  added logic to bypass sensitivity interpolation if input sensitivity
  argument is NULL.





* Added "Changes in ..." (latest version) to Intro. in all User Guides.
