
  This supplementary document explains how to use HDF4 handler to serve NASA 
HDF4/HDF-EOS products in a manner that follows the CF conventions.

-------------------------------------------------------------------------------
Contents

1. What are the CF conventions and how is it related to the new HDF4 handler?
2. How to install CF-enabled HDF4 Handler correctly
3. BES Keys in h4.conf
4. How to easily modify attributes that do not follow the CF conventions 
   using NcML handler.
-------------------------------------------------------------------------------

1. What are the CF conventions and how is it related to the new HDF4 handler?

  Before we discuss the usage further, it's very important to know what the CF
conventions are.  The CF conventions precisely define metadata that provide a 
description of physical, spatial, and temporal properties of the data. 
This enables users of data from different sources to decide which quantities 
are comparable, and facilitates building easy-to-use visualization tools with 
 maps in different projections.

  Here, we define the two levels of meeting the CF conventions --- basic and 
  extra.

  1) basic: CF conventions have basic (syntactic) rules in describing the 
            metadata itself correctly. For example, dimensions should have 
            names; certain characters are not allowed; no duplicate variable /
            dimension names are allowed.
  2) extra: All physical, spatial, and temporal properties of the data are 
            correctly described so that visualization tools (e.g., IDV and 
            Panoply) can pick them up and display datasets correctly with the 
            right physical units. A good example is the use of "units" and 
            "coordinates" attributes.
 
  If you look at NASA HDF4 and HDF-EOS2 products, they are very diverse in
 self-describing data and fail to meet CF conventions in many ways. Thus, 
 the HDF4 handler aims to meet the conventions by correcting OPeNDAP
 attribute(DAS)/description(DDS)/data outputs on the fly. Although we tried
 our best effort to implement the "extra" level of meeting the CF conventions,
 some products are inherently difficult to meet such level. In those cases,
 we ended up meeting the basic level of meeting the CF conventions.


2. How to install CF-enabled HDF4 Handler correctly

  The first step of using the HDF4 handler with CF option is to install
the handler correctly because it has three different options. We'll call them
default, generic, and hdfeos2 for convenience.

  1) default: This option gives the same output as the legacy handler.
  2) generic: This option gives the output that meets the basic CF conventions 
              regardless of HDF4 and HDF-EOS2 products. Some HDF4 products
              can meet the extra CF conventions while most HDF-EOS2 products 
              will fail to meet the extra CF conventions.
  3) hdfeos2: This option treats HDF-EOS2 products differently so that 
              their output follows not only the basic CF conventions but 
              also the extra CF conventions. For HDF4 products, the output is 
              same as the generic option.

2.1 Pick the right RPM instead of building from source

   If you use Linux system that supports RPM package manager and have a super
 user privilege, the easiest way to install the HDF4 handler is using the latest
 RPMs at hdfeos.org website.

  The OPeNDAP's download website provides two RPMs --- one with HDF-EOS and one
 without HDF-EOS. You should pick the one with HDF-EOS if you want to take 
 advantage of the extra CF support provided by the handler. If you pick one 
 without HDF-EOS, please make sure that the H4.EnableCF key is set "true" in 
 h4.conf file. See section 3.1 below for the full usage.

  Here are two basic commands for deleting and adding RPMs:

  o Remove any existing RPM package using 'rpm -e <package_name>'. 
  o Install a new RPM package using 'rpm -i <package_name.rpm>'. 

1) Download and install the latest "libdap", "BES", and "General purpose 
 handlers (a.k.a. dap-server)" RPMs first from

   http://opendap.org/download/hyrax

3) Download and install the latest "hdf4_handler" RPM from

   http://hdfeos.org/software/hdf4_handler.php#download

4) (Optional) Configure the handler after reading the section 3 below.

5) (Re)start the BES server.
 
   %/usr/bin/besctl (re)start

2.2 Build with the HDF-EOS2 library if you plan to support HDF-EOS2 products.

  If you plan to build one instead of using RPMs and to support HDF-EOS2
 products, please consider installing the HDF-EOS2 library first. Then, build 
 the handler by specifying --with-hdfeos2=/path/to/hdfeos2-install-prefix 
during the configuration stage like below:
   
   ./configure --with-hdf4=/usr/local --with-hdfeos2=/usr/local/

  Although the HDF-EOS2 library is not required to clean dataset names and 
attributes that CF conventions require, visualization will fail for most 
HDF-EOS2 products without the use of HDF-EOS2 library. Therefore, it is 
strongly recommended to use --with-hdfeos2 configuration option if you plan
to serve NASA HDF-EOS2 data products. The --with-hdfeos2 configuration option 
will affect only the outputs of the HDF-EOS2 files including hybrid files,
not pure HDF4 files.

  As long as the H4.EnableCF key is set to be true as described in section 3.1
 below, the HDF4 handler will generate outputs that conform to the basic CF 
conventions even though the HDF-EOS2 library is not specified with the 
--with-hdfeos2 configuration option. All HDF-EOS2 objects will be treated as 
pure HDF4 objects.

  Please see the INSTALL document on step-by-step instruction on building the
 handler.

3. BES Keys in h4.conf

 You can control HDF4 handler's output behavior significantly by changing
key values in a configuration file called "h4.conf".

 If you used RPMs, you can find the h4.conf file in /etc/bes/modules/.
 If you built one, you can find the h4.conf file in {prefix}/etc/bes/modules.

  As we improve the performance and support more NASA HDF4 and HDF-EOS2 products,
More BES keys are added to the CF option of the handler. 
Please refer to the file h4.conf.in for the descriptions and usage of the BES keys,
especially for NASA data centers that may need to choose different key values for
their specific usage. 
The default configuration values are specified after the "=" sign.

3.1 H4.EnableCF=true

  If this key's value is false, the handler will behave the same as the default 
handler. The output will not follow basic CF conventions. Object and attribute
names will not be corrected to follow the CF conventions.  Most NASA 
products cannot be visualized by visualization tools that follow the CF 
conventions. Such tools include IDV and Panoply.

  The rest of keys below relies on this option. This key must be set to be 
"true" to ensure other keys to be valid. Thus, this is the most important key 
to be turned on.

3.2 BES keys removed at this version

none

3.3 BES keys added at this version

H4.EnableDataCacheFile=false (Only apply to data variables read via HDF4 SDS interfaces)
H4.EnableMetaDataCacheFile=false(Only apply to AIRS version 6 HDF products)


3.4 BES keys from the previous versions

H4.EnablePassFileID=false
H4.DisableStructMetaAttr=true
H4.EnableEOSGeoCacheFile=false
H4.DisableECSMetaDataMin=true
H4.DisableECSMetaDataAll=false
H4.EnableVgroupAttr=true
H4.EnableSwathGridAttr=true

H4.EnableSpecialEOS=true
H4.DisableScaleOffsetComp=true
H4.EnableCheckScaleOffsetType=false
H4.EnableHybridVdata=true
H4.EnableCERESVdata=false
H4.EnableVdata_to_Attr=true
H4.EnableCERESMERRAShortName=true
H4.DisableVdataNameclashingCheck=true
H4.EnableVdataDescAttr=false
H4.EnableCheckMODISGeoFile=false

3.5 DO read h4.conf for more detailed description of BES keys.

4. How to easily modify attributes that do not follow the CF conventions 
   using NcML handler

  Although HDF4 handler adjusts many things to meet the CF conventions 
 automatically, it cannot correct all attributes. For example, some old HDF4
 products use "add_off" instead of "add_offset."  To correct attribute name
 and value, OPeNDAP provides NcML handler. With this module, a data provider 
 can augment datasets served on their local Hyrax server in the following ways:

  o Add, modify, or remove both attributes and variables
  o Combine multiple datasets into a single "virtual" data set using "union", 
  "joinNew", or "joinExisting" aggregations.

 Writing an NcML file is as easy as writing an HTML file. Please refer to 
 the NcML example page [2] at HDF-EOS Tools and Information Center website.


[1] http://modis.gsfc.nasa.gov/data/dataprod/nontech/MOD0203.php
[2] http://hdfeos.org/examples/ncml.php
