#
# This content is written originally by Mark Goodwin.
# Updated by Martin Hicks on Feb. 23, 2009
#
# (c) Copyright 2007-2009, Silicon Graphics Inc.  All Rights Reserved.
#
# Permission is granted to copy, distribute, and/or modify this document
# under the terms of the Creative Commons Attribution-Share Alike,
# Version 3.0 or any later version published by the Creative Commons
# Corp. A copy of the license is available at
# http://creativecommons.org/licenses/by-sa/3.0/us/
#

NB: The following document discusses racks of cluster nodes.  This
    should be taken to mean any convenient sub-cluster of nodes.
    For SGI this was on a per-rack basis.


Monitoring Large Systems - Scalability issues
---------------------------------------------

o Overheads
        - One fetch per host (for thousands of hosts?)
        - Latency and time synchrony of results
        - Synchronous request/reponse model
        - NEED: one fetch per rack

o Performance intrusion
        - OS Jitter
        - Network Interrupts
        - Pull model not suitable
        - NEED: Push model within the rack -- light weight


Solution: the Cluster PMDA
--------------------------

o Cluster PMDA
        - Services pmcd requests on the rack leader
        - Manages connections from cluster node daemons
           - metric config file sent to each cluster node daemon
             when it connects.
           - see /var/lib/pcp/pmdas/cluster/config
           - Metrics list is configurable per-node by
             creating a /var/lib/pcp/pmdas/cluster/nodes/<hostname>
             file in the same format as the global 'config' file.
           - The metric list can be modified for a node on-the-fly
             by storing the desired metric config into the correct
             cluster.control.metrics instance.
        - Receives and caches performance data from the cluster
          nodes

o Client daemon (pmclusterd)
        - Run on each cluster node, dynamically loads required
          PMDAs on startup
        - Pushes data to the Cluster PMDA on the rack leader
        - Avoids OS jitter
           - setpriority
           - No unexpected network interrupts
        - Zero configuration required on cluster nodes
           - Started by rc script
           - Config set to pmclusterd at startup
        - Negligible overhead on cluster nodes
           - Sends a 1k (approx) packet to the rack leader every
             N seconds, depending on the configuration.

o Scalable
        - PCP Clients fetch from the rack leaders, not from the
          cluster nodes.
        - Avoids firewall issues, since there is no need to access
          the cluster node directly

o Zero per-node configuration
        - A "default" list of metrics is stored in a config file
          on the rack leader.
        - Custom metric list can be specified per-blade through
          the cluster.control.metrics

o per-rack instance domains for cluster nodes
        - Prefix "cluster" to all metric names
                e.g., cluster.kernel.percpu.user
        - Prefix rXiYnZ to all instance names.
           - This is very SGI specific...
           - rack X, IRU Y, Node Z
           - e.g., "r1i2n3 cpu0"

