16.5 So Implementational Details

The following is a chart describing the flow of processing in the So algorithm, starting with the epoch process, since higher levels do not interact with the details of particular algorithms:

EpochProcess: {
  Init: {
    environment->InitEvents();          // init events (if dynamic)
    event_list.Add() 0 to environment->EventCount(); // get list of events
    if(order == PERMUTE) event_list.Permute();       // permute if necessary
    GetCurEvent();                      // get pointer to current event
  }
  Loop (trial): {                      // loop over trials
    SoTrial: {                         // trial process (one event)
      Init: {                          // at start of trial
        cur_event = epoch_proc->cur_event; // get cur event from epoch
      }
      Loop (once): {                   // only process this once per trial
        network->InitExterns();         // init external inputs to units
        cur_event->ApplyPatterns(network); // apply patterns to network
        Compute_Act(): {               // compute the activations
          network->layers: {           // loop over layers
            layer->Compute_Net();       // compute net inputs
            layer->Compute_Act();       // compute activations from net in
          }
        }
        network->Compute_dWt();         // compute weight changes from acts
      }
    }
    if(wt_update == ON_LINE or wt_update == SMALL_BATCH and trial.val % batch_n)
      network->UpdateWeights(); // after trial, update weights if necc
    GetCurEvent();              // get next event
  }
  Final:
    if(wt_update == BATCH)  network->UpdateWeights(); // batch weight updates
}

The layer->Compute_Act() function has several sub-stages for different versions of algorithms, as detailed below:

For non-input layer hard competitive learning units:

ClLayerSpec::Compute_Act() {
  SoUnit* win_u = FindWinner(lay);
  float lvcnt = (float)lay->units.leaves;
  lay->avg_act =  // compute avg act assuming one winner and rest losers..
        (act_range.max / lvcnt) + (((lvcnt - 1) * act_range.min) / lvcnt);
  win_u->act = act_range.max; // winning unit gets max value
  win_u->act_i = lay->avg_act;	// and average value goes in _i 
}

For non-input layer soft competitive learning units:

SoftClLayerSpec::Compute_Act() {
  float sum = 0;
  for(units) {                 // iterate over the units 
    unit->Compute_Act();                        // compute based on netin
    unit->act = exp(softmax_gain * unit->act);  // then exponential
    sum += unit->act;                           // collect sum
  }
  for(units) {                 // then make a second pass
    unit->act =                 // act is now normalized by sum
        act_range.min + act_range.range * (unit->act / sum);
  }
  Compute_AvgAct();             // then compute average act over layer
}

The code for the SOM case is more complicated than the description, which is just that it finds the winner and pastes the kernel onto the units surrounding the winner.