
== Waf architecture overview

This chapter provides describes the Waf library and the interaction between the components.

=== Modules and classes

==== Core modules

Waf consists of the following modules which constitute the core library. They are located in the directory `waflib/`. The modules located under `waflib/Tools` and `waflib/extras` are only extension and are not part of the Waf core.

.List of core modules
[options="header", cols="1,6"]
|=================
|Module    | Role
|Build     | Defines the build context classes (build, clean, install, uninstall), which holds the data for one build (paths, configuration data)
|Configure | Contains the configuration context class, which is used for launching configuration tests and writing the configuration settings for the build
|ConfigSet | Contains a dictionary class which supports a lightweight copy scheme and provides persistence services
|Context   | Contains the base class for all waf commands (context parameters of the Waf commands)
|Errors    | Exceptions used in the Waf code
|Logs        | Loggging system
|Node        | Contains the file system representation class
|Options     | Provides a custom command-line option processing system based on optparse
|Runner      | Contains the task execution system (thread-based producer-consumer)
|Scripting   | Constitutes the entry point of the Waf application, executes the user commands such as build, configuration and installation
|TaskGen     | Provides the task generator system, and its extension system based on method addition
|Task        | Contains the task class definitions, and factory functions for creating new task classes
|Utils       | Contains support functions and classes used by other Waf modules
|=================

Not all modules are required for using Waf as a library, and the following diagram is provided to help including the required modules. For each module represented depends on all other modules below, for example, the module 'Node' requires both modules 'Utils' and 'Errors'. Conversely, if the module 'Build' is used alone, then the modules 'Scripting' and 'Configure' are not necessary.

image::core{PIC}["Module dependencies"{backend@docbook:,height=400:},align="center"]

==== Context classes

The context classes (which inherit from the class 'Context') are used by the Waf commands. The base class 'Context' may be used directly for the Waf commands that do not have an associated context subclass. The metaclass 'store_context' is used for recording any context subclass created.

image::classes{PIC}["Context classes"{backend@docbook:,width=500:},align="center"]

==== Build classes

The structure of the build classes is represented on the following diagram:

image::classes_build{PIC}["Build classes"{backend@docbook:,width=500:},align="center"]

=== Context objects

==== Context commands and recursion

The context commands are designed to be as independent as possible, and may be executed concurrently. The main application is the execution of small builds as part of configuration tests. For example, the method 'waflib.Tools.c_config.run_c_code' creates a private build context internally to perform the tests.
Here is an example of a build that creates and executes simple configuration contexts concurrently:

// architecture_link
[source,python]
---------------
import os
from waflib.Configure import conf, ConfigurationContext
from waflib import Task, Build, Logs

def options(ctx):
	ctx.load('compiler_c')

def configure(ctx):
	ctx.load('compiler_c')

def build(ctx):
	ctx(rule=run_test, always=True, header_name='stdio.h') <1>
	ctx(rule=run_test, always=True, header_name='unistd.h')

def run_test(self):
	top = self.generator.bld.srcnode.abspath()
	out = self.generator.bld.bldnode.abspath()

	ctx = ConfigurationContext(top_dir=top, out_dir=out) <2>
	ctx.init_dirs() <3>

	ctx.in_msg = 1 <4>
	ctx.msg('test') <5>

	header = self.generator.header_name
	logfile = self.generator.path.get_bld().abspath() + os.sep \
		+ header + '.log'
	ctx.logger = Logs.make_logger(logfile, header) <6>

	ctx.env = self.env.derive() <7>
	ctx.check(header_name=header) <8>
---------------

<1> Create task generators which will run the method 'run_test' method defined below
<2> Create a new configuration context as part of a 'Task.run' call
<3> Initialize ctx.srcnode and ctx.bldnode (build and configuration contexts only)
<4> Set the internal counter for the context methods 'msg', 'start_msg' and 'end_msg'
<5> The console output is disabled (non-zero counter value to disable nested messages)
<6> Each context may have a logger to redirect the error messages
<7> Initialize the default environment to a copy of the task one
<8> Perform a configuration check

After executing 'waf build', the project folder will contain the new log files:

[source,shishell]
---------------
$ tree
.
|-- build
|   |-- c4che
|   |   |-- build.config.py
|   |   `-- _cache.py
|   |-- config.log
|   |-- stdio.h.log
|   `-- unistd.h.log
`-- wscript
---------------

A few measures are set to ensure that the contexts can be executed concurrently:

. Context objects may use different loggers derived from the 'waflib.Logs' module.
. Each context object is associated to a private subclass of 'waflib.Node.Node' to ensure that the node objects are unique. To pickle Node objects, it is important to prevent concurrent access by using the lock object 'waflib.Node.pickle_lock'.
. Each build context is associated with a fixed amount of thread consumers. Because some interpreters cannot destroy thread objects, the consumers are returned to a thread pool after use

==== Build context and persistence

The build context holds all the information necessary for a build. To accelerate the start-up, a part of the information is stored and loaded between the runs. The persistent attributes are the following:

.Persistent attributes
[options="header", cols="1,3,3"]
|=================
|Attribute | Description                                            | Type
|root      | Node representing the root of the file system          | Node
|node_deps | Implicit dependencies                                  | dict mapping Node to signatures
|raw_deps  | Implicit file dependencies which could not be resolved | dict mapping Node ids to any serializable type
|task_sigs | Signature of the tasks executed                        | dict mapping a Task computed uid to a hash
|=================

////
==== Parallelization concerns

Nearly all the code is executed in the main thread. The other threads are merely waiting for new tasks, and executing the methods 'run' and 'install' from the task instances. As a consequence, such methods should contain as little code as possible, and access the BuildContext in a read-only manner. If such tasks must declare new nodes while executing the build (find_dir, find_resource, ..), then locks must be used to prevent concurrent access (concurrent directory and node creation).

In particular, direct _Node_ instances are not used anywhere in the Waf code. Instead, each build context creates a new Node subclass (bld.node_class), on which the build context instance is attached as a class attribute.

The TaskBase methods 'run' and 'post_run' are executed by threads ('TaskConsumer'). It is therefore discouraged to search for files and folder in these methods unless the nodes have already been created. For processing tasks dynamically during the build, it is recommended to use the TaskBase method 'runnable_status' which is executed by the main thread.

Should this be insuficient, the Task method method 'get_out' of the class Parallel should be overridden to execute methods in an event-like manner (data is attached to the task, and the method get_out executes the code).
////

=== Support for c-like languages

==== Compiled and link tasks

The tool _waflib.Tools.ccroot_ provides a system for creating object files and linking them into a single final file. The method _waflib.Tools.ccroot.apply_link_ is called after the method _waflib.TaskGen.process_source_ to create the link task. In pseudocode:

[source,shishell]
---------------
call the method process_source:
  for each source file foo.ext:
    process the file by extension
      if the method create_compiled_task is used:
        create a new task
        set the output file name to be foo.ext.o
        add the task to the list self.compiled_tasks

call the method apply_link
  for each name N in self.features:
    find a class named N:
      if the class N derives from 'waflib.Tools.ccroot.link_task':
        create a task of that class, assign it to self.link_task
        set the link_task inputs from self.compiled_tasks
        set the link_task output name to be env.N_PATTERN % self.target
        stop
---------------

This system is used for _assembly_, _C_, _C++_, _D_ and _fortran_ by default. Note that the method _apply_link_ is supposed to be called after the method _process_source_.

We will now demonstrate how to support the following mini language:

[source,shishell]
---------------
cp: .ext -> .o
cat: *.o -> .exe
---------------

Here is the project file:

// architecture_link
[source,python]
---------------
def configure(ctx):
	pass

def build(ctx):
	ctx(features='mylink', source='foo.ext faa.ext', target='bingo')

from waflib.Task import Task
from waflib.TaskGen import feature, extension, after_method
from waflib.Tools import ccroot <1>

@after_method('process_source')
@feature('mylink')
def call_apply_link(self): <2>
	self.apply_link()

class mylink(ccroot.link_task): <3>
	run_str = 'cat ${SRC} > ${TGT}'

class ext2o(Task):
	run_str = 'cp ${SRC} ${TGT}'

@extension('.ext')
def process_ext(self, node):
	self.create_compiled_task('ext2o', node) <4>
---------------

<1> This import will bind the methods such as _create_compiled_task_ and _apply_link_task_
<2> An alternate definition would be calling _waflib.TaskGen.feats[`mylink'] = [`apply_link']_
<3> The link task must be a subclass of another link task class
<4> Calling the method _create_compiled_task_

The execution outputs will be the following:
// why the extra space after "setting top to"?
[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.005s)
Setting top to   : /tmp/architecture_link
Setting out to   : /tmp/architecture_link/build
'configure' finished successfully (0.008s)
Waf: Entering directory `/tmp/architecture_link/build'
[1/3] ext2o: foo.ext -> build/foo.ext.0.o
12:50:25 runner ['cp', '../foo.ext', 'foo.ext.0.o']
[2/3] ext2o: faa.ext -> build/faa.ext.0.o
12:50:25 runner ['cp', '../faa.ext', 'faa.ext.0.o']
[3/3] mylink: build/foo.ext.0.o build/faa.ext.0.o -> build/bingo
12:50:25 runner 'cat foo.ext.0.o faa.ext.0.o > bingo'
Waf: Leaving directory `/tmp/architecture_link/build'
'build' finished successfully (0.041s)
---------------

NOTE: Task generator instances have at most one link task instance


////
==== include processing

TODO

==== uselib processing

TODO

==== local libraries

TODO


=== Execution overview

==== File system access

File system access is performed through an abstraction layer formed by the build context and 'Node' instances. The data structure was carefully designed to maximize performance, so it is unlikely that it will change again in the future. The idea is to represent one file or one directory by a single Node instance. Dependent data such as file hashes are stored on the build context object and allowed to be persisted. Three kinds of nodes are declared: files, build files and folders. The nodes in a particular directory are unique, but build files used in several variant add duplicate entry on the build context cache.

To access a file, the methods 'Node::find_resource', 'Node::find_build' (find an existing resource or declare a build node) and 'Node::find_dir' must be used. While searching for a particular node, the folders are automatically searched once for the files. Old nodes (which do not have a corresponding file) are automatically removed, except for the build nodes. In some cases (lots of files added and removed), it may be necessary to perform a 'Waf clean' to eliminate the information on build files which do not exist anymore.

==== Task classes

The whole process of generating tasks through Waf is performed by methods added on the class task_gen by code injection. This often puzzles the programmers used to static languages where new functions or classes cannot be defined at runtime.

The task generators automatically inherit the build context attribute 'bld' when created from bld(...). Likewise, tasks created from a task generator (create_task) automatically inherit their generator, and their build context. Direct instantiation may result in problems when running Waf as a library.

The tasks created by task generator methods are automatically stored on the build context task manager, which stores the task into a task group. The task groups are later used by the scheduler to obtain the task which may run (state machine). Target (un)installation is performed right after a task has run, using the method 'install'.

=== Performance and build accuracy

From the experience with tools such as SCons, users may be concerned about performance and think that all build systems based on interpreted languages such as Python would not scale. We will now describe why this is not the case for Waf and why Waf should be chosen for building very large projects.

==== Comparing Waf against other build systems

Since Waf considers the file contents in the build process, it is often thought that Waf would be much slower than make. For a test project having 5000 files (generated from the script located in `tools/genbench.py`), on a 1.5Ghz computer, the Waf runtime is actually slightly faster than the Gnu/Make one (less than one second). The reason is the time to launch a new process - make is usually called recursively, once by directory.

For huge projects, calling make recursively is necessary for flexibility, but it hurts performance (launch many processes), and CPU utilization (running tasks in parallel). Make-based build systems such as CMake or Autotools inherit the limitations of Make.

Though Waf uses a similar design as SCons, Waf is about 15 times faster for similar features and without sacrificing build accuracy. The main reasons for this are the following:

. The Waf data structures (file system representation, tasks) have been carefully chosen to minimize memory usage and data duplication
. For a project of the same size, SCons requires at least 10 times as many function calls

A few benchmarks are maintained at http://freehackers.org/~tnagy/bench.txt[this location]

==== Waf hashing schemes and build accuracy

To rebuild targets when source file change, the file contents are hashed and compared. The hashes are used to identify the tasks, and to retrieve the files from a cache (folder defined by the environment variable 'WAFCACHE'). Besides command-lines, this scheme also takes file dependencies into account: it is more accurate than caching systems such as 'ccache'.

The Waf hashing scheme uses the md5 algorithm provided by the Python distribution. It is fast enough for up to about 100Mb of data and about 10000 files and very safe (virtually no risk of collision).

If more than 100Mb of data is present in the project, it may be necessary to use a faster hashing algorithm. An implementation of the fnv algorithm is present in the Waf distribution, and can replace md5 without really degrading accuracy.

If more than 10000 files are present, it may be necessary to replace the hashing system by a 'file name+size+timestamp hash scheme'. An example is provided in the comment section of the module `Utils.py`. That scheme is more efficient but less accurate: the Waf cache should not be used with this scheme.
////

