Otter Documentation

Formal Specification

OtterScript Banner

This document describes the specific behavior of the Inedo execution engine (used by Otter and BuildMaster), and is primarily intended as a reference. It will probably not be helpful in learning how to create plans, although it may explain why particular "block" behaves in a certain way, and why a plan's "Text Mode" (i.e. OtterScript) looks a bit different than the visual version.

BuildMaster note - because BuildMaster is designed to deploy applications to servers, it's usage of the execution engine is closer to how orchestration jobs work in Otter; BuildMaster-specific behavior will be denoted in blocks like this.

General Notes

Plan Editor & Abstraction

To help make Infrastructure as Code as accessible as possible, the Plan Editor and the corresponding documentation abstract some of the implementation details.

  • General Block - this is a combination of Anonymous Block Statements, Set Context Statements, and Execution Directive Statements
  • Loop Block - this is either a Context Iteration Statement or a Iteration Block Statement

Terminology: Plan vs OtterScript

These terms are mostly interchangeable. OtterScript refers to the textual representation of a plan, and a plan is an abstract syntax tree of an OtterScript. This document describes the elements of a plan, whereas the Formal Grammar describes the grammar of a valid OtterScript.

Execution Engine

The execution engine evaluates statements in a plan, in the order they appear. Whenever an error is raised, the execution status will change to failing and the execution will halt; this will cause the execution to terminate, unless the error occurred within a Try Statement

Execution Properties

An execution has the following properties.

  • Run State - indicates current running state of the execution: Pending, Executing, Completed
  • Status - status to report to the user: Normal, Warning, Error
  • Execution Mode - Collect Only, Collect And Execute,Execute Only
  • Simulation - true or false

Execution Modes

The execution engine can run a plan in three different modes:

  • Collect Only- only ICollectingOperation operations will run; if the operation is a IComparingOperation, then drift may be indicated. All ensure operations implement both interfaces.
  • Collect then Execute - a collection pass is performed as described above; if any drift is indicated, an execution pass is performed that runs:
    • operations that indicated drift
    • IExecutingOperation operations in the same scope as a drift-indicating operation that do not implement IComparingOperation; this is all execution actions
    • operations with an execution policy of AlwaysExecute; this can only be set on a Context Setting Statement
  • Execute Only- only IExecutingOperation operations will run; all ensure and execute operations implement this interface

BuildMaster plans execute in a Execute Only mode

Simulated Executions

Any execution may be run a a simulation; when this is done, operations are responsible for respecting this flag. The agent will automatically return a IFileOperationsExecuter that runs in simulation mode.

Configuration and Orchestration Plans

The service will create executions from three sources. In each case, the service wraps the user's plan(s) in different statements.

Type TimedExecuter Mode
Routine Configuration Check RoutineConfiguration CollectOnly or CollectAndExecute
Configuration Job JobDispatcher CollectAndExecute
Orchestration Job JobDispatcher ExecuteOnly

A Routine Configuration Check will create a distinct execution per server. Each of these executions will be of a plan comprised of single Set Context Statement (with a ContextType of server, and ContextValue of the server name); the Body of that statement will be comprised of each of the role's configuration plans (wrapped in a Set Context Statement for "role" and the role name), followed by the server's configuration plan (wrapped in an Anonymous Block Statement).

A Configuration Job targets servers directly or indirectly (role and/environment). A list of servers will be gathered, and a plan comprising of a single Context Iteration Statement will be created with the Source set to a literal expression of the server names (e.g. @(Server1, Server2, Server3)). The Body contain an Execution Directive Statement with an Asynchronous flag, and the Body of that will contain the actual plan.

An Orchestration Job may target servers, or not. If there are servers targeted, then the above logic is performed. If no servers are specified, the plan is executed directly.

BuildMaster plans are executed in the same manner as Orchestration Jobs; they are run by the PlanActionExecuter executer and run in ExecuteOnly mode

Plan Elements & Statements

A plan is essentially a Scoped Statement Block with Additional Headers.

Additional Headers

A set of strings that may metadata for the plan. These are effectively special comments that are not currently used by the execution engine, but are used by the plan editor to determine whether to display in visual or text mode, etc.

Scoped Statement Block

This is a grouping of statements and named template methods; it is not a statement itself, but an element of several "block" statements. Both runtime variables and templates declared in a scoped statement block are accessible to all nested scopes, and a plan is simply a scoped statement block.

Named Templates

A Named Template is like a method or subroutine, and are generally stored external to a plan. They may be declared within a Scoped Statement Block, in which case they are only accessible to are accessible only within the block and nested blocks.

A Named Template is comprised of:

  • Name – a valid name
  • Body – a scoped statement block
  • Parameters – a string-keyed dictionary of Template Arguments

A Template Argument is comprised of a Name, Output Indicator, and Default Value.

Statements

There are several different statement types; the only common element between statements is a description (see comments and descriptions).

Action Statement

This invokes an Operation, and is comprised of:

  • Action Name – a qualified name of the operation's namespace and script alias
  • Arguments – a string-keyed dictionary of string values
  • Positional Arguments – a list of strings; Otter operations currently support only a single positional argument (a property with a DefaultPropertyAttribute)
  • Output Arguments – a string-keyed dictionary of runtime variable names

When this statement is encountered, the execution engine does the following.

  1. Attempt to load the Operation with from the ScriptAlias and ScriptNamespace; an error is raised if cannot be found
  2. A log scope is created using the Operation's description
  3. The Arguments and Positional Arguments are mapped to either the Operation's template's properties (for EnsureOperation<TConfig> operations) or the Operation's properties using the property's ScriptAlias
  4. Unless the property has a DisableVariableExpansionAttribute, the values are evaluated
  5. The values are assigned to the mapped property
  6. During a collection run, if the Operation type is:
    1. Collecting: the Collect method is invoked
    2. Ensure: the Collect, Store, then Compare methods are invoked
    3. Intrinsic: the Execute method is invoked
  7. During an execution run, if the Operation type is:
    1. Executing or Intrinsic: the Execute method is invoked
    2. Ensure, and the execution type is Execute Only: then the Execute method is invoked
    3. Ensure, and the execution type is Collect and Execute: if the Compare method reported differences, then the Execute method is invoked
  8. If an exception occurred within one of the above methods or a log message was written with an error level, then the execution state will change to Failing and an error will be Raised
  9. If a log message was written with a warning level, then the execution state will change to Warning unless it's already in a Failing state
  10. The Output Arguments are mapped from the Operation or Configuration using the Script Alias, and the specified runtime variables are created or assigned
  11. The log scope is closed

BuildMaster does support Ensure operations (as well as Configuration Templates); however, this is really just for code-sharing purposes. They are effectively just Execute operations and run as if it were a Execute Only plan run.

Anonymous Block Statement

This is comprised of a single, Scoped Statement Block. When this statement is encountered, the execution engine does the following.

  1. If the statement has a description, creates a log scope with the first line of that description
  2. Executes the Scoped Statement Block
  3. Closes the log scope if it was created

Assign Variable Statement

This is comprised of a Variable Name and a Variable Value. When this statement is encountered, the execution engine does the following.

  1. Looks for a runtime variable created in the current or parent scopes
  2. If a runtime variable was not found, a new runtime variable is created for the current (and nested) scope
  3. The Variable Value is evaluated; if the expression type does not match the variable type (Scalar, Vector, Map), an error is raised; otherwise, the value is assigned to the found or created runtime variable

Note that if a configuration variable already exists (such as a Server or Environment variable), this statement will still create a runtime variable, which will override the configuration variable in current and nested scopes.

Note that, unless a variable is in the global scope (i.e. declared with a Declare Global Variable Statement), asynchronously-running blocks create a copy of the parent scope's runtime variables, so assigning an runtime variable in an asynchronous block will not update the parent, non-asynchronous scope.

Await Statement

This is comprised of an optional Token name. When this statement is encountered, the execution engine does the following.

  1. Looks for all child execution engines created from an Execution Directive Statement
  2. If a token is specified, engines without the matching token are ignored
  3. If no matches were found, a warning message is logged
  4. Execution pauses until the child engines complete
  5. If the execution state of the child execution engine is error/warning, the parent execution engine inherits this state. If the state is "failing," and the await statement is inside a TryStatement, its catch handler will be executed.

Call Template Statement

This invokes a Named Template, and is comprised of:

  • Template Name – a qualified name of the template's raft container and declared name
  • Arguments –  a string-keyed dictionary of value strings

When this statement is encountered, the execution engine does the following.

  1. If no raft name is specified, searches for a Named Template in the current scope, and then each parent scopes
  2. If no template was found, searches for a template file in the specified raft container, or the current raft
  3. If no template was found, an error is raised
  4. The found template's arguments are then enumerated
    1. If no matching argument was in the statement, and the template argument does not have a default value, then an error is raised
    2. If no matching argument was in the statement, but the template argument has a default value, then the argument is created as a runtime variable in the current scope
    3. If a matching argument was found, then the value is evaluated and created as a runtime variable in the current scope
  5. If the statement has a description, creates a log scope with the first line of that description
  6. Executes the Body of the Template
  7. Closes the log scope if it was created

BuildMaster does currently have rafts; instead, the templates are qualified as either global or «application-name».

Context Iteration Statement

This enumerates a vector expression, changes the context each time, and executing the contained statements. It's comprised of

  • Context Type – either directory, role, or server
  • Source – a valid vector expression
  • Body –  a scoped statement block

BuildMaster also supports a deployable Context Type.

When this statement is encountered, the execution engine does the following:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. Evaluates the Source; if the expression is not a vector, then an error is raised
  3. Reads the current item in Source; the same logic is performed as Set Context Statement, as if it were the Context Value
  4. If the statement has a description, creates a log scope with current item
  5. Executes the Body
  6. Closes the log scope if created
  7. Moves next on the Source; if there is a current item, Step 2 is performed
  8. Closes the outer log scope if created

Declare Global Variable

This is comprised of a Variable Name and an optional Variable Value. When this statement is encountered, the execution engine does the following.

  1. If a statement other than a Declare Global Variable statement has been executed, raises a fatal (uncatchable) error
  2. Initializes a new runtime variable in the global scope
  3. If there is a Variable Value, it is evaluated; if the expression type does not match the variable type (Scalar, Vector, Map), an error is raised; otherwise, the value is assigned to the runtime variable
  4. If there is not a Variable Value, an empty string, vector, or map is assigned to the variable

Variables declared in the global scope are assignable by asynchronously-running blocks.

Error Statement

This has no elements; when this statement is encountered, the execution engine changes the execution status to Failing.

Execution Directive Statement

This instructs the executer to change the execution behavior of the contained statements. It's comprised of

  • Retry Count – an integer that specifies the times to retry the body; the default is 0
  • Asynchronous – a flag that indicates the body should run asynchronously
  • AsyncToken – a token name that can be awaited on
  • LockToken – a token name (or empty string) that may be prefixed with ! to indicate a global lock
  • Timeout – the number of seconds to attempt the block for
  • Execution Policy – either always or onChange
  • Additional Flags – currently unused
  • Body – a scoped statement block

When this statement is encountered, the execution engine does the following:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. If Execution Policy is set, all Ensure Operations will execute in the execution phase of a Collect and Execute phase, regardless of whether changes were detected
  3. If a Retry value is specified, and a non-fatal error is encountered during any of the following steps (and the retry count hasn't been exceeded), then the execution status is reverted to whatever status it was upon entering the block, and execution begins again from the following step
  4. If a Timeout value is specified, a CancellationTokenSource is created with the specified value; if the timeout has elapsed before the remainder of the statement is executed, a CancellationToken will be created that signals the current task to safely abort
  5. If a Lock Token is specified, then
    1. If the current statement is within a block that specifies a Locked flag, a fatal error is raised
    2. If there are any other statements that are currently executing in a statement that shares a LockTocken, either in the current execution (or in all executions when the token is prefixed with a !), wait until the other statements finish
  6. If Asynchronous is specified, then
    1. A new child execution engine is created with the specified token
    2. All current runtime variables from the current execution are copied to the child
    3. The child execution engine then executes the Body
    4. If an error is encountered during the child execution, the error will be raised in the parent execution engine, where the with block began
  7. Otherwise (no async), the Body is executed
  8. Closes the outer log scope if created

Note that timeout (or cancellation in general) is not guaranteed, and requires that the task (generally, an operation) properly check for a cancellation token. For example, an operation that simply did a Thread.Sleep for 1000 seconds could not be timed out after 10 seconds.

Fail Statement

This Statement has no elements; when this statement is encountered, the execution engine changes the execution status to Failing and immediately terminates the execution; recovery is not possible, even if within a Try/Catch statement.

Force Normal Statement

This Statement has no elements; when this statement is encountered, the execution engine changes the execution status to Normal.

Iteration Block Statement

This enumerates a vector expression, changes the context each time, and executing the contained statements. It's comprised of

  • Iteration Variable Name – a valid scalar variable expression
  • Source – a valid vector expression
  • Body –  a scoped statement block

When this statement is encountered, the execution engine does the following:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. Evaluates the Source; if the expression is not a vector, then an error is raised
  3. Reads the current item in Source; creates a runtime variable with the specified name in the current scope
  4. If the statement has a description, creates a log scope with current item
  5. Executes the Body
  6. Closes the log scope if created
  7. Moves next on the Source; if there is a current item, Step 2 is performed
  8. Closes the outer log scope if created

Note that a runtime variable is always created, never assigned. This means that if a runtime variable was already created in a higher scope, that value would not be overwritten.

Log Statement

This statement has two elements:

  • Log Message – an expression
  • Log Level – an integer enum with valid values of (0=Debug, 10=Information, 20=Warning, 30=Error)

When this statement is encountered, the execution engine writes the specified message to the specified level. If Warning or Error is specified, the execution status will change appropriately (Error causes Failing, Warning causes Warning unless already Failing), but an error will not be raised.

Predicate Statement

This enumerates changes the current execution context. It's comprised of

  • Predicate – one of the six types of predicates
  • True Statements – a scoped statement block
  • False Statements – a scoped statement block

There are six types of Predicates; each can evaluate as true or false.

  • And Predicate – contains a set of Predicates; evaluates as true if all contained predicates evaluate as true, otherwise false
  • Or Predicate – contains a set of Predicates; evaluates as true if any of the contained predicates evaluate as true, otherwise false
  • Not Predicate – contains a single Predicate; evaluates as true if the contained predicate evaluates as false; otherwise true
  • Equality Predicate – contains two expressions; evaluates as true if both evaluated expressions are (case sensitively) identical
  • Inequality – contains two expressions; evaluates as true if both evaluated expressions are (case sensitively) not identical
  • True – contains a single expression; evaluates as true if the expression is (case insensitively) identical to “true”

When this statement is encountered, the execution engine does the following:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. Tests the Predicate, and evaluates either True Statements or the False Statements
  3. Closes the outer log scope if created

Set Context Statement

This changes the current execution context. It's comprised of

  • Context Type – either directory, role, or server
  • Context Value – an expression
  • Body –  a scoped statement block

When this statement is encountered, the execution engine does the following:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. Evaluates the Context Value; if the Context Type is
    1. directory – creates a new context based on current context, but with the current item as the working directory
    2. role – if the server in context does not have the specified role, then the entire block is skipped; otherwise, creates a new context based on the current context with the current item as the specified role
    3. server - if the execution allows it (currently, only Orchestration Jobs that don't target any servers), then creates a new context based on the current context with the current item as the specified server; otherwise, raises an error
  3. Closes the outer scope if created

BuildMaster supports the deployable Context Type as well, and it's behavior is as follows: if the release in context does not have the specified deployable, then the entire block is skipped; otherwise, creates a new context based on the current context with the current deployable as the specified deployable

Throw Statement

This has no elements; when this statement is encountered, the execution engine changes the execution status to Failing and raises an error. This error may be caught if within a Try/Catch statement.

Try Statement

This has two scoped statements blocks: a Body, and an Error Handler. When this statement is encountered by the execution engine:

  1. If the statement has a description, creates a log scope with the first line of that description
  2. If an error occurs during the following step, the Error Handler is executed instead of the normal behavior of raising an error and changing the execution status
  3. The Body is executed
  4. Closes the log scope if it was created

Warn Statement

This Statement has a single, optional element: Force flag. When this statement is encountered, the execution engine changes the execution status to Warning, unless the Force flag is not specified and the current execution status is Failing.