SimpleUAM Configuration System

SimpleUAM uses an internal configuration system based on OmegaConf. It reads YAML files in a platform specific directory for settings that are used throughout the system. In addition there is a utility, suam-config, provided with this repo to help manage configuration information.

All commands are run in <repo-root>.

Configuration Semantics

The SimpleUAM configuration model is based on a few core principles:

• Configuration variables are split into two categories:
  • Per-machine: These are unique to a single machine, but are shared between all the processes or tasks on that machine.
  • Per-deployment: These are unique to single deployment are can be shared without modification between all the machines in a deployment.
• Configuration files are split into per-machine and per-deployment categories as well so that they can respectively be defined locally and linked from a central shared location.
• At a per-machine level there is a single opinionated config directory where all config files must be found.
• Config files come with functional defaults whenever possible.
• Config files can be defined partially, with missing fields being populated by defaults or lower priority config files.

Together these guidelines give us a basic model of configuration semantics based on a fixed root location for files, incremental overloading, and linking of shared configuration into the fixed root.

Root Directory

Config files are all located in an fixed root directory. Run the following to find that directory:

pdm run suam-config dir

See this section for more information on the command and here for information on the specific config files you can find there.

Overloading

You can incrementally define configurations with multiple files that overload each other.

There are 3 general levels of configuration options, from lowest to highest priority.

• Default: These are defined in the code as a python class. Attributes in that class correspond to fields of the config file.
• Base Config: This is the base configuration directory as found with pdm run suam-config dir.
• CLI Argument Config: This is the config dir added to the stack when a command is given --config-dir=<config-dir-arg> as an argument.

For example consider the case for a hypothetical foo.conf.yaml and the raw definitions of each configuration:

DefaultBase ConfigCLI Argument Config

from attrs import define

@define
class FooConfig():

    name : str = "John"
    age  : int = 23
    occupation : str = "Example Salesman"
    hobby : str = "Sampling"

name: Andy
hobby: Cooking

name: Jane
age: 32

And the corresponding config as seen by the program in each case.

DefaultBase ConfigCLI Argument Config

These are configuration values when:

• No config-dir/foo.conf.yaml exists
• No --config-dir=config-dir-arg argument has been used

name: John
age: 23
occupation: Example Salesman
hobby: Sampling

These are the loaded values when:

• config-dir/foo.conf.yaml exists
• --config-dir=config-dir-arg isn't an argument

name: Andy
age: 23
occupation: Example Salesman
hobby: Cooking

Note how only the values in <config-dir>/foo.conf.yaml change with the rest remaining at their defaults.

These are the loaded values when

• config-dir/foo.conf.yaml exists
• config-dir-arg/foo.conf.yaml exists
• A --config-dir=config-dir-arg argument was given

name: Jane
age: 32
occupation: Example Salesman
hobby: Cooking

Look here for information on the specific config files, their source locations, expected filenames, and key fields.

Linking

The ideal way to work with per-deployment configs is to put them all in a single location, accessible from all the nodes in your deployment, and symlink them into the root configuration directories of each machine.

See the install command, which makes that process easier. Also see the individual config files for information on whether each file is per-machine or per-deployment.

Configuration File Format

All SimpleUAM configuration files are in YAML. See this guide for a quick introduction to the format.

Variable Interpolation

We support OmegaConf's variable interpolation to allow referencing values in one config file from another.

Terms between ${ and } in a configuration variable are interpreted as a reference to another field in the current configuration or, with the appropriate key, a field in another configuration file.

Interpolation happens after all config files are loaded and all overloading is finished.

Basic Interpolation

Within a single file you can interpolate variables at the same level:

field1: Test
field2: Field1 is '${field1}'
field3: 123
field4: ${field3}
subsection:
  field1: Other Value
  field2: Sibling Field1 is '${field1}'

Which resolves to:

field1: Test
field2: Field1 is 'Test'
field3: 123
field4: 123
subsection:
  field1: Other Value
  field2: Sibling Field1 is 'Other Value'

Advanced Interpolation

You can also look at parent and child fields with interpolation:

field1: Test
field2: Child Field1 is '${subsection.field1}'
subsection:
  field1: Other Value
  field2: Parent Field1 is '${.field1}'

Which resolves to:

field1: Test
field2: Child Field1 is 'Other Value'
subsection:
  field1: Other Value
  field2: Parent Field1 is 'Test'

Cross-File Interpolation

Each SimpleUAM config file also comes with an interpolation key, which can be used to reference its fields in other locations.

Consider the following config with the interpolation key "example":

parent-field: Example Parent
subsection:
  sub-field: Example Child

And the following raw config:

field1: Parent is '${example:parent-field}'
field2: Child is '${example:subsection.sub-field}'

Which resolves to:

field1: Parent is 'Example Parent'
field2: Child is 'Example Child'

See each config file's info to find out their interpolation keys.

Using the suam-config CLI

There is an suam-config script included with SimpleUAM for manipulating configuration files.

You can get a list of all of its sub-commands by running:

pdm run suam-config --list

Each sub-command has help output that can be accessed with:

pdm run suam-config <sub-command> --help

More details on each sub-command follow.

The dir sub-command

The dir command prints out the current, highest-priority config directory. It can be run as follows:

pdm run suam-config dir

Arguments:

• --all/-a: This prints out a list of all the config directories that are being read from lowest to highest priority. Only useful when called with --config-dir=.

The list-* sub-commands

The various list-* commands print out information on each of the configuration files. The outputs can be used to specify individual config files for the file, print, write, and install commands.

list-files

Prints a list of recognized configuration files that are searched for in each configuration directory. It can be run as follows:

pdm run suam-config list-files

Sample Output of pdm run suam-config list-files

paths.conf.yaml
auth.conf.yaml
win_setup.conf.yaml
craidl.conf.yaml
corpus.conf.yaml
d2c_workspace.conf.yaml
broker.conf.yaml
d2c_worker.conf.yaml

Each output filename can be used as a config file identifier in other commands.

list-keys

Prints a list of recognized interpolation keys. It can be run as follows:

pdm run suam-config list-keys

Sample Output of pdm run suam-config list-keys

path
auth
win_setup
craidl
corpus
d2c_workspace
broker
d2c_worker

Each interpolation key can be used in configuration files and as a config file identifier in other commands.

list-classes

Prints a list of python classes corresponding to each configuration file. It can be run as follows:

pdm run suam-config list-classes

Sample Output of pdm run suam-config list-classes

PathConfig
AuthConfig
WinSetupConfig
CraidlConfig
CorpusConfig
D2CWorkspaceConfig
BrokerConfig
D2CWorkerConfig

Each python class name can be used as a config file identifier in other commands.

The file sub-command

The file sub-command is the single-file counterpart to the dir sub-command. It prints out where SimpleUAM is looking for a configuration file. It can be run as follows:

pdm run suam-config file -c <config-id>

Arguments:

• --config=STRING/-c STRING (Mandatory) : The config file whose search location you're asking for. "STRING" can be any of the output lines from the list-* subcommands.
• --all/-a: This prints out a list of all the config file that are being read from lowest to highest priority. Only useful when called with --config-dir=.

The print sub-command

The print sub-command will print out the currently loaded configuration after all input files have been read and processed. This is the configuration as other parts of SimpleUAM see it if run in similar circumstances.

To print a specific configuration use:

pdm run suam-config print -c <config-id>

With <config-id> as one of the outputs from a list-* command. This flag can be given multiple times in order to print multiple config files, as follows:

pdm run suam-config print -c path -c auth

To print all the configurations use:

pdm run suam-config print --all

Add the --resolved flag to print the configurations after interpolations have been applied.

Arguments:

• --config=STRING/-c STRING (Mandatory) : The config file to be printed out. "STRING" can be any of the output lines from the list-* subcommands. Can be given multiple times to print out multiple configs.
• --all/-a: This prints out a list of all the config file that are being read from lowest to highest priority. Only useful when called with --config-dir=.
• --resolved/-r: Instead of printing out the unevaluated interpolations, the default, resolve all the fields before printing.

The write sub-command

the write sub-command will write the currently loaded configuration to the default configuration directory. This is a good way to initialize the configuration files for a machine. By default this will only write fully commented configuration files to disk.

To write out all config files to the default configuration directory use:

pdm run suam-config write

Individual and multiple config files can be specified in the same way as print.

Arguments:

• --config=STRING/-c STRING: The config file to be written out. "STRING" can be any of the output lines from the list-* subcommands. Can be given multiple times to print out multiple configs.
• --no-comment: Don't comment the output files.
• --overwrite/-o: Don't skip files that exist, overwrite them with the new versions instead.
• --output=DIR/-u DIR: Write config files out to DIR instead of the default config directory.

The install sub-command

The install sub-command links or copies config files from a provided directory to the default config directory. Note that config files being copied must have names that match the ones from the list-files sub-command.

To symlink all the config files from <dir> into the config directory run:

pdm run suam-config install -i <dir>

To copy all the config files from <dir> into the config directory run:

pdm run suam-config install -i <dir> --no-symlink

To symlink individual files run:

pdm run suam-config install -i <file>

Arguments:

• --input=STRING/-i STRING (Mandatory): A config file or directory to copy into the default search location. If STRING is a directory the argument can only be given once, if a file then it can given multiple times.
• --no-symlink: Don't create symlinks, just copy the files over.
• --overwrite/-o: Don't skip files that exist, overwrite them with the new versions instead.

Configuration Files

Some basic information on each config file

paths.conf.yaml

Defines paths that SimpleUAM will use for caches, logs, data, etc...

Most people shouldn't need to change this. Locations for more specific tasks are defined in their respective configuration files and changing those should suffice.

Changes to this config will require reinstallation or rebuilding large portions of a worker node. Try to keep this static after initial setup.

Properties

• Python Dataclass: simple_uam.util.config.path_config.PathConfig (API)
• Purview Category: Per-Machine
• Config File Name: paths.conf.yaml
• Interpolation Key: path

Contents

Default paths.conf.yaml

### paths.conf.yaml ###

### NOTE: All defaults are system specific

config_directory: ...
cache_directory: ...
log_directory: ...
work_directory: ...
data_directory: ...

Key Fields

• config_directory: This will add another configuration directory to the chain of overloaded config directories. Try to avoid setting this without good reason.
• cache_directory: Location for files which are used in install and execution but can be regenerated after deletion.
• log_directory: Location to place any logs that are generated by SimpleUAM.
• work_directory: Location for files that are used as various SimpleUAM commands and programs run.
• data_directory: Location for static files that need to present at runtime but which aren't, usually, directly modified.

auth.conf.yaml

Authentication keys or tokens needed by a SimpleUAM install.

Try not to check this into a repo or make it public.

Properties

• Python Dataclass: simple_uam.util.config.auth_config.AuthConfig (API)
• Purview Category: Per-Deployment
• Config File Name: auth.conf.yaml
• Interpolation Key: auth

Contents

Default auth.conf.yaml

### auth.conf.yaml ###

isis_user: null
isis_token: null

Example auth.conf.yaml

### auth.conf.yaml ###

isis_user: myIsisUsername
isis_token: 'glpat-XXXXXXXXXXXXXXXXXXXX'

Key Fields

• isis_user: Either null or your username for the Isis GitLab instance.
• isis_token: Either null or an API token for the Isis GitLab instance.

corpus.conf.yaml

The repositories and git refspecs used in an install of SimpleUAM. These can be changed to specify the versions of any SWRi code used and make a deployment more repeatable.

Properties

• Python Dataclass: simple_uam.util.config.corpus_config.CorpusConfig (API)
• Purview Category: Per-Deployment
• Config File Name: corpus.conf.yaml
• Interpolation Key: corpus

Contents

Default corpus.conf.yaml

### corpus.conf.yaml ###

trinity:
  repo: https://git.isis.vanderbilt.edu/SwRI/ta1/sri-ta1/trinity-craidl.git
  branch: main
  examples_dir: examples
graphml_corpus:
  repo: https://git.isis.vanderbilt.edu/SwRI/athens-uav-workflows.git
  branch: uam_corpus
  graphml_file: ExportedGraphML/all_schema_uam.graphml
creopyson:
  repo: https://git.isis.vanderbilt.edu/SwRI/creoson/creopyson.git
  branch: main
creoson_server:
  api: https://git.isis.vanderbilt.edu/api/v4/projects/499/jobs/3827/artifacts/out/CreosonServerWithSetup-2.8.0-win64.zip
  manual: https://git.isis.vanderbilt.edu/SwRI/creoson/creoson-server/-/jobs/artifacts/main/raw/out/CreosonServerWithSetup-2.8.0-win64.zip?job=build-job
direct2cad:
  repo: https://git.isis.vanderbilt.edu/SwRI/uam_direct2cad.git
  branch: main

Key Fields

• trinity: Defines the location and version of the trinity-craidl repo to copy example designs from.
• graphml_corpus: Defines where to find the graphml dump of the component and design corpus.
• creopyson: Defines where to find the python package with Creoson bindings.
• creoson_server: URLs for both API (GitLab Token) based and manual downloads of the modified creoson server used by direct2cad.
• direct2cad: Defines the location of the core direct2cad repo used for a deployment.

win_setup.conf.yaml

Options for packages to install on various windows nodes.

Should be left unchanged by most, this mainly exists to make larger scale deployment easier to customize.

Properties

• Python Dataclass: simple_uam.util.config.win_setup_config.WinSetupConfig (API)
• Purview Category: Per-Deployment
• Config File Name: win_setup.conf.yaml
• Interpolation Key: win_setup

Contents

Default win_setup.conf.yaml

### win_setup.conf.yaml ###

global_dep_packages:
- checksum
- wget
- 7zip
broker_dep_packages:
- rabbitmq
worker_dep_packages:
- openjdk11
- rsync
- nssm
worker_pip_packages:
- psutil
- parea
- numpy
license_dep_packages: []
graph_dep_packages:
- openjdk11
- nssm
qol_packages:
- firefox
- notepadplusplus
- foxitreader
- tess
- freecad

Key Fields

• global_dep_packages: Chocolatey packages to install on all node, please only add to this list.
• qol_dep_packages: QoL chocolatey packages.
• *_dep_packages: A list of chocolatey packages to install on the corresponding type of node. Please only add to this list.
• worker_pip_packages: Pip packages to be installed globally on the worker (as opposed to within a venv via PDM), needed for direct2cad.

craidl.conf.yaml

Settings for working with a corpus DB or static corpus.

Properties

• Python Dataclass: simple_uam.util.config.craidl_config.CraidlConfig (API)
• Purview Category: Per-Machine
• Config File Name: craidl.conf.yaml
• Interpolation Key: craidl

Contents

Default craidl.conf.yaml

### craidl.conf.yaml ###

example_dir: ${path:data_directory}/craidl_examples
stub_server:
  cache_dir: ${path:cache_directory}/corpus_stub_cache
  server_dir: ${path:data_directory}/corpus_stub_server
  graphml_corpus: ${path:data_directory}/corpus_stub.graphml
  host: localhost
  port: 8182
  read_only: false
  service:
    priority: NORMAL
    exit_action: Restart
    restart_throttle: 5000
    restart_delay: 1000
    redirect_io: false
    stdout_file: ${path:log_directory}/craidl_stub_db/stdout.log
    stderr_file: ${path:log_directory}/craidl_stub_db/stderr.log
    rotate_io: true
    auto_start: false
    console: true
    interactive: false
server_host: localhost
server_port: ${stub_server.port}
static_corpus: ${path:data_directory}/corpus_static_dump.json
static_corpus_cache: ${path:cache_directory}/static_corpus_cache
use_static_corpus: true

Key Fields

• example_dir: Location for saved example designs. Possibly better thought of as a static design corpus to live alongside the static component corpus.
• stub_server: Settings for running the TinkerPop server reference implementation with a provided graphml corpus dump.
• graphml_corpus: The graphml corpus file to load on server start.
• host: The host to serve the graph database to. localhost will only serve to clients on the same machine. 0.0.0.0 will serve to any client.
  • port: Port to server the database on.
  • read_only: Should the stub server allow writes? Note that even if false no writes will be persisted, the state of the DB is reset on each run.
• server_host: The corpus DB to connect to when generating info files or creating a static component corpus.
• server_port: The port of the corpus DB to connect to for various tasks.
• static_corpus: The static corpus to use when performing various tasks.
• static_corpus_cache: The location of the cache used when generating a new static corpus from the corpus DB.
• use_static_corpus: Use the static corpus when possible if true otherwise default to using a running corpus DB

d2c_workspace.conf.yaml

Settings for the execution of direct2cad tasks on a worker node

Properties

• Python Dataclass: simple_uam.util.config.d2c_workspace_config.D2CWorkspaceConfig (API)
• Purview Category: Per-Deployment (mostly)
• Config File Name: d2c_workspace.conf.yaml
• Interpolation Key: d2c_workspace

Contents

Default d2c_workspace.conf.yaml

### d2c_workspace.conf.yaml ###

workspace_subdir_pattern: workspace_{}
reference_subdir: reference_workspace
assets_subdir: assets
locks_subdir: workspace_locks
results_dir: ${workspaces_dir}/results
results:
  max_count: -1
  min_staletime: 3600
  metadata_file: metadata.json
  log_file: log.json
workspaces_dir: ${path:work_directory}/d2c_workspaces
cache_dir: ${path:cache_directory}/d2c_workspaces
max_workspaces: 1
exclude:
- .git
result_exclude:
- .git
- workingdir/*.prt

Key Fields

• results_dir: The location to place analysis result zip files. Should be set to point to the results storage. If workers are different then use a symlink to the intended directory.
• results: Options on how to generate and store results.
  • max_count: Number of results to allow in results_dir. The oldest zip files will be deleted until the count is low enough. -1 disables pruning entirely.
  • min_staletime: Time after last access to prevent the deletion of a record in seconds. Results that aren't stale enough will not be deleted even if there are more than max_count.

broker.conf.yaml

Settings that define how to connect to a message broker and backend. It is used by both worker nodes and clients, and is very important to get correct in order for a SimpleUAM deployment to function.

Thankfully, this is set up such that all the workers and clients should be able to share a single file unless there's network or DNS weirdness.

Properties

• Python Dataclass: simple_uam.util.config.broker_config.BrokerConfig (API)
• Purview Category: Per-Deployment
• Config File Name: broker.conf.yaml
• Interpolation Key: broker

Contents

Default broker.conf.yaml

### broker.conf.yaml ###

protocol: amqp
host: 127.0.0.1
port: 5672
db: ''
url: ${.protocol}://${.host}:${.port}${.db}
backend:
  enabled: false
  protocol: redis
  host: 127.0.0.1
  port: 6379
  db: /0
  url: ${.protocol}://${.host}:${.port}${.db}

Key Fields

• protocol: The protocol to use for the broker. Generally either amqp or amqps for RabbitMQ and redis for Redis.
• host: The IP or URL of the message broker.
• port: The port to connect to on the message broker.
• db: The Redis db if needed, this should include a beginning / if provided at all. (e.g. use /0 instead of just 0)

• url: The full url of the message broker including protocol host, port, and the like.

  If provided then protocol, host, port, and db are ignored. Make sure that url contains all necessary information.

• backend: Settings for an optional Message Backend (used for completion notifications)

  • enabled: Set to true in order to enable the backend.
  • protocol: Protocol to use for the backend, the only supported option is redis.
  • host: The IP or URL of the message backend.
  • port: The port to connect to on the message backend.
  • db: The Redis db this should include a beginning / if provided at all. (e.g. use /0 instead of just 0)

  • url: The full url of the messag backend including protocol host, port, and the like.

    If provided then backend.protocol, backend.host, backend.port, and backend.db are ignored. Make sure that backend.url contains all necessary information.

d2c_worker.conf.yaml

Settings for running the process that listens to the message broker for new tasks, executes them via the workspace code, and potentially returns confirmations or other information to a backend.

Properties

• Python Dataclass: simple_uam.util.config.d2c_worker_config.D2CWorkerConfig (API)
• Purview Category: Per-Machine
• Config File Name: d2c_worker.conf.yaml
• Interpolation Key: d2c_worker

Contents

Default d2c_worker.conf.yaml

### d2c_worker.conf.yaml ###

max_processes: ${d2c_workspace:max_workspaces}
max_threads: 1
shutdown_timeout: 600000
skip_logging: false
service:
  priority: NORMAL
  exit_action: Restart
  restart_throttle: 5000
  restart_delay: 1000
  redirect_io: false
  stdout_file: ${path:log_directory}/d2c_worker/stdout.log
  stderr_file: ${path:log_directory}/d2c_worker/stderr.log
  rotate_io: true
  auto_start: false
  console: true
  interactive: false

Key Fields

• max_processes: The number of simultaneously running direct2cad processes. Due to limitations with Creoson the only supported number is 1.
• max_threads: The number of threads, per-process, on which to run direct2cad tasks. The only currently supported value is 1.
• shutdown_timeout: How long to wait for a worker to shutdown in milliseconds.
• skip_logging: Do we preserve the logs that dramatiq produces? Note that this does not effect the usual structlog based logging, just the logging from the library that manages a worker process.
• service: Settings for running the direct2cad worker as a service. These correspond closely to the corresponding settings in NSSM.