5. Automated Testing

The UFS Weather Model repository on GitHub employs two types of automated testing. One is the CI/CD on cloud and the other is the AutoRT on NOAA R&D platforms. Both are application level tests, and utilize the regression testing framework discussed in Section 3.4.1.

5.1. CI/CD

The following summarizes the CI/CD used in the UFS Weather Model:

  • GitHub Actions (GHA), a GitHub-hosted continuous integration service is used.

  • Build jobs are done on GHA-provided virtual machines.

  • Test jobs are performed on Amazon cloud using a number of EC2 instances.

  • Builds and tests are carried out using a Docker container.

  • Docker container has prerequisite libraries installed via the hpc-stack.

  • Input data needed to run tests are stored as a separate Docker container.

When a developer makes a pull request (PR) to the UFS Weather Model repository, and a code manager subsequently adds the run-ci label, the CI/CD workflow is triggerd:

  1. A check is performed to make sure the UFS Weather Model and its first level subcomponents are up to date with the top of develop branch.

  2. If the check is successful, build jobs are started on GHA-provided virtual machines by downloading the hpc-stack Docker container stored in Docker Hub.

  3. Once all build jobs are successful, the created executable files are stored as artifacts in GHA.

  4. A number of AWS EC2 instances are started.

  5. Test jobs are started on Amazon cloud by downloading the hpc-stack Docker container, the executable file from the build job, and the input-data Docker container.

  6. When all tests are finished, EC2 instances are stopped. Test results are reported on GitHub.

The GHA-related yaml scripts are located in the .github/workflows/ directory. build_test.yml is the main workflow file, and aux.yml is an auxiliary file responsible for checking the up-to-dateness of the PR branch, and starting and stopping the EC2 instances. Other CI-related scrips are located in the tests/ci/ directory. ci.sh is the main script that invokes Docker build and run. Dockerfile is used to build UFS Weather Model. Other shell and python scripts help with various tasks such as checking the up-to-dateness of the PR branch (repo_check.sh), checking the status of EC2 instances (check_status.py), and configuring the test cases to carry out in the CI/CD workflow (setup.py and ci.test).

5.2. Auto RT

The Automated Regression Testing (AutoRT) system:

  • Automates the process of regression testing on NOAA HPC platforms.

  • Written in python.

  • Contains the following files:

Table 5.1 Files for Automated Regression Testing (AutoRT) system

File Name

Description

start_rt_auto.sh

Verifies HPC name, sets the python paths

rt_auto.py

Python interface between the HPC and the github API

jobs/bl.py

Functions for the baseline job

jobs/rt.py

Functions for the regression test job

5.2.1. AutoRT Workflow

  • Cron-job on supported HPC systems runs start_rt_auto.sh bash script every 15 minutes.

    • This script verifies the HPC name, and sets the python paths. Runs rt_auto.py.

  • rt_auto.py: Uses the Github API (Through pyGitHub)

    • Checks the pull requests to ufs-community/ufs-weather-model for labels specific to the HPC name. If no match to HPC name, exits. (i.e. hera-intel-RT or cheyenne-gnu-BL)

    • If the HPC name matches the label in ufs-weather-model pull request, the label provides the HPC with the compiler and job to run on the machine.

      • For example the label gaea-intel-BL will be recognized by the HPC machine ‘Gaea’, set the RT_COMPILER variable to ‘intel’ and run the baseline creation script (bl.py).

    • Creates a Job class that contains all information from the machine that the job will need to run. That is sent into the jobs/rt[bl].py script.

  • rt.py: Sets directories for storage, gets repo information, runs RT, post processes.

def run(job_obj):
    logger = logging.getLogger('RT/RUN')
    workdir = set_directories(job_obj)
    branch, pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
    run_regression_test(job_obj, pr_repo_loc)
    post_process(job_obj, pr_repo_loc, repo_dir_str, branch)

  • bl.py: (similar to rt.py) Adds functionality to create baselines before running regression testing.

  def run(job_obj):
      logger = logging.getLogger('BL/RUN')
      workdir, rtbldir, blstore = set_directories(job_obj)
      pr_repo_loc, repo_dir_str = clone_pr_repo(job_obj, workdir)
      bldate = get_bl_date(job_obj, pr_repo_loc)
      bldir = f'{blstore}/develop-{bldate}/{job_obj.compiler.upper()}'
      bldirbool = check_for_bl_dir(bldir, job_obj)
      run_regression_test(job_obj, pr_repo_loc)
      post_process(job_obj, pr_repo_loc, repo_dir_str, rtbldir, bldir)