NOTICE: This software (or technical data) was produced for the U.S. Government under contract, and is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2023 The MITRE Corporation. All Rights Reserved.

WARNING: For most component developers, these steps are not necessary. Instead, refer to the C++, Python, or Java README for developing a Docker component in your desired language.

WARNING: This guide is a work in progress and may not be completely accurate or comprehensive.

Overview

The following instructions are for setting up an environment for building and running OpenMPF outside of Docker. They serve as a reference for developers who want to develop the Workflow Manager web application itself and perform end-to- end integration testing.

Setup VM

sudo systemctl disable docker.service;
sudo systemctl stop docker.service;
sudo systemctl enable docker.socket;

sudo apt update
sudo apt install docker-compose-plugin

sudo -i -u postgres createuser -P mpf
sudo -i -u postgres createdb -O mpf mpf

mkdir /tmp/opencv-contrib;
wget -O- 'https://github.com/opencv/opencv_contrib/archive/4.5.5.tar.gz' \
    | tar --extract --gzip --directory /tmp/opencv-contrib;
mkdir /tmp/opencv;
cd  /tmp/opencv;
wget -O- 'https://github.com/opencv/opencv/archive/4.5.5.tar.gz' \
    | tar --extract --gzip;
cd opencv-4.5.5;
mkdir build;
cd build;
export OpenBLAS_HOME=/usr/lib/x86_64-linux-gnu/openblas-pthread; \
cmake -DCMAKE_INSTALL_PREFIX:PATH='/opt/opencv-4.5.5' \
    -DWITH_IPP=false \
    -DBUILD_EXAMPLES=false \
    -DBUILD_TESTS=false \
    -DBUILD_PERF_TESTS=false \
    -DWITH_CUBLAS=true \
    -DOPENCV_EXTRA_MODULES_PATH=/tmp/opencv-contrib/opencv_contrib-4.5.5/modules \
    ..;
sudo make --jobs "$(nproc)" install;
sudo ln --symbolic '/opt/opencv-4.5.5/include/opencv4/opencv2' /usr/local/include/opencv2;
sudo sh -c 'echo /opt/opencv-4.5.5/lib > /etc/ld.so.conf.d/mpf.conf'
sudo ldconfig;
sudo rm -rf /tmp/opencv-contrib /tmp/opencv;

mkdir /tmp/activemq-cpp;
cd /tmp/activemq-cpp;
wget -O- https://dlcdn.apache.org/activemq/activemq-cpp/3.9.5/activemq-cpp-library-3.9.5-src.tar.gz \
    | tar --extract --gzip;
cd activemq-cpp-library-3.9.5;
./configure;
sudo make --jobs "$(nproc)" install;
sudo rm -rf /tmp/activemq-cpp;

mkdir /tmp/noto;
cd /tmp/noto;
wget https://noto-website-2.storage.googleapis.com/pkgs/NotoEmoji-unhinted.zip;
unzip NotoEmoji-unhinted.zip;
sudo mkdir --parents /usr/share/fonts/google-noto-emoji;
sudo cp NotoEmoji-Regular.ttf /usr/share/fonts/google-noto-emoji/;
sudo chmod a+r /usr/share/fonts/google-noto-emoji/NotoEmoji-Regular.ttf;
rm -rf /tmp/noto;

mkdir /tmp/pngdefry;
cd /tmp/pngdefry;
wget -O- 'https://github.com/openmpf/pngdefry/archive/v1.2.tar.gz' \
    | tar --extract --gzip;
cd pngdefry-1.2;
sudo gcc pngdefry.c -o /usr/local/bin/pngdefry;
rm -rf /tmp/pngdefry;

wget -O- 'https://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz' \
        | sudo tar --extract --gzip --directory /opt;
sudo ln --symbolic /opt/apache-maven-3.3.3/bin/mvn /usr/local/bin;

mkdir /tmp/libheif;
cd /tmp/libheif;
wget -O- https://github.com/strukturag/libheif/archive/refs/tags/v1.12.0.tar.gz \
    | tar --extract --gzip;
cd libheif-1.12.0;
mkdir build;
cd build;
cmake3 -DCMAKE_INSTALL_PREFIX=/usr -DWITH_EXAMPLES=false ..;
sudo make --jobs "$(nproc)" install;
cd;
sudo rm -rf /tmp/libheif;

wget -O- https://archive.apache.org/dist/activemq/5.17.0/apache-activemq-5.17.0-bin.tar.gz \
    | sudo tar --extract --gzip --directory /opt;
sudo ln --symbolic /opt/apache-activemq-5.17.0 /opt/activemq;
sudo chown -R mpf:mpf /opt/apache-activemq-5.17.0

git clone https://github.com/openmpf/openmpf-projects.git --recursive;
cd openmpf-projects;
git checkout develop;
git submodule foreach git checkout develop;

At this point you may wish to install additional dependencies so that you can build specific OpenMPF components. Refer to the commands in the Dockerfile for each component you're interested in.

Configure Users

To change the default user password settings, modify openmpf-projects/openmpf/trunk/workflow-manager/src/main/resources/properties/user.properties. Note that the default settings are public knowledge, which could be a security risk.

Note that mpf remove-user and mpf add-user commands explained in the Command Line Tools section do not modify the user.properties file. If you remove a user using the mpf remove-user command, the changes will take effect at runtime, but an entry may still exist for that user in the user.properties file. If so, then the user account will be recreated the next time the Workflow Manager is restarted.

Build and Run the OpenMPF Workflow Manager Web Application

cd ~/openmpf-projects/openmpf;
mvn clean install \
    -DskipTests -Dmaven.test.skip=true \
    -DskipITs \
    -Dcomponents.build.components=openmpf-components/cpp/OcvFaceDetection \
    -Dstartup.auto.registration.skip=false;

Look for this log message in the terminal with a time value indicating the Workflow Manager has finished starting:

2022-10-11 12:21:16,447 INFO [main] o.m.m.Application - Started Application in 22.843 seconds (JVM running for 24.661)

After startup, the Workflow Manager will be available at http://localhost:8080/workflow-manager. Browse to this URL using Firefox or Chrome.

If you want to test regular user capabilities, log in as the "mpf" user with the "mpf123" password. Please see the OpenMPF User Guide for more information. Alternatively, if you want to test admin capabilities then log in as "admin" user with the "mpfadm" password. Please see the OpenMPF Admin Guide for more information. When finished using OpenMPF, stop Workflow Manager with ctrl-c and then run mpf stop to stop the other system dependencies.

The preferred method to start and stop services for OpenMPF is with the mpf start and mpf stop commands. For additional information on these commands, please see the Command Line Tools section. These will start and stop the ActiveMQ, PostgreSQL, Redis, Node Manager, and Workflow Manager processes.

Known Issues

o.m.m.m.c.JobController - Failure creating job. supplier.get()

If you see an error message similar to:

2022-02-07 17:17:30,538 ERROR [http-nio-8080-exec-1] o.m.m.m.c.JobController - Failure creating job. supplier.get()
java.lang.NullPointerException: supplier.get()
    at java.util.Objects.requireNonNull(Objects.java:246) ~[?:?]
    at java.util.Objects.requireNonNullElseGet(Objects.java:321) ~[?:?]
    at org.mitre.mpf.wfm.util.PropertiesUtil.getHostName(PropertiesUtil.java:267) ~[classes/:?]
    at org.mitre.mpf.wfm.util.PropertiesUtil.getExportedJobId(PropertiesUtil.java:285) ~[classes/:?]

Open /etc/profile.d/mpf.sh and change export HOSTNAME to export HOSTNAME=$(hostname). Then, restart the VM.

Appendices

Command Line Tools

OpenMPF installs command line tools that can be accessed through a terminal on the development machine. All of the tools take the form of actions: mpf <action> [options ...].

Execute mpf --help for general documentation and mpf <action> --help for documentation about a specific action.

Packaging a Component

In a non-Docker deployment, admin users can register component packages through the web UI. Refer to Component Registration.

Once the descriptor file is complete, as described in Component Descriptor Reference, the next step is to compile your component source code, and finally, create a .tar.gz package containing the descriptor file, component library, and all other necessary files.

The package should contain a top-level directory with a unique name that will not conflict with existing component packages that have already been developed. The top-level directory name should be the same as the componentName.

Within the top-level directory there must be a directory named “descriptor” with the descriptor JSON file in it. The name of the file must be “descriptor.json”.

Example:

//sample-component-1.0.0-tar.gz contents
SampleComponent/
  config/
  descriptor/
    descriptor.json
  lib/

Installing and registering a component

The Component Registration web page, located in the Admin section of the OpenMPF web user interface, can be used to upload and register the component.

Drag and drop the .tar.gz file containing the component onto the dropzone area of that page. The component will automatically be uploaded and registered.

Upon successful registration, the component will be available for deployment onto OpenMPF nodes via the Node Configuration web page and /rest/nodes/config end point.

If the descriptor contains custom actions, tasks, or pipelines, then they will be automatically added to the system upon registration.

NOTE: If the descriptor does not contain custom actions, tasks, or pipelines, then a default action, task, and pipeline will be generated and added to the system.

The default action will use the component’s algorithm with its default property value settings. The default task will use the default action. The default pipeline will use the default task. This will only be generated if the algorithm does not specify any requiresCollection states.

Unregistering a component

A component can be unregistered by using the remove button on the Component Registration page.

During unregistration, all services, algorithms, actions, tasks, and pipelines associated with the component are deleted. Additionally, all actions, tasks, and pipelines that depend on these elements are removed.

Web UI

The following sections will cover some additional functionality permitted to admin users in a non-Docker deployment.

Node Configuration and Status

This page provides a list of all of the services that are configured to run on the OpenMPF cluster:

Node and Process Status Page

Each node shows information about the current status of each service, if it is unlaunchable due to an underlying error, and how many services are running for each node. If a service is unlaunchable, it will be indicated using a red status icon (not shown). Note that services are grouped by component type. Click the chevron ">" to expand a service group to view the individual services.

An admin user can start, stop, or restart them on an individual basis. If a non-admin user views this page, the "Action(s)" column is not displayed. This page also enables an admin user to edit the configuration for all nodes in the OpenMPF cluster. A non-admin user can only view the existing configuration.

An admin user can add a node by using the "Add Node" button and selecting a node in the OpenMPF cluster from the drop-down list. You can also select to add all services at this time. A node and all if its configured services can be removed by clicking the trash can to the right of the node's hostname.

An admin user can add services individually by selecting the node edit button at the bottom of the node. The number of service instances can be increased or decreased by using the drop-down. Click the "Submit" button to save the changes.

When making changes, please be aware of the following:

Component Registration

This page allows an admin user to add and remove non-default components to and from the system:

Component Registration Page

A component package takes the form of a tar.gz file. An admin user can either drag and drop the file onto the "Upload a new component" dropzone area or click the dropzone area to open a file browser and select the file that way. In either case, the component will begin to be uploaded to the system. If the admin user dragged and dropped the file onto the dropzone area then the upload progress will be shown in that area. Once uploaded, the workflow manager will automatically attempt to register the component. Notification messages will appear in the upper right side of the screen to indicate success or failure if an error occurs. The "Current Components" table will display the component status.

Component Registration Page

If for some reason the component package upload succeeded but the component registration failed then the admin user will be able to click the "Register" button again to try to another registration attempt. For example, the admin user may do this after reviewing the workflow manager logs and resolving any issues that prevented the component from successfully registering the first time. One reason may be that a component with the same name already exists on the system. Note that an error will also occur if the top-level directory of the component package, once extracted, already exists in the /opt/mpf/plugins directory on the system.

Once registered, an admin user has the option to remove the component. This will unregister it and completely remove any configured services, as well as the uploaded file and its extracted contents, from the system. Also, the component algorithm as well as any actions, tasks, and pipelines specified in the component's descriptor file will be removed when the component is removed.