FarEarth uses powerful Workflows to ensure Orders are automatically assigned and processed with the correct Executors. We will explore the relationships between Workflows and Orders, steps and Jobs, and how failures of individual Jobs affect the status of an Order.
The Workflow
Workflows describe the process that ingested data must follow in the system for processing. A Workflow is like a cooking recipe. A cooking recipe specifies basic requirements, such as “mix until combined,” but it is not opinionated about the details of how to achieve the goal.
A FarEarth Workflow describes the sequence of steps for the system to follow. When data is ingested for processing, FarEarth selects a suitable Workflow and creates an Order.
The Order
In FarEarth, an Order is a specific implementation of a Workflow. To use the cooking recipe metaphor, an Order would be the same as placing an order at a restaurant, where you ask the chef to prepare a specific recipe.
The graphic below illustrates the relationship between Workflows and Orders. For each step in the Workflow, FarEarth creates one or more Jobs. In the example, the output of one step, such as the RAW to Level 0 processing step, will result in multiple Jobs that can be executed concurrently. For example, FarEarth will use this when it splits a long image interval into smaller intervals with some overlap and runs multiple Jobs simultaneously.
The Executor
FarEarth Executors are the lowest-level components responsible for completing a single Job. FarEarth will automatically identify an Executor and determine where to run it to complete the task.
An Executor takes input and produces one or more outputs. For example, the “generic-l1c-to-l2a” executor in our first graphic produces Level 2 surface reflectance products from Level 1 top-of-atmosphere data.
When compiling the Workflow, you do not have to specify the exact Executor to use. You can select what you would like to accomplish and let FarEarth decide which is the most suitable Executor for the Job. For example, you may specify that the Level 1C output must be processed into a Level 2A product and leave it to FarEarth to find the best match.
You can override the default behaviour of an Executor from the Workflow. Just as the recipe might tell the chef to mix eggs using a cold metal spoon, you can send additional parameters to the Executor, specified in the Workflow.
FarEarth can be configured to run individual Jobs on multiple Executors, potentially located in different processing centres. Fortunately, you do not need to worry about the complexities of getting the data from one Job to the next and tracking the progress. FarEarth will do it for you!
The Archive
The result of a Workflow is a series of archiving steps. The products resulting from intermediate steps can also be stored in an Archive.
It is not necessary to specify the specific Archive where data will be stored. FarEarth will choose the relevant Archive based on rules. For example, all Level 2 products can be stored in an Archive in a different location. Or you can configure it so that data acquired over a specific location, like Europe, is Archived separately.
Order success
If a specific Job within an Order fails, there is a decision to be made about the status of the Order as a whole. Do we fail the whole Order, or do we continue with the remaining Jobs in the Order?
Consider the following Workflow, where Level 1C (L1C) products are processed to NDVI and Level 2A (L2A) products in parallel. The L2A processing is marked as Optional. Let us assume that each of the NDVI and L2A Workflow steps creates two Jobs for this Order.
The graphic below illustrates two scenarios when a Job of the Optional step fails.
On the left: if a Job associated with the Optional step fails, the remaining Jobs of that step are halted, and no data is archived.
On the right: if any of the Jobs associated with the Optional step fail, the remaining Jobs will still be attempted and archived as usual. This is because the optional step has been selected to “continue on fail”.
An optional step means that if that step fails, the Order may still succeed. An optional step that is also marked to “continue on fail” means that the rest of the Jobs associated with that step must still be attempted.
FarEarth can handle anything from a small mission with uncomplicated processing pipelines to multiple constellations with a variety of sensors. Our Workflows are designed to be easy to define, while allowing for complex distributed processing.