SA01 - Data orchestration used by a Participant
To help understand the content of this document, readers should familiarize themselves with the key definitions and actors.
Overview
This supporting activity enables Providers and Consumers to design, execute, and monitor traceable and repeatable data processing workflows, facilitating consistent and auditable data transformation activities within the data space. These workflows orchestrate a sequence of services, allowing Participants to automate complex data operations in a structured and governed way. In the case of the Provider, the data orchestration capability can also be used for data preparation workflows that, for example, can include anonymisation, pseudonymisation, or data cleansing steps. Once triggered, the workflow executes each defined step in order, processing selected data sets and delivering outputs to user-defined destinations according to the configured logic. The selection and configuration of processing services to be included in each workflow remain the responsibility of the individual Participant, based on their specific use case and intended outcomes. To support this responsibility, the constraints and limitations of each service will be clearly communicated, ensuring that End-Users understand the scope, assumptions, and risks associated with incorrect usage. These services, outside the scope of this supporting activity, may consist of components or applications available through the application catalogue.
The data orchestration capability in Simpl-Open is based on a two-layered model:
- Service: A service represents a single, self-contained unit of work. It performs a specific function such as data transformation, or communication with an external service. Services are designed to be modular, reusable, and independently testable components that can be orchestrated as part of a larger process.
- Workflow: A workflow is a structured sequence of services that are executed in a defined order to achieve a specific business or data processing goal. It defines the orchestration logic, including service dependencies, execution conditions, and data flow. Workflows enable the automation of complex processes by coordinating multiple services into a cohesive pipeline. A workflow receives inputs sources that will be mainly data sets. The execution of the workflow is initiated by a triggering mechanism.
- During consumption of target data: This means that after the consumption requested by the Consumer, the workflow will be triggered by passing the data set that is present in the source at the moment of the consumption request. This means explicitly that the workflow will use the dataset as it exists in the source at the exact time of the request. The triggered workflow will be executed and, after the processing is over and the new data set is stored on the Provider side, it will be shared. It is important to note that the consumed dataset is constrained by the Provider’s update frequency and freshness policies:
- It may not include changes made to the source after the request was issued
- It may reflect only the latest snapshot made available according to the Provider’s refresh cycle
- Continuous updates are not guaranteed
- Source data change: This means that the Provider can set a trigger that creates an event when a data set is modified, and this will trigger the workflow execution. The triggered workflow will be executed, and after the processing is over, the new data set is stored on the Provider side.
- Time schedule: The workflow runs on a predefined schedule, for instance, once per day at 3.am.
- Manual trigger: The workflow is triggered manually from the UI.
- During consumption of target data: This means that after the consumption requested by the Consumer, the workflow will be triggered by passing the data set that is present in the source at the moment of the consumption request. This means explicitly that the workflow will use the dataset as it exists in the source at the exact time of the request. The triggered workflow will be executed and, after the processing is over and the new data set is stored on the Provider side, it will be shared. It is important to note that the consumed dataset is constrained by the Provider’s update frequency and freshness policies:
The scheduling possibilities of a workflow in the orchestration platform that are depicted in the diagram are:
This supporting activity covers the following main steps:
- Define a workflow: Initial definition of the workflow and associated services.
- Update a workflow: Update of existing workflows.
- Delete a workflow: Deletion of existing workflows.
- Execute a workflow: Execution of the defined workflow.
- Monitor a workflow: Verification of execution status, access to real-time or historical logs, and identification of errors or bottlenecks.
Actors
The actor involved in this business process is referred to as the Participant, and can corresponds to:
- Consumer
- Provider
Assumptions
The following assumptions are made:
- Simpl-Open will not inspect or validate the internal business logic configured by the Participants within their workflows.
- The services that a Participant can use are either built-in Simpl-Open, defined by the Participant itself or selected in the application catalogue
Prerequisites
The following prerequisites must be fulfilled:
- Participant onboarded: The Participant should have successfully completed the onboarding business process (Business Process 3A) before they can trigger the orchestration.
- End-User authentication and authorisation: The End-User must be authenticated and possess the necessary roles and permissions to execute the process steps (Business Process 3B).
Details
The following shows the detailed supporting activity diagram and gives the step descriptions.
Trigger Data Orchestration
The Participant initiates the orchestration process. It is triggered by a End-User who intends to define a new workflow, manage, execute or monitor an existing one. This step acts as the entry point to the orchestration lifecycle.
SA01.01 Create a workflow
The Participant initiates the creation of a new workflow. They define the logical sequence of services, select the input sources, configure the triggering mechanism, and specify the expected outputs. Once all parameters are set, the workflow is available for execution.
SA01.02 Update a workflow
The Participant selects an existing workflow and modifies its configuration to meet new requirements. This may include changing the service sequence, updating data sources or outputs, adjusting the logic, or modifying the trigger. After applying the changes, the updated workflow replaces the previous version. A versioning system ensures that each modification is tracked, allowing rollback to previous states and maintaining an auditable history of changes.
SA01.03 Delete a workflow
The Participant identifies a workflow that is no longer needed and deletes it from the system. If the workflow has no active dependencies, it is deleted: the workflow will no longer be available for future execution, and its definition will be removed, while past logs and history are retained for auditability and traceability. If there are any dependencies, deletion is prevented, and the Participant is notified of the existing dependencies.
SA01.04 Execute a workflow
The Participant triggers the execution of a workflow, either manually or as per trigger in the workflow. The workflow runs according to its defined logic. If any of the services included in the workflow need extra information (such as choosing a file or entering a parameter), the Participant may be asked to provide it before the execution begins. Execution results for each step are logged and made available for monitoring. If a workflow is deleted but still referenced (e.g., triggered upon dataset consumption), an error will be thrown because the workflow cannot be found. In such cases, no data will be disclosed and the transfer will not occur, but an error will be returned to the consumer and/or provider. This ensures that the middleware remains robust.
SA01.05 Monitor a workflow execution
The Participant can observe the execution status in real time or access historical records of past runs. The monitoring functionality includes visual indicators of the current status (e.g., running, completed, failed), detailed logs for each step or service. In the event of failures or bottlenecks, the End-User can inspect log traces to identify the root cause and take corrective action.
Outcomes
- Workflow created, updated, or deleted: The Participant has completed the operation of creation, update, or deletion of a workflow.
- Workflow executed: The Participant executed the workflow. This can lead to a successful execution or a failing one.
- Workflow monitored: The Participant has monitored the workflow by accessing the relevant logs and status information.
| Supporting Activity | Status: Proposed |
High Level Requirements
1.1 - Participant consults their own data orchestration workflows
Simpl-Open shall allow a Participant to consult their own data ...1.2 - Participant creates a new data orchestration workflow
Simpl-Open shall allow a Participant to create a new data ...1.3 - Participant creates a new version of a data orchestration workflow
Simpl-Open shall allow a Participant to create a new version of ...1.4 - Participant deletes a data orchestration workflow
Simpl-Open shall allow a Participant to delete an existing ...1.5 - Participant executes a data orchestration workflow
Simpl shall provide capabilities to allow the Participant to start ...1.6 - Participant monitors and accesses data orchestration workflow execution logs
Simpl shall provide capabilities that enable the Participant ...
Please log in or sign up to comment.