SA02 - Data processing services used by a Participant
To help understand the content of this document, readers should familiarize themselves with the key definitions and actors.
Overview
This supporting activity serves to present a set of data processing services that enable both Data Providers and Data Consumers to prepare, analyse and explore their datasets. In the context of Simpl-Open data processing encompasses data preparation, data analytics and data visualisation.
Data preparation is based on ETL (Extract, Transform, Load) processing, a structured approach for extracting data from multiple sources, transforming it into a consistent and usable format, and making it available for other uses, for example for publication in the catalogue or for further analysis. For less technical users, Simpl-Open provides minimal services that support common preparation tasks and also serve as practical examples that Participants can reuse or extend. For more technical users, Simpl-Open also offers libraries to implement their own custom services and workflow, allowing them to manage and transform data according to their specific needs.
In addition to data preparation, this supporting activity also covers data analytics and data visualisation. These capabilities allow Participants to analyse datasets, identify patterns and trends, and explore results through visual representations, supporting both operational and strategic decision-making. Analytics and visualisation functionalities are not tightly coupled to predefined workflows and can be leveraged independently.
These services provided by Simpl-Open will be made available in the Simpl-Open agent. Some services will be available as part of the deployment of Simpl-Open, others can be added by the participants.
The ETL offered services are:
1. Data standard-format processing
These services support Participants (either Provider and Consumer) in processing different types of dataset in standard formats.
2. Data curation services
These services support Participants (either Provider and Consumer) in data curation. This includes cleaning, normalising, and filtering datasets based on specific conditions to improve their quality, consistency, and suitability for further analysis and use.
3. Data aggregation services
These services support Participants (either Provider and Consumer) in data aggregation, This includes grouping records by one or more keys and summarising values using metrics such as sum, average, count, or minimum/maximum, to consolidate information and produce specific data views that support analysis and reporting.
4. Pseudonymisation and anonymisation services
These services support Participants (either Provider and Consumer) in protecting personal data, as well as business-critical information, confidential datasets, and other sensitive records, by enabling the application of a range of data protection techniques.
Pseudonymisation allows for reversible transformations, making it possible to re-identify data subjects when justified, through the use of securely managed auxiliary information. Anonymisation, on the other hand, ensures that data is irreversibly transformed in such a way that no direct or indirect link to the original data subjects remains.
5. Data quality check services
These services support Participants (both Provider and Consumer) in performing automated data quality checks. This includes executing automatic validations based on user-defined metrics and quality gates, ensuring that datasets meet the required quality standards before being used or shared.
Simpl-Open also shall provide libraries to enable the creation of custom services and the definition of complete ETL workflows, covering all stages of the process, from data extraction to data loading including data harvesting, data cleansing, normalisation, aggregation, filtering, reshaping, parsing and data quality checks.
This approach ensures maximum flexibility for both less technical and more advanced users, allowing Participants to define their own workflows, which can be executed through the data orchestration functionality provided by Simpl-Open (SA01).
Actors
The following actors are involved:
- Participant (Provider and Consumer)
Assumptions
The following assumptions are made:
- The pseudonymisation and anonymisation services allow users to balance data protection with data utility by selecting and applying the most appropriate techniques based on their operational needs. However, users retain full responsibility for making informed decisions on the data to be processed and the techniques to be used, validating the effectiveness of the chosen methods, and managing any associated risks, as the services provide support but does not replace expert judgment.
Prerequisites
The following prerequisites must be fulfilled:
- Participant onboarded: The Participant should have successfully completed the onboarding business process (Business Process 3A);
- End user authenticated & authorised: The End-User is authenticated and has the appropriate role and permissions to perform the steps in the process (Business Process 3B).
Details
The following shows the details business process diagram and gives the step descriptions.
Trigger data processing services
The Participant (either Provider or Consumer) decides to use one of the available data processing services offered by Simpl-Open based on their operational needs.
SA02.01 Data standard-format processing
The Participant (either Provider or Consumer), after evaluating the dataset and the context of use, decides that it is necessary to prepare it for future uses or analysis. The dataset is then ingested into their process and, after being appropriately transformed, is loaded into the appropriate target systems.
SA02.02 Data harvesting
The Participant (either Provider or Consumer), during the extraction phase of their process, decides to perform data harvesting, intended as web scraping, enabling the automated extraction of information from HTML pages by fetching the pages, parsing them, and extracting the relevant content.
SA02.03 Data curation
The Participant (either Provider or Consumer) decides to add a data curation phase to their process in order to improve dataset quality and consistency through data cleansing (such as removing duplicated records, filling missing values, standardising categorical values, and correcting typos), data normalisation (including the standardisation of date and time formats, numeric values, and geographic data), and data transformation activities (such as filtering relevant records, reshaping datasets, and parsing unstructured or semi-structured fields to support further analysis and use).
SA02.04 Data aggregation
The Participant (either Provider or Consumer) decides to add a data aggregation phase to their process in order to consolidate information and produce specific data views by grouping records based on one or more keys or summarising values using aggregation metrics such as sum, average, count, and minimum or maximum, thereby supporting analysis and reporting.
SA02.05 Data pseudonymisation
The Participant (either Provider or Consumer), after evaluating the dataset and the context of use, decides to apply a pseudonymisation technique on the dataset. This technique can either be configured by the Participant at execution time or a configuration from a previous workflow execution can be reused in a subsequent execution.
SA02.06 Data anonymisation
The Participant (either Provider or Consumer), after evaluating the dataset and the context of use, decides to apply an anonymisation technique on the dataset. This technique can either be configured by the Participant at execution time or a configuration from a previous workflow execution can be reused in a subsequent execution.
SA02.07 Data quality checks
The Participant (either Provider or Consumer) also decides to add a data quality checking phase to their process in order to ensure that only high-quality and reliable data are made available for further use and analysis, by performing automatic validations based on defined metrics and quality gates.
SA02.08 Data analytics
The Participant (either Provider or Consumer), after preparing the dataset, decides to perform data analytics activities in order to analyse the data, identify patterns, trends, or relationships, and generate insights that support operational or strategic decision-making.
SA02.09 Data visualisation
The Participant (either Provider or Consumer) decides to use data visualisation capabilities to explore and present data and analytical results through visual representations, facilitating interpretation, communication, and decision-making.
SA02.10 Data profiling
The Participant (either Provider or Consumer) decides to use data profiling to collect descriptive statistics on the data, collect data types, tag data with keywords and other steps. It helps in analysing data, implementing a data governance strategy and determining data quality.
Outcomes:
- The Participant (either Provider or Consumer) used the data standard-format processing services.
- The Participant (either Provider or Consumer) used the data harvesting services.
- The Participant (either Provider or Consumer) used the data curation services.
- The Participant (either Provider or Consumer) used the data aggregation services.
- The Participant (either Provider or Consumer) used the data pseudonymisation services.
- The Participant (either Provider or Consumer) used the data anonymisation services.
- The Participant (either Provider or Consumer) used the data quality check services.
- The Participant (either Provider or Consumer) performed data analytics activities.
- The Participant (either Provider or Consumer) used data visualisation capabilities.
- The Participant (either Provider or Consumer) used data profiling capabilities.
| Supporting Activity | Status: Proposed |
High Level Requirements
2.1 - Participant - pseudonymises a dataset
Simpl shall provide a service that allows a Participant ...2.2 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.3 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.4 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.5 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.6 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.7 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.8 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.9 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...2.10 - Participant - anonymises a dataset
Simpl shall provide a service that allows a Participant ...
Please log in or sign up to comment.