12B – Single node logging & monitoring
Description
To help understand the content of this document, readers should familiarize themselves with the key definitions and actors and the business process introduction containing the diagram legend.
This workflow depicts the collecting, standardising, and storing of logs and metrics, monitoring of the Simpl-Agent, and the visualisation and reporting of data. The logs include gathering metrics like performance data, error logs, and usage statistics. Furthermore, visualisations and reports provide insights into system performance and user activities. This workflow is applicable to all the Participants, namely the Consumers, Providers and the Governance Authority.
The actor involved in this process is the: Participant.
Prerequisites:
The prerequisites for this workflow are outlined below. These prerequisites must be met to enable the process to occur:
- Participant Onboarded: Before the Participant can log & monitor its Simpl-Agent, they should have successfully completed the onboarding business process (Business Process – 3A).
Workflow Diagram & Steps
This chapter presents a diagram visualising the workflow, labelled with specific steps. Each step is further detailed in the accompanying 'Step Description'.
Step Description:
Below is a description of the steps involved in this workflow. Each step outlines the specific actions and decisions required to successfully complete the workflow:
- Log consumption of a resource (infra/data/app): During the consumption of a (infrastructure/data/application) resource, Simpl-Open agent generates technical logs that will be used for various reasons (billing, audit, policy enforcement, regulation compliance, etc.);
- Log usage of Simpl-Open agent components: While they are running, the components of Simpl-Open agent generate both technical logs and business logs that will be used for the purposes of audit and troubleshooting;
- Log business actions: For significant events or actions (related to steps within a business process or other functional use cases), Simpl-Open agent generates business logs that will be used mostly for the purpose of audit;
- Collect infrastructure metrics and service health: On top of the logs (which are push-based mechanism), Simpl-Open agent also collects infrastructure metrics and service health (pull-based mechanism) from its components. Infrastructure metrics are also collected from the resources being consumed;
- Collect technical logs: Simpl-Open agent gathers the technical logs generated in steps 1 and 2;
- Collect business logs: Simpl-Open agent gathers the business logs generated in steps 2 and 3;
- Standardise log format: Convert all collected logs into a consistent format to facilitate analysis and integration with other monitoring tools;
- Persist logs & metrics: Persist the collected logs and metrics in a durable, accessible location for future analysis and access;
- Monitor consumption of a resource (infra/data/app): Monitor the consumption of resources through a dashboard to enforce policies and regulations compliance;
- Monitor usage and health of Simpl-Open agent: Monitor the Simpl-Open agent usage and health through a dashboard to ensure it is functioning correctly, tracking the usage, its performance, and any errors or warnings;
- Monitor business actions: Monitor the business actions (logged in step 3) through a dashboard;
- Generate alerts: Alerts get generated when health checks fail or predefined thresholds are exceeded;
- Explore logs: Perform queries and provide aggregated views (e.g., table, diagrams, etc.) of the collected logs to identify patterns, trends, and anomalies (example of who is doing this: IT admin of the Simpl-Open agent);
- Generate reports: Compile the logged data into reports to provide insights (e.g., system performance, user activities, and issues encountered);
- Visualise reports: Access the reports via the interface and UI, and use visualisations from the reports to make the data more accessible;
- Export reports: Export of generated reports in various formats for sharing and further analysis.
Use cases and types of logs are described in details in the Logging, Monitoring & Reporting section of the Architecture Document:
Types of logs - Reference model
The following table identifies the different types of logs that can be generated by an IT system together with their definition/description:
| Grouping | Type of logs | Description |
|---|---|---|
| Business logs | Business logs | Record significant events or actions (related to steps within a business process or other functional use cases) that occur within a system, typically used for security, audit, and troubleshooting purposes. |
| Technical logs | Application logs | Record events and activities generated by an application during its runtime, typically used for troubleshooting, monitoring performance, and auditing activities within the application. |
| Database logs | Record events and activities generated by a database (queries, transactions, schema changes), typically used for troubleshooting, ensuring data integrity and auditing access. | |
| System logs | Record events and activities generated by the operating system (OS) and system-level processes. These logs provide valuable information for monitoring system health, diagnosing issues, and ensuring security. System logs can include: low-level system events (kernel event, hardware error), system-level events (service startups/shutdowns/failure), authentication and authorisation events (login attempts, privilege escalation). | |
| Network logs | Record events and activities related to network traffic, devices, and communications within a network. These logs are essential for monitoring network health, diagnosing issues, and ensuring security. Network logs can include: firewall logs (allowed/denied connections, intrusion detection alerts, security policy violations), router and switch logs (device startups, interface status changes, routing protocol updates), DNS logs (queries/responses, cache activity, DNS server configuration changes and errors), proxy logs (user access, URL requests, content filtering, bandwidth usage) and network traffic logs (packet-level data, including source and destination IP addresses, port numbers, protocols, packet payloads). | |
| Security logs | Security logs are not a distinct type of log, they are a subset of all the other logs listed above, which allow to detect and respond to security incidents effectively. Ex: Intrusion detection alerts, security policy violations, anti-virus scans. | |
| Infrastructure metrics | Infrastructure metrics | A metric is a piece of data that has a name, optional labels, and a value. It is not a log per-se, as they need to be retrieved by periodically scrapping an endpoint of the host system (pull instead of push paradigm). Once retrieved, the information is then persisted as a log. |
| Health check | Health check | A health check is a procedure that helps to determine if a component is functioning correctly or not. Just like infrastructure metrics, health checks are not logs per-se, it is an API exposed by each component to return a simple status on the health of the component, which is queried periodically. |
Submission of a contract offer by a provider to a consumer
For the sake of simplicity, application, database, system, network and security logs are grouped under the more generic term of Technical Logs.
Use Cases and Types of Logs
| Use case | Type of logs required | Type of metrics required | Description |
|---|---|---|---|
| Log and monitor business actions, mostly for audit purposes. | Business logs | A business log in this case represents a specific step in a business process that is relevant/meaningful to be tracked. E.g. Submission of an onboarding request. | |
| Log and monitor consumption of a resource (infra/data/app) for various reasons (billing, audit, policy enforcement, regulations compliance ...). | Infrastructure metrics | Depending on the type of data or infrastructure resource that is being consumed, different metrics can be relevant: CPU, RAM, I/O, transfer speed, ... | |
| Technical logs | For application usage and for some data usage cases, application and database logs will give information on what is being done with the data/application. | ||
| Log and monitor the usage of a Simpl-Open agent (of its components) for the purposes of audit and troubleshooting. | Technical logs | All types of technical logs are relevant for troubleshooting purposes and some may also be relevant for audit. | |
| Business logs | A business log is generated for each incoming and outgoing operation at the boundaries of the agent (communication towards Tier 1 or Tier 2 users). | ||
| Infrastructure metrics | Infrastructure metrics generated by the deployed components of the agent (CPU, RAM, Disk, ...). | ||
| Monitor the health of the Simpl-Open agent. | Health check | Health is not logged but only monitored (the monitoring queries each technical component in real time to get its health status). |
| L0 - Business Process | Status: Proposed |
Associated L1s - High Level Requirements
To be determined
Please log in or sign up to comment.