Platform event monitoring

Requirements

• active Router2 application with EximeeEvents

  • application delivered with the Eximee platform since version 2.3167.0
• active WebformsRest application

  • application delivered with the Eximee platform
• active MonitoringLogProcessor application

  • application delivered with the Eximee platform 2.3019.0
• active database storing events

  • PostgreSQL version >=9.5 is required for the application to run
• active Kafka instance

  • we recommend version >= 3.4
• active Grafana instance min 8.X.X

  • a sample monitoring dashboard is delivered with the platform in the grafana.json file

Enabling the functionality in the platform is controlled by adding the value "monitoring" to the COMPOSE_PROFILES variable.

Operating assumptions

The Webforms application generates events related to the state of applications.

Different implementations of event logging can be active at the same time. This depends on the configuration and deployment. EximeeEventsConnector (one possible implementation) collects events and sends them to EximeeEvents.

EximeeEvents saves them to the appropriate Kafka topic.

MonitoringEventProcessor reads events from Kafka, verifies them, and saves them to the database (PostgreSQL) in a format that can be read by Grafana.

Events in the database can be read by external applications for visualization or analysis. A Grafana configuration for event visualization will be delivered with the platform.

Figure 1. Architecture of the event monitoring solution

Router2 application with EximeeEvents

Purpose

EximeeEvents is a plugin for the Router2 application. It provides an endpoint that accepts events, which are then saved to the appropriate Kafka topic.

The topic and message group sent to Kafka must be consistent with the MonitoringLogProcessor configuration.

Configuration

Configuration and more information here [Process event registration]

MonitoringLogProcessor application

Purpose

The application's task is to populate the database (currently only PostgreSQL) with platform events.

The application reads events from the source (Kafka) and verifies whether it is a system event (originating from the platform). Then the system events are adapted for Grafana and saved to the database.

Configuration

Key	Requirement	Example	Description
Database - target location for storing events
MONITORING_LOG_PROCESSOR_DATABASE_DIALECT	YES	org.hibernate.dialect.PostgreSQL95Dialect	Database dialect intended for storing events.
MONITORING_LOG_PROCESSOR_DATABASE_URL	YES	jdbc:postgresql://postgresql01.host:5432/monitoring_events	Database connection details.
MONITORING_LOG_PROCESSOR_DATABASE_USERNAME	YES	db_user	User with permission to write events.
MONITORING_LOG_PROCESSOR_DATABASE_PASSWORD	YES	db_pass	Password of the user accessing the database.

Kafka

Data source	Requirement
MONITORING_LOG_PROCESSOR_KAFKA_URL	YES	http://kafka.host:9092	Kafka connection details
MONITORING_LOG_PROCESSOR_KAFKA_TOPIC	YES	all	The topic to which event messages will be sent
MONITORING_LOG_PROCESSOR_KAFKA_GROUP	YES	eximee-events	Eximee event group

Stored events

Events must be of system origin (origin = SYSTEM).

Event types are converted to business types. A type that has no corresponding business event is given an empty value. Information about the event type mapping is below:

System event (eximee)	Business event	Description
FORM_NEW_STARTED	FORM_ENTRY	Entering the application
FORM_NEXT_PAGE FORM_PREVIOUS_PAGE	PAGE_SAVE	Saving the state of the current page - moving to the next or previous page
FORM_CLIENT_PAGE_CHANGED	PAGE_ENTRY	Page change - entering another page.
FORM_SAVE	FORM_SAVE	Submitting the application
FORM_DRAFT_SAVE	FORM_DRAFT_SAVE	Saving a draft version of the application
FORM_DRAFT_RESTORE	FORM_DRAFT_RESTORE	Restoring a draft version of the application

Database structure (PostgreSQL)

Specified database structure with business events.

Schema			monitoring_events
Table			events
Field name	Type	nullable	Description
id	bigserial	FALSE	Consecutive event identifiers
origin	text	FALSE	Event origin (SYSTEM / BUSINESS)
type	text	FALSE	Business event type (the event type table is above)
timestamp	timestamp	FALSE	Event invocation time (date and time)
payload	jsonb	TRUE	Event data in JSON format, if the event has additional data.

Migration scripts

Migration scripts for creating and modifying the table are saved in an archive to be delivered together with the platform.

Webforms application with event logging

Purpose

The event logging mechanism is intended to collect events related to application state changes across the entire platform. Events are sent in batches to Kafka using EximeeEvents.

Configuration

To activate event collection, the configuration should be set accordingly webforms.xml. Below is a configuration snippet responsible for the active event collection flow.

The connector for eximeeEvents is delivered in the eximee-events library

<?xml version="1.0" encoding="UTF-8" ?>
<webforms>
   ....
   <server>
      ...
      <features>
         ...
         <eventLog>
            <enabled>true</enabled>
            <coreExecutorPoolSize>5</coreExecutorPoolSize>
            <maxExecutorPoolSize>10</maxExecutorPoolSize>
            <eximeeEventsProcessor>
               <enabled>true</enabled>
               <url>http://localhost:9002</url>
            </eximeeEventsProcessor>
         </eventLog>
      </features>
   </server>
</webforms>

And a table presenting the installation configuration.

Installation key	Configuration key	Example value	Description
Active event collection mechanism
	server.features.eventLog.enabled	true	Event collection mechanism activity
	server.features.eventLog.coreExecutorPoolSize	5	Number of threads used for event processing
	server.features.eventLog.maxExecutorPoolSize	10	Maximum number of threads used for event processing
Active event logging mechanism
	server.features.eventLog.eximeeEventsProcessor.enabled	true	Activity of the EximeeEvents mechanism
	server.features.eventLog.eximeeEventsProcessor.url	http://localhost:9002	Endpoint for the EximeeEvents application for writing events to Kafka