Monitoring platform events

Requirements

• active Router2 application with EximeeEvents

  • application delivered with the Eximee platform starting from version 2.3167.0

• active WebformsRest application

  • application delivered with the Eximee platform

• active MonitoringLogProcessor application

  • application delivered with the Eximee platform 2.3019.0

• active database storing events

  • the application requires PostgreSQL version >=9.5 to run

• active Kafka instance

  • we recommend version >= 3.4

• active Grafana instance min 8.X.X

  • a sample monitoring dashboard is provided with the platform in the grafana.json file

Launching the functionality in the platform is controlled by adding the value "monitoring" to the COMPOSE_PROFILES variable.

Operating assumptions

The Webforms application generates events related to the state of applications.

Different implementations of event logging can be active at the same time. This depends on the configuration and deployment. EximeeEventsConnector (one of the possible implementations) collects events and sends them to EximeeEvents.

EximeeEvents saves them to the appropriate Kafka topic.

MonitoringEventProcessor reads events from Kafka, verifies them, and saves them to the database (PostgreSQL) in a form readable by Grafana.

Events in the database can be read by external applications for visualization or analysis. Grafana configuration for visualizing events will be provided together with the platform.

Figure 1. Architecture of the event monitoring solution

Router2 application with EximeeEvents

Purpose

EximeeEvents is a plugin for the Router2 application. It provides an endpoint that receives events, which are then saved to the appropriate Kafka topic.

The topic and message group passed to Kafka must be consistent with the MonitoringLogProcessor configuration.

Configuration

Configuration and more information here [Process event registration]

MonitoringLogProcessor application

Purpose

The application is responsible for filling the database (currently only PostgreSQL) with platform events.

The application reads events from the source (Kafka) and verifies whether it is a system event (coming from the platform). Then the system events are adapted for Grafana and saved to the database.

Configuration

Key	Required	Example	Description
Database - target location for storing events
MONITORING_LOG_PROCESSOR_DATABASE_DIALECT	YES	org.hibernate.dialect.PostgreSQL95Dialect	Database dialect intended for storing events.
MONITORING_LOG_PROCESSOR_DATABASE_URL	YES	jdbc:postgresql://postgresql01.host:5432/monitoring_events	Database address.
MONITORING_LOG_PROCESSOR_DATABASE_USERNAME	YES	db_user	User with permission to write events.
MONITORING_LOG_PROCESSOR_DATABASE_PASSWORD	YES	db_pass	Password for the database access user.

Kafka

Data source	Required
MONITORING_LOG_PROCESSOR_KAFKA_URL	YES	http://kafka.host:9092	Kafka address
MONITORING_LOG_PROCESSOR_KAFKA_TOPIC	YES	all	Topic to which event messages will be sent
MONITORING_LOG_PROCESSOR_KAFKA_GROUP	YES	eximee-events	Eximee event group

Saved events

Events must be of system origin (origin = SYSTEM).

Event types are modified to a business type. A type that does not have an equivalent in the business event receives an empty value. Information about the event type mapping is below:

System event (eximee)	Business event	Description
FORM_NEW_STARTED	FORM_ENTRY	Entering the application
FORM_NEXT_PAGE FORM_PREVIOUS_PAGE	PAGE_SAVE	Saving the current page state - moving to the next or previous page
FORM_CLIENT_PAGE_CHANGED	PAGE_ENTRY	Page change - entering another page.
FORM_SAVE	FORM_SAVE	Submitting the application
FORM_DRAFT_SAVE	FORM_DRAFT_SAVE	Saving a draft version of the application
FORM_DRAFT_RESTORE	FORM_DRAFT_RESTORE	Restoring a draft version of the application

Database structure (PostgreSQL)

Specified database structure with business events.

Schema			monitoring_events
Table			events
Field name	Type	nullable	Description
id	bigserial	FALSE	Sequential event identifiers
origin	text	FALSE	Event origin (SYSTEM / BUSINESS)
type	text	FALSE	Business event type (the event type table is above)
timestamp	timestamp	FALSE	Time the event was triggered (date and time)
payload	jsonb	TRUE	Event data as JSON, if the event has additional data.

Migration scripts

Migration scripts for creating and modifying the table are stored in the archive to be delivered together with the platform.

Webforms application with event logging

Purpose

The event logging mechanism is responsible for collecting events related to changes in the state of applications across the entire platform. The events are sent in batches to Kafka using EximeeEvents.

Configuration

To activate event collection, the configuration should be set appropriately webforms.xml. Below is a configuration fragment that is responsible for the active event collection flow.

The connector to eximeeEvents is delivered in the eximee-events library

<?xml version="1.0" encoding="UTF-8" ?>
<webforms>
   ....
   <server>
      ...
      <features>
         ...
         <eventLog>
            <enabled>true</enabled>
            <coreExecutorPoolSize>5</coreExecutorPoolSize>
            <maxExecutorPoolSize>10</maxExecutorPoolSize>
            <eximeeEventsProcessor>
               <enabled>true</enabled>
               <url>http://localhost:9002</url>
            </eximeeEventsProcessor>
         </eventLog>
      </features>
   </server>
</webforms>

And a table showing the installation configuration.

Installation key	Configuration key	Example value	Description
Active event collection mechanism
	server.features.eventLog.enabled	true	Activity of the event collection mechanism
	server.features.eventLog.coreExecutorPoolSize	5	Number of threads within event processing
	server.features.eventLog.maxExecutorPoolSize	10	Maximum number of threads within event processing
Active event logging mechanism
	server.features.eventLog.eximeeEventsProcessor.enabled	true	Activity of the EximeeEvents mechanism
	server.features.eventLog.eximeeEventsProcessor.url	http://localhost:9002	Address of the EximeeEvents application for saving events to Kafka