Monitoring platform events

Requirements

• active Router2 application with EximeeEvents

  • application provided with the Eximee platform since version 2.3167.0

• active WebformsRest application

  • application provided with the Eximee platform

• active MonitoringLogProcessor application

  • application provided with the Eximee platform 2.3019.0

• active database storing events

  • the application requires PostgreSQL version >=9.5 to operate

• active Kafka instance

  • we recommend version >= 3.4

• active Grafana instance min 8.X.X

  • a sample monitoring dashboard is provided with the platform in the grafana.json file

Enabling the functionality in the platform is controlled by adding the value "monitoring" to the COMPOSE_PROFILES variable.

Operational assumptions

The Webforms application generates events related to the state of forms.

Various implementations of event logging can be active at the same time. It depends on configuration and deployment. EximeeEventsConnector (one of the possible implementations) collects events and sends them to EximeeEvents.

EximeeEvents writes them to the appropriate Kafka topic.

MonitoringEventProcessor reads events from Kafka, verifies them and writes them to the database (PostgreSQL) in a way that can be read by Grafana.

Events in the database can be read by external applications for visualization or analysis. Grafana configuration for visualizing events will be delivered together with the platform.

Figure 1. Architecture of the event monitoring solution

Router2 application with EximeeEvents

Purpose

EximeeEvents is a plugin for the Router2 application. It provides an endpoint that accepts events which are then written to the appropriate Kafka topic.

The topic and the message group sent to Kafka must be consistent with the MonitoringLogProcessor configuration.

Configuration

Configuration and more information here [Process event registration]

MonitoringLogProcessor application

Purpose

The application's task is to populate the database (currently only PostgreSQL) with platform events.

The application reads events from the source (Kafka) and verifies whether it is a system event (originating from the platform). Then system events are adapted for Grafana and written to the database.

Configuration

Key	Requirement	Example	Description
Database - target place for storing events
MONITORING_LOG_PROCESSOR_DATABASE_DIALECT	YES	org.hibernate.dialect.PostgreSQL95Dialect	Dialect of the database intended for storing events.
MONITORING_LOG_PROCESSOR_DATABASE_URL	YES	jdbc:postgresql://postgresql01.host:5432/monitoring_events	Database connection string.
MONITORING_LOG_PROCESSOR_DATABASE_USERNAME	YES	db_user	User with permissions to write events.
MONITORING_LOG_PROCESSOR_DATABASE_PASSWORD	YES	db_pass	Password of the database access user.

Kafka

Data source	Requirement
MONITORING_LOG_PROCESSOR_KAFKA_URL	YES	http://kafka.host:9092	Kafka endpoint
MONITORING_LOG_PROCESSOR_KAFKA_TOPIC	YES	all	Topic where event messages will go
MONITORING_LOG_PROCESSOR_KAFKA_GROUP	YES	eximee-events	eximee event group

Stored events

Events must be of system origin (origin = SYSTEM).

Event types are mapped to business types. A type that has no equivalent in the business event receives an empty value. Information about the event type mappings is provided below:

System event (eximee)	Business event	Description
FORM_NEW_STARTED	FORM_ENTRY	Form entry
FORM_NEXT_PAGE FORM_PREVIOUS_PAGE	PAGE_SAVE	Saving the state of the current page - moving to the next or previous page
FORM_CLIENT_PAGE_CHANGED	PAGE_ENTRY	Page change - entering a different page.
FORM_SAVE	FORM_SAVE	Form submission
FORM_DRAFT_SAVE	FORM_DRAFT_SAVE	Saving a draft version of the form
FORM_DRAFT_RESTORE	FORM_DRAFT_RESTORE	Restoring a draft version of the form

Database structure (PostgreSQL)

Defined database structure with business events.

Schema			monitoring_events
Table			events
Field name	Type	nullable	Description
id	bigserial	FALSE	Sequential event identifiers
origin	text	FALSE	Event origin (SYSTEM / BUSINESS)
type	text	FALSE	Business event type (the table with event types is shown above)
timestamp	timestamp	FALSE	Event invocation time (date and time)
payload	jsonb	TRUE	Event data in JSON form, if the event has additional data.

Migration scripts

Migration scripts to create and modify the table are saved in an archive to be delivered with the platform.

Webforms application with event logging

Purpose

The event logging mechanism's task is to collect events related to changes in the state of forms across the platform. Events are sent in batches to Kafka using EximeeEvents.

Configuration

To enable event collection, you need to set the configuration appropriately webforms.xml. Below is a fragment of the configuration responsible for the active event collection flow.

The connector to eximeeEvents is provided in the eximee-events library

<?xml version="1.0" encoding="UTF-8" ?>
<webforms>
   ....
   <server>
      ...
      <features>
         ...
         <eventLog>
            <enabled>true</enabled>
            <coreExecutorPoolSize>5</coreExecutorPoolSize>
            <maxExecutorPoolSize>10</maxExecutorPoolSize>
            <eximeeEventsProcessor>
               <enabled>true</enabled>
               <url>http://localhost:9002</url>
            </eximeeEventsProcessor>
         </eventLog>
      </features>
   </server>
</webforms>

And a table showing the installation configuration.

Installation key	Key in configuration	Sample value	Description
Active event collection mechanism
	server.features.eventLog.enabled	true	Activity of the event collection mechanism
	server.features.eventLog.coreExecutorPoolSize	5	Number of threads for event processing
	server.features.eventLog.maxExecutorPoolSize	10	Maximum number of threads for event processing
Active event logging mechanism
	server.features.eventLog.eximeeEventsProcessor.enabled	true	Activity of the EximeeEvents mechanism
	server.features.eventLog.eximeeEventsProcessor.url	http://localhost:9002	Endpoint for the EximeeEvents application to write events to Kafka