SmartCARE Self-Monitoring

Logging Settings

To monitor the most critical SmartCARE services (collector-http and collector-nginx), the Log4j library is used, which is the de facto standard for Java applications logging. Logging of both Collectors has the following similarities:

service logs are divided into 2 types:
- for Apache Spark used — external-libs-x.log,
- for applications — collector-http-x.log and collector-nginx-x.log, where x — the log number (0 or 1);

When rotation is enabled (log4j.appender.rollingFile.RollingPolicy.maxIndex == 1), 2 logs are written for each application — first 0, then 1.
Logs are rotated (overwritten) when a certain size is reached (TriggeringPolicy.MaxFileSize == 536870912 — 512 MB). Rotation is performed according to the following algorithm:
- 0-th log reached 512 MB;
- 1-st log is created, to which data from the 0-th is transferred;
- 0-th continues to be recorded;
- when the 0-th and 1-st reached 512 MB, the 1-st is overwritten, the data from the 0-th is again transferred to it. Thus, the 0-th log will contain the most recent data, and both logs will occupy a maximum of 1 GB.

Configuring Reference

The configuration files location:

collector-http: /opt/smartlabs/smartcare/collector-http/conf/log4j.properties
collector-nginx: /opt/smartlabs/smartcare/collector-nginx/conf/log4j.properties

Most configuration file parameters are standard for Log4j logging and are described on the Apache Log4j 2 website. However, some settings sections require comments:

Settings sections	Description
rollingFile by size	Main log file rotation settings.
rootLogger precision conf	Settings of the built-in library for working with mongoDB.
spark && others appender	Apache Spark built-in library settings. ATTENTION: default settings are optimal in terms of space occupied by logs, logs are written in a separate file. Be careful when changing settings.
rolling file by size	Special settings for Spark log files rotation.

Example of the Collector-http Logging Configuration

log4j.rootLogger=DEBUG, rollingFile

# rollingFile by size
log4j.appender.rollingFile=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.rollingFile.RollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.rollingFile.RollingPolicy.maxIndex=1
log4j.appender.rollingFile.TriggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.rollingFile.TriggeringPolicy.MaxFileSize=536870912
log4j.appender.rollingFile.RollingPolicy.FileNamePattern=/var/log/smartlabs/smartcare/collector-http/collector-http-%i.log
log4j.appender.rollingFile.RollingPolicy.ActiveFileName=/var/log/smartlabs/smartcare/collector-http/collector-http-0.log
log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p %C{1}.%M (line:%L): %m%n

# rootLogger precision conf
log4j.logger.org.mongodb.driver=WARN

# spark && others appender
log4j.logger.org.apache.spark=DEBUG, external-libs
log4j.additivity.org.apache.spark=false

log4j.logger.org.spark_project=DEBUG, external-libs
log4j.additivity.org.spark_project=false

log4j.logger.org.apache.hadoop=DEBUG, external-libs
log4j.additivity.org.apache.hadoop=false

log4j.logger.io.netty=DEBUG, external-libs
log4j.additivity.io.netty=false

# rolling file by size
log4j.appender.external-libs=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.external-libs.RollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.external-libs.RollingPolicy.maxIndex=1
log4j.appender.external-libs.TriggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.external-libs.TriggeringPolicy.MaxFileSize=536870912
log4j.appender.external-libs.RollingPolicy.FileNamePattern=/var/log/smartlabs/smartcare/collector-http/external-libs-%i.log
log4j.appender.external-libs.RollingPolicy.ActiveFileName=/var/log/smartlabs/smartcare/collector-http/external-libs-0.log
log4j.appender.external-libs.layout=org.apache.log4j.PatternLayout
log4j.appender.external-libs.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p %C{1}.%M (line:%L): %m%n

Example of the Collector-nginx Logging Configuration

log4j.rootLogger=DEBUG, rollingFile

# rollingFile by size
log4j.appender.rollingFile=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.rollingFile.RollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.rollingFile.RollingPolicy.maxIndex=1
log4j.appender.rollingFile.TriggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.rollingFile.TriggeringPolicy.MaxFileSize=536870912
log4j.appender.rollingFile.RollingPolicy.FileNamePattern=/var/log/smartlabs/smartcare/collector-nginx/collector-nginx-%i.log
log4j.appender.rollingFile.RollingPolicy.ActiveFileName=/var/log/smartlabs/smartcare/collector-nginx/collector-nginx-0.log
log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p %C{1}.%M (line:%L): %m%n

# rootLogger precision conf
log4j.logger.org.mongodb.driver=WARN

# spark && others appender
log4j.logger.org.apache.spark=DEBUG, external-libs
log4j.additivity.org.apache.spark=false

log4j.logger.org.spark_project=DEBUG, external-libs
log4j.additivity.org.spark_project=false

log4j.logger.org.apache.hadoop=DEBUG, external-libs
log4j.additivity.org.apache.hadoop=false

log4j.logger.io.netty=DEBUG, external-libs
log4j.additivity.io.netty=false

# rolling file by size
log4j.appender.external-libs=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.external-libs.RollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.external-libs.RollingPolicy.maxIndex=1
log4j.appender.external-libs.TriggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.external-libs.TriggeringPolicy.MaxFileSize=536870912
log4j.appender.external-libs.RollingPolicy.FileNamePattern=/var/log/smartlabs/smartcare/collector-nginx/external-libs-%i.log
log4j.appender.external-libs.RollingPolicy.ActiveFileName=/var/log/smartlabs/smartcare/collector-nginx/external-libs-0.log
log4j.appender.external-libs.layout=org.apache.log4j.PatternLayout
log4j.appender.external-libs.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p %C{1}.%M (line:%L): %m%n

Collector Module Statistics

The main menu section “SETTINGS → Collectors stats” allows you to control the overall performance of collecting statistics from SmartCARE clients. The section displays the information about the collector-nginx and collector-http submodules.

The “Total stat” area provides information on event processing for the entire service life, while the “Stat since last restart” area — since the last restart of the Collector module.

The chart displays information only for the last 24 hours.

Collector module statistics

Parameter	Description	Data source (collection.field)
Total stat — statistics for all the time of the Collector module
Start	Date and time of the first Collector launch.	ngServiceStat.type==”all” → ngServiceStat.start clServiceStat.type==”all” → clServiceStat.start
Success	Number of correct events processed for all the time of the Collector module.	ngServiceStat.type==”all” → ngServiceStat.success + clServiceStat.type==”all” → clServiceStat.success
Error	Number of error events with initially incorrect structure (events did not have enough required data, etc.) and therefore not processed by Collector for all the time of the module.	ngServiceStat.type==”all” → ngServiceStat.error + clServiceStat.type==”all” → clServiceStat.error
Stat since last restart — statistics for the Collector module since its last restart
Start	Date and time of the Collector’s last restart.	ngServiceStat.type==”restart” → ngServiceStat.start clServiceStat.type==”restart” → clServiceStat.start
Success	Number of valid events processed by the Collector since its last restart.	ngServiceStat.type==”restart” → ngServiceStat.success + clServiceStat.type==”restart” → clServiceStat.success
Error	Number of error events not processed by the Collector since its last restart.	ngServiceStat.type==”restart” → ngServiceStat.error + clServiceStat.type==”restart” → clServiceStat.error