Quality Monitoring

The Quality monitoring section allow you to analyze in detail the changes in various parameters of streaming quality for a certain period.

BITRATE SWITCH COUNT STACK

Each column of this chart reflects the distribution of client sessions by the number of stream bitrate switching during each session. For this purpose the color coding is used. Decoding is given when you hover the cursor over the chart. New columns are drawn every minute. The data source for the chart is the BITRATECHANGE events received from end-user devices.

Distribution of client sessions by the number of stream bitrate switching

For example, in the image below you can see that in the selected minute there was 1 session with more than 10 bitrate switches, 1 session with bitrate switches between 4 and 10 times, 6 sessions with bitrate switches between 2 and 3 times, and so on.

Session Quality Metrics

This section allows you to place several metrics of session quality on the chart to analyze their mutual dependence in time. Also by clicking you can display the selected metrics for each minute as a table.

Summary chart of the streaming quality parameters

By moving the cursor over the chart, you can explore every minute of the platform’s operation for a quantitative analysis of the following broadcast quality parameters:

Parameter	Description
Session quality (unicast / multicast)	Average weighted quality of all sessions per minute.
Session count (unicast / multicast)	Number of active sessions (not timed out) per minute.
CPU usage	Subscriber devices’ CPU usage: Realtime — average % of the CPU usage for all sessions that generated events CPU_LOAD at a given minute. Average — average % of the CPU usage for all sessions that have not timed out.
Memory usage	Subscriber devices’ memory usage: Realtime — average % of the memory usage for all sessions that generated events MEMORY at a given minute; Average — average % of the memory usage for all sessions that have not timed out.
Join time	The time between a subscriber pressing a button on the remote control (not releasing the button) and the start of playback: Realtime — average join time for all sessions that generated events JOIN_TIME at a given minute. Average — average join time for all sessions that have not timed out.
Duration of sessions	Duration of client sessions: Realtime — average duration of all sessions that generated events at a given minute. Average — average duration for all sessions that have not timed out.
Authorization count	Number of subscriber device authorizations per minute.
Bitrate change	Number of stream bitrate switching over all subscriber devices per minute.
Switch session count	Number of session closing events with the opening of a new one (switching to another channel / movie) for all subscriber devices per minute.
Metadata error count	Number of metadata (playlists) download errors for all subscriber stations per minute. Counting the number of errors is made if 2 conditions are met: If there is no .ts or .mp4 in the URL in the incoming log, then this is metadata. If the incoming log contains httpStatus <200 or httpStatus> = 400, then an error has occurred.
Chunk load error count	Number of mediadata (chunks) download errors for all subscriber stations per minute. Counting the number of errors is made if 2 conditions are met: if .ts or .mp4 is present in the URL in the incoming log, then this is mediadata; if the incoming log contains httpStatus <200 or httpStatus> = 400, then an error has occurred.
Buffer run count	Number of stream buffering events (starting or completion)
Online device count	Number of devices online

The Show aggregated data checkbox allows you to quickly switch between 2 states of the chart:

Checked — display graph on all session modes (Live, DVR, VoD, Other).
Unchecked — display a graph filtered by session mode and device type specified below.

Session Quality Stack

Each minute column of this chart reflects the quantitative distribution of client sessions according to their bit rate. For this purpose the color coding is used. Description is given when you hover the cursor over the chart.

Quantitative distribution of sessions by bitrate

Bitrate gradation — Low, Medium and High — is determined by the properties of each TV channel and VoD unit in the respective SmartTUBE dictionaries. Each of these bitrates also has a specific numerical designation, e.g. 2000000 bit/s. The value Not streams corresponds to the situation when such dictionaries are missing or the bitrate values are not specified.

Streaming Servers

Analysis of the streaming servers operating parameters, such as network interfaces utilization and CPU usage, is performed in the “Streaming servers” section. All section metrics are displayed on a single chart with the ability to enable/disable each of them. The chart data is updated every minute.

The data source for the chart is the zbMinuteAgg collection of the mongoDB database (see the “mongoDB Database Structure” section).

Parameter	Description	Data source (collection.field)
Input stream bitrate	Average bitrate of the incoming traffic from the DVR/VoD content storage per minute (with the ability to filter the streaming servers and network interfaces).	zbMinuteAgg.values.inBitrate
Output stream bitrate	Average bitrate of outgoing traffic per minute (with the ability to filter streaming servers and network interfaces).	zbMinuteAgg.values.outBitrate
Interface utilization (input stream)	Average load on all or a particular network interface (selected using a filter) to process incoming traffic per minute.	zbMinuteAgg.values.inUtilization
Interface utilization (output stream)	Average load on all or a particular network interface (selected using a filter) to process outgoing traffic per minute.	zbMinuteAgg.values.outUtilization
CPU IOWait	Average CPU IOWait value per minute.	zbMinuteAgg.cpuIowait

Optionally, this section can display data about any other system on which a zabbix agent is installed and properly configured (for example, DVR / VoD content storage system). To do this, add the agent IP address and access details in the Collector module configuration (see the “Configuring Components” section). Information about such a system can be displayed on the chart using the filter in the upper right corner of the Care UI.

Input Streams

The “Input streams” section is represented by two metrics to assess the quality of incoming TV streams delivery from head-end stations to recording servers. Metrics can be displayed on one chart for comparative analysis. Data is updated every minute.

The data source for the chart is the smMinuteAgg and smMinuteAggByParams collections of the mongoDB database (see the “mongoDB Database Structure” section).

Recording servers monitoring

Parameter

Description

Data source (collection.field)

Errors

Number of incoming streams delivery errors for all recording servers and channels or with filtering.

Total value: smMinuteAgg.avgBitrate

Filtered by servers and channels: fields of the smMinuteAggByParams collection.

Average bitrate

Average bitrate of the incoming streams summarized over all recording servers and channels or with filtering.

Total value: smMinuteAgg.errors

Filtered by servers and channels: fields of the smMinuteAggByParams collection.

Table parameter	Description	Data source (collection.field)
Date	Date and time of the event.	smEvent.date
Type	Event type. In the current version only error events are counted and displayed.	smEvent.type
Hostname	Recording server URL.	smEvent.hostname
Content	Channel name.	smEvent.content
Message	Full event text.	smEvent.message

Important Messages and Warnings

Only important log entry messages that may affect the quality of service are described here.

Info messages

Reading bitrate per channel and per stream:

recorderId=on-tor-king-rec01[2019-11-06 05:25:28.045233] [0x00007f0c927fc700] [debug] [default] overall reading: 2.707 Mb/sec. Channel: CH_WEANH, source: src_CH_WEANH_3200000

recorderId=on-tor-king-rec01[2019-11-06 05:25:28.045297] [0x00007f0c927fc700] [debug] [default] overall reading: 3.652 Mb/sec. Channel: CH_WEANH, source: src_CH_WEANH_4500000

recorderId=on-tor-king-rec01[2019-11-06 05:25:28.045326] [0x00007f0c927fc700] [debug] [default] overall reading: 5.481 Mb/sec. Channel: CH_WEANH, source: src_CH_WEANH_6500000

New period started:

recorderId=on-tor-king-rec01[2019-11-06 05:23:43.560767] [0x00007f0c89ffb700] [debug] [default] New mp4 mongo multitrack header with id = “5dc29f2f5ce87928aa4a7f48”, npvr_id = “CH_APTNH”

Warnings and errors

DTS desync:

recorderId=on-tor-king-rec01[2019-11-06 05:25:30.078730] [0x00007f0c577fe700] [error] [default] MasterSynchronizer: master-track key frame desync; track = dmx_CH_MSNHD_6500000$video/bw6500000:vid0 dts = 8547497459, real = 141573231897826, diff = 20102873, Channel: CH_MSNHD

Track syncing failed:

recorderId=on-tor-king-rec01[2019-11-06 05:25:06.468044] [0x00007f0c91ffb700] [error] [default] Failed to synchronize tracks for channel “CH_FOXNH”. Sample buffer size = 1500, Channel: CH_FOXNH

Chank generation failed:

recorderId=on-tor-king-rec01[2019-11-06 00:02:36.395760] [0x00007f0c927fc700] [debug] [default] Can’t add sample – reset all TrackChunkers on CH_ABCHW_TOSS

Chank generation failed, buffer overflow:

recorderId=on-tor-king-rec01[2019-11-06 00:02:38.056360] [0x00007f0c93fff700] [error] [default] Chunker buffer overflow CH_VCTV2 dmx_CH_VCTV2_6500000$video/bw6500000:vid0

Continuity counter (CC) errors:

recorderId=on-tor-king-rec01[2019-11-06 00:02:38.188904] [0x00007f0c89ffb700] [warning] [default] packet warning TSWARN CC occured 6 times. Channel: CH_GLBHT, source: src_CH_GLBHT_6500000

Abnormal Clusters

This section allows you to identify various abnormal events in the broadcasting system, identify groups of devices where problems occur, and localize the most likely cause.

Summary of anomalies grouped by their probable causes

Parameter	Description	Data source (collection.field)
Param	Parameter (object) on which anomalies were observed.	multi_outliers-bad.param
Param value	Parameter value.	multi_outliers-bad.par_val
Average abnormality	Average abnormality.
Average bad sessions count per time check	Average bad sessions count per time check.	multi_outliers-bad.mean_bad
Average quality	Average quality of bad sessions associated with this object.	multi_outliers-bad.mean_qual
Average total sessions count per time check	Average total sessions count per time check with this multi_outliers-bad.par_val.	multi_outliers-bad.mean_tot
Slices	Total number of meaningful time checks, i.e. when the problem on this object was observed and at least 1 new bad session was added.	multi_outliers-bad.nslices
Last changed	Time of the last slice when the anomaly for this object was detected.	multi_outliers-bad.last

Realtime Indicators

The “Realtime indicators” section is represented by 2 indicators of the streaming quality, 2 indicators of the of current sessions number and 1 summary report.

Unicast & Multicast Session Quality

The formula for calculating these indicators allows to objectively determine the quality of current unicast and multicast session. To calculate quality factors, the data stored in the clSession and clEvent collections is used (see the section “mongoDB Database Structure”).

Calculation and updating of the indicator values is performed every minute according to the Algorithm 3.

What could the session quality mean?

Current Sessions Number

These indicators display the number of currently active unicast and multicast sessions, which can be used to estimate the service usage level. The indicators update period in CareUI is 1 minute, in mongoDB database — 20 seconds (configurable parameter, see “Configuring Components“).

The indicator values are calculated using the clSession.active == true (multicast) and ngSession.active == true (unicast) fields in the mongoDB database (see the “mongoDB Database Structure” section).

Unrecognized Events

This report displays events when the end-user device sent a care-request with an incorrect parameter value and SmartCARE was forced to

completely reject this event, if an incorrect parameter is required for this care-request (e.g. csi), or
reject this parameter, but the event was saved.

Unrecognized events can also be detected in case when the client app starts sending some new type of event, but SmartCARE system did not recognized it as some known event (like Keep Alive, Playback, Standby, etc.), because it has not been updated yet.

List of unrecognized events

Details on unrecognized event