EPG Delivery and Caching

This topic describes the EPG (Electronic Program Guide) processing — from import to the SmartTUBE database to delivery to client devices, EPG caching levels, and managing caching parameters.

Levels of Caching

The source EPG data is provided by third-party systems as a single XML file containing information for all channels. The file format supported by SmartTUBE is described in the chapter “EPG file format“.

Multi-level caching is used to reduce the load on various nodes of the broadcast platform and speed up EPG data processing. The purpose of each level in the EPG processing chain (in order from highest to lowest) is described below.

EPG storing and caching levels

SmartTUBE SDP Server

SmartTUBE SDP server imports the EPG file automatically at the frequency defined by the corresponding job in the SmartTUBE Admin Console (usually 1 hour or more). Thus, the SmartTUBE SDP database contains the most up-to-date information about the program guide in the system. Creating and configuring an EPG import job is described here.

The SmartTUBE Admin Console also allows to edit EPG for each channel manually — in the Channels > EPG menu.

When importing XML, all manual changes will be overwritten by the imported data (if the XML contains data about the changed TV programs). For this reason, manual EPG editing is mainly used for test purposes.

The SmartTUBE SDP server controls the caching settings of HTTP servers (Nginx) and client devices (see the “Managing the EPG Cache Validity Timeout” chapter).

SmartTUBE Cache Servers

Direct interaction between the SmartTUBE SDP server and client devices is performed by SmartTUBE Cache servers. Each Cache server performs all the main server functions, including storing and providing EPG data. This data is updated from SmartTUBE SDP Server as soon as the import process is complete or manual changes in SmartTUBE Admin Console are made (typically within 1 minute).

To improve fault tolerance, the task to update EPG on Cache servers is usually performed at night. This task is created in the SmartTUBE Admin Console > Administration > Scheduled jobs using the method “Reload cache data (cacheoperations#reload)” with the parameter caches=”Epg”.

The SmartTUBE Cache server caches in the memory the response to the channels/getLastUpdate request.

Nginx Cache

In addition, to accelerate the response and allow to use CDN to store EPG data, each HTTP server (Nginx) uses a cache option to store this data as static content.

Nginx servers cache the response to the EPG request cached/epg/list.

Typically, client devices request EPG per channel per day. Therefore, Nginx stores EPG in the appropriate form — as one static unit per day per channel.

Client Apps Cache

Client apps store EPG data locally, which allows to avoid downloading up-to-date data every time application restarted, exits from standby mode, or restores the network connection. This mechanism allows to reduce the response time of UI.

The local EPG cache is updated in the following cases:

• At midnight (see the chapter “Next day EPG Download”);
• When the cache size limit is reached:

For example, if a user downloaded more data than allowed by the configuration while scrolling a program guide, the oldest data is erased from the local cache.

Cache size is specified by the parameter networkDiskCacheSize_<STB model or client type> in the [UI] section of the configuration file /opt/smartlabs/nginx/html/srv-pub/networkConfig/config.cfg (located on the SmartTUBE cache-server). Specified in Mbytes. Example:

[UI]    networkDiskCacheSize_android = 0    networkDiskCacheSize_SML-482 = 5

• By notifications reloadChannelsModifyTime and reloadEpg from the SmartTUBE SDP server.
• By timeout specified in the SmartTUBE Cache response (in the header Cache-Control: max-age) to the last client request cached/epg/list.

Managing the EPG Cache Validity Timeout

The validity time of EPG data cached on Nginx servers and client devices is limited and managed by the SmartTUBE SDP configuration file system.conf.

• Nginx cache time management is performed via the X-Accel-Expires header in the SmartTUBE Cache response to the client request cached/epg/list.
• Validity time of the local EPG cache on the client devices can be managed with the Cache-Control: max-age header in the SmartTUBE Cache response to the client request cached/epg/list.
• Settings of reloadChannelsModifyTime and reloadEpg notifications define the delay required for the nginx cache to update and the period during which clients should request EPG at a random point in time.

The default system.conf settings are shown below:

# for /v1/cached/epg/ epg.cached.setCacheMaxAge.cacheControl.sec = 2592000 # sets a header Cache-Control: max-age=2592000 (30 days) for the client device, i.e. it is assumed that the client will update the EPG only by notifications or by its own request, when the EPG ends for the current day epg.cached.setCacheMaxAge.xAccessExpires.sec = 600 # sets a header X-Accel-Expires: 600 (10 minutes) for Nginx (for the /cached/epg/ requests) and SmartTUBE Cache (for the /channels/getLastUpdate request) # for notifications epg.update.notification.hold.sec = 600 # Delay added to the Nginx caching interval (epg.cached.setCacheMaxAge.xAccessExpires.sec) before sending the reloadChannelsModifyTime notification epg.update.notification.delay.sec = 3600 # The period during which clients should request EPG at a random point in time (to avoid the DDOS effect). Passed with the reloadEpg notification

Updating Caches

The logic for updating EPG caches at different levels depends on the events described below.

EPG Update on Re-import

In case of new EPG was re-imported from provider, SmartTUBE SDP server waits for SmartTUBE Cache server DB to update and the Nginx cache to expire (typically set to 20 minutes) and then sends notification reloadChannelsModifyTime to all active clients devices to invalidate their local cache.

To avoid revalidation of all EPG data, client app requests actual list of changes with the channels/getLastUpdate method, and invalidates only EPG which was actually modified.

Then the client app gets the new EPG for invalidated channels (with the cached/epg/list request) for the current date at random moment within a configurable interval (10 minutes by default).

EPG update on re-import

In case of manual changes via the SmartTUBE Admin Console, the SmartTUBE Cache DB updates immediately after saving the changes (typically within 1 minute). But the reloadEpg notification is sent to the clients 20 minutes after the Notify about new EPG events button is clicked. This notification defines the channels, the EPG of which the client needs to re-download.

EPG update on manual changes in Admin Console

EPG Download on Client Startup

If the client device does not have valid EPG cache (checked by the channels/getLastUpdate request) on the moment of startup, exit from standby mode, or network connection appearance the device downloads it for the current day and stores in local cache.

Thus, if the device restarted immediately it would not lead to another EPG download.

EPG update on client startup, exit from standby mode, network connection appearance

Next day EPG Download

To be sure the device always has relevant EPG for the current day, it automatically downloads next day data before midnight (with the cached/epg/list request).

To avoid simultaneous download and DDOS effect this time varies for different devices, so each device downloads it at random time within a configurable period (from 23:00 to 23:59 by default).

EPG Update on Client UI Request

If the subscriber is scrolling through the program guide, the client app requests the EPG page-by-page using the method cached/epg/list until it reaches the EPG storage limit (i.e. receives an empty response). This limit is set on the SmartTUBE SDP server in the system.conf file:

epg-import.days = 7 # EPG storage limit in days after the current date epg-import.minusDays = 7 # EPG storage limit in days before the current date

Methods and Notifications Used

PUSH notification: reloadChannelsModifyTime

Purpose

Notifies the client application to request a list of EPG modification timestamps for all channels.

Parameters Passed

Parameter	Type	Required?	Description
requestDelay	integer	yes	The period during which the client application must request an EPG at a random time after receiving the notification, in sec.

Notification Example

command?commandType=Control&commandName=reloadChannelsModifyTime&requestDelay=3600

PUSH notification: reloadEpg

Purpose

Notifies the client application to request EPG of the specified channel.

Parameters Passed

Parameter	Type	Required?	Description
Channel	integer	yes	IDs of the channels whose EPG was changed on the last import from the provider, separated by a comma.
day	date	yes	The date on which the EPG should be updated, in the format MM/DD/YYYY.
requestDelay	integer	yes	The period during which the client application must request an EPG at a random time after receiving the notification, in sec.

Notification Example

command?commandType=Control&commandName=reloadEpg&Channel=23648161,23262298&day=04/10/2019&requestDelay=3600

API method: /channels/getLastUpdate

Purpose

The method is called by the client apps and returns the list of all channels available to the subscriber. The client determines the channels for which EPG was changed during the last import using the time parameter in the SmartTUBE Cache response.

Call parameters

Parameter	Type	Required?	Description
lang	String	yes	EPG language

Return value

If executed successfully, the method returns the following data for each channel:

Field	Description
id	Channel ID
time	Timestamp of the last changes in the EPG for the specified channel, in UTC.

Return Example

{

“result”: {

“list”: [

{

“id”: 21885276,

“time”: 1575841318

},

{

“id”: 21885381,

“time”: 1575841297

},

{

“id”: 21885271,

“time”: 1575841693

},

…]

}

}

API method: /cached/epg/list

Purpose

The method is called by the client apps and returns EPG for the channel requested. This method does not require client authorization.

Note: Client apps also use the /epg/list request (not /cached/epg/list) to the SmartTUBE Cache server to search through EPG. This request is not cached by Nginx servers and requires client authorization.

Call parameters

Parameter	Type	Required?	Description
lang	String	yes	EPG language
channelId	Integer	yes	Channel ID. Must contain only one ID per request (for optimal caching on Nginx servers).Note: If you need to pass multiple channel IDs in a single request, use the /epg/list method.
startDate	UTC	yes	Initial timestamp of the EPG requested. In the current implementation, it coincides with the beginning of the day.
stopDate	UTC	yes	Final timestamp of the EPG requested. In the current implementation, it coincides with the end of the day.
from	Integer	no	The number of the initial entry in the requested EPG.
to	Integer	no	The number of the final entry in the requested EPG.

Return value

If executed successfully, the method returns the following data:

Field	Description
id	The identifier of the programme
name	Programme name
dsc	Any additional information
sdate	Beginning of the programme; date and time (UTC time zone)
fdate	End of the programme; date and time (UTC time zone). This time = time of the next programme beginning
chId	The identifier of a TV channel the program is shown at
cRId	Not used
cId	Content identifier
discr	The discriminator: EPG
clogo	Image
al	Age requirements of the program
alSort	Position in the list of age requirements
ir	1 – the programme is recommended; 0 – the programme is not recommended
select	Not used
bundle	If the programme is an episode of a series – name of the series
genre	Programme main genre
imdbR	Content raiting at www.imdb.com
version	Date of the last programme changing in cache. Could be used as a version of the programme for EPG changes monitoring from a subscriber’s device
recSeries	Recording series if true
metaContentSeriesId	Not used
seriesNumber	A unique ID of the series in SmartTUBE database; created automatically. It is used, for example, for nPVR recording of series — to identify which programme belongs to which series.
seasonNumber	Number of the season in the series
seasonSeriesNumber	Number of the episode in the season
firstShowing	Indicates that the episode is being aired for the first time (not a repeat). Optional, if not set, it is assumed to be “false”.
originalAirDate	Original air date (first broadcast, not repeat)
year	Year of the programme release
country	A unique ID of the country in SmartTUBE database; created for it automatically
category	A unique ID of the category in SmartTUBE database; created for it automatically
isLive	Identifies that this is live-program (if 1 – live, if the field is missing in the method’s answer – not live)
isRepeat	Identifies that this is the repeat of the program (if 1 – repeat, if the field is missing in the method’s answer – not repeat)
fs	Identifies that this is the premier (if 1 – premier, if the field is missing in the method’s answer – not premier)
ls	Identifies that this is the last showing of the program (if 1 – last showing, if the field is missing in the method’s answer – not last showing)
isFree	Identifies that this program is free (if 1 – free, if the field is missing in the method’s answer – not free)
oName	Programme name in the original language
eName	Name of the corresponding episode
descLong	Full description; displayed in a subscriber’s UI
actors	Actors
director	Directors
screenplay	Screenplays
operator	Operators
composer	Composers
producer	Producers
leading	Leadings
banner	The banner
genreList	The list of programme’s genres
categoryList	The list of programme’s categories
rubrics	Type of programme
countries	List of countries
sCnt	The quantity of series in the season
isHD	1 – the programme supports HD format 0 – the programme is broadcasted only in SD format
packages	Not used
Section metaContent — The content associated with this programme
id	A unique ID of the metacontent in SmartTUBE database; created automatically
type	The type of the metacontent (video, episode, season, series)
seasonId	A unique ID of the season in SmartTUBE database; created automatically
seriesId	A unique ID of the series in SmartTUBE database; created automatically