Apache HTTP Server monitoring

Apache performance and understanding /server-status metrics

Apache Statistics Module: mod_status

Understanding website performance and Apache uptime should begin with external monitoring, i.e. Zabbix, which allows you to discover problems from a user’s perspective. Such problems involve:

• Response timeout – no timely response from the server;
• HTTP errors instead of a response;
• Slow response – say, comparing to the last month’s average.

All these problems should be confirmed by monitoring from multiple locations to eliminate network related problems. Only when the same issue is observed from several locations at the same time can we conclude there is something wrong on the server side of things.

The next step is to check whether everything is OK with the server itself. This is where basic metrics like CPU, memory, disk and swap come into play offering you some guidance on where to focus further.

The most frequent reason for website performance issues is malfunctioning, buggy or not optimized application (e.g. PHP script), and slow DB queries. Before digging deep into profiling of an application and its queries, however, it makes sense to check Apache configuration and statistics – and this is what this post is all about.

Apache web server exposes metrics through its status module, mod_status. If your server is running and mod_status is enabled, your server’s status page should be available at http://<YOUR_DOMAIN>/server-status. If that link does not work, it means you need to enable mod_status in your configuration file.

It’s also possible that Location specified in your configuration file is not /server-status, intentionally or not. Follow the directions below to locate your mod_status configuration file, and look for a directive that contains SetHandler server-status. If you see that it specifies a Location other than /server-status, either update it accordingly (and restart Apache) or try accessing that endpoint to see if mod_status is enabled at that location. After enabling mod_status and restarting Apache, you will be able to see your status page at http://<YOUR_DOMAIN>/server-status.

Apache processes and threads architecture

Latest Apache versions implement hybrid multi-threaded and multi-process models to server requests. This means that you will see multiple Apache processes running and each process contains multiple threads. This allows a great trade-off between efficiency and stability.

Apache always maintains a number of idle (spare) workers, single server threads that process the requests, across all the processes as this allows it to immediately assign a request to a thread for processing, without the need to spawn a thread, which would heavily increase processing latency. Workers that are already processing requests are called busy workers. Depending on the number of idle workers Apache is able to fork or kill processes. Under normal conditions the number of idle workers should be more or less stable thanks to Apache self-regulation.

The way Apache forks processes and threads is defined by a particular Multi-Processing Module (MPM) implementation. This module is also responsible for binding ports, accepting connections, and dispatching it to workers. There are several of them depending on OS with prefork and worker being most popular on Unix OS family. The difference is that prefork doesn’t use threads and preforks all the necessary processes, while worker makes use of both processes and threads. Thus prefork is less memory efficient, but allows more stability in case of non-thread safe applications.

Workers’ configuration

A typical Multi-Processing Module (MPM) configuration looks like this (taken from apache.org):

ServerLimit       16
StartServers      2
MaxClients        150
MinSpareThreads   25
MaxSpareThreads   75
ThreadsPerChild   25
MaxRequestsPerChild  10000

The following is a brief explanation of these configuration directives:

• ServerLimit is a hard limit on the number of active Apache child processes. It should follow this rule ServerLimit >= MaxClients / ThreadsPerChild.
• StartServers is a number of child processes launched initially.
• MaxClients is a very important parameter that sets the maximum number of workers (all threads in all processes), and also sets the limit of the maximum number of client requests that may be served simultaneously. Any connection attempts over the MaxClients limit will normally be queued, up to a specific number guided by ListenBacklog directive. Note that in Apache version 2.4 this directive is renamed to MaxRequestWorkers.
• MinSpareThreads and MaxSpareThreads are the boundaries of idle workers number.
• ThreadsPerChild specifies the fixed number of threads created by each child process.
• MaxRequestsPerChild is the number of served requests (or connections depending on the particular type of MPM in use), after which the child process will die. The purpose of this directive is to fight accidental memory leaks.

Know what workers are doing

Mod_status provides information about what each worker is doing in the form of a scoreboard, which looks like this:

_RRR_RRRRRKR_WR___R_KWW_RRR_RR_RWRWR_R_RWRR_RK__K_RRRRRR__RRRWRR
_RRR__W_K__RR___WR___RW_RRR_WRR__WK_R_RKR__R_RRR_KRWWWRR_RRRW___
________________________________________________________________
_R_____________________________________R________________________
_R______R_________R___R_______________W__________W___K__________
__R______________R_RR______________________R____________________
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
RRK_RR_WR___R____RRR_R_R_R__RR_RRWWW__R__R__RRK_R__R_RWW____R__R
W_RR____RW_RRW____R___RRW__RWR_RR__KRWKR_R___R_WR____R_RRRRR_RKR
................................................................
................................................................
................................................................

Each character has the following meaning:

Idle workers:
"_" Waiting for Connection

Busy workers:
"S" Starting up
"R" Reading Request
"W" Sending Reply
"K" Keepalive (read)
"D" DNS Lookup,
"C" Closing connection
"L" Logging
"G" Gracefully finishing
"I" Idle cleanup of worker

No worker running (but configuration allows it to be started if needed):
 "." Open slot with no current process

Normally, the majority of workers should be in R/W or idle (“_”) state. If you see a big number of workers in other states like “K”, “D” or “L”, there’s likely a corresponding problem with keep-alive settings, DNS resolving or logging.

Sizing Apache to the server

The most important server characteristic in relation to a web server is the amount of RAM. A web server should never swap because it will make request processing latency unacceptable. It is important to understand the maximum number of workers that fit RAM of your server and adjust the MaxClients (or MaxRequestWorkers) directive accordingly. In order to do this you should observe how much RAM is consumed by how many Apache processes under normal conditions. Divide the first by the second and compare it to your total available physical memory to understand how many Apache processes you can have on this server.

Knowing the maximum number of Apache workers per server will in turn give you some insight into the traffic you can serve by the server. You can use this information to anticipate future upgrades of your infrastructure.

Watching busy and idle workers

Watching the number of busy and idle workers is a good, proactive way to find Apache configuration problems early enough.

If in case of peak traffic the number of idle workers approaches or hits zero, this may result in some requests being queued; waiting to be processed by an available worker. Such queued requests must wait for older ones to be processed, which results in lower response times of your website. To improve the situation you should consider increasing MaxClients (or MaxRequestWorkers), which is the limit of simultaneous connections. Be aware that more workers will need more server resources, so if you’re short on them (primarily RAM), then changing MaxClients/MaxRequestWorkers will be counterproductive. In this scenario the only way forward is to solve resources bottleneck – upgrade the server or buy another one and load-balance, move static data away from the server.

Tips and tricks

• If you want your mod_status page to automatically refresh at regular intervals, add ?refresh=X to the end of your URL to refresh every X seconds (e.g. http://<YOUR DOMAIN>/server-status?refresh=5)
• To access the status page in a machine-readable format if ExtendedStatus is enabled: visit (e.g. http://<YOUR DOMAIN>/server-status?auto) → it will generate server-status output more friendly for automatic value parsing
• You can combine both parameters together (e.g. http://<YOUR DOMAIN>/server-status?auto&refresh=1)

192.168.3.156
ServerVersion: Apache/2.4.25 (Debian) mod_fcgid/2.3.9 OpenSSL/1.0.2l
ServerMPM: event
Server Built: 2017-09-19T18:58:57
CurrentTime: Tuesday, 07-May-2019 11:06:21 EEST
RestartTime: Thursday, 21-Feb-2019 14:24:34 EET
ParentServerConfigGeneration: 76
ParentServerMPMGeneration: 75
ServerUptimeSeconds: 6464506
ServerUptime: 74 days 19 hours 41 minutes 46 seconds
Load1: 0.00
Load5: 0.00
Load15: 0.00
Total Accesses: 124115
Total kBytes: 4672936
CPUUser: 4.05
CPUSystem: 2.41
CPUChildrenUser: 0
CPUChildrenSystem: 0
CPULoad: 9.99303e-5
Uptime: 6464506
ReqPerSec: .0191995
BytesPerSec: 740.209
BytesPerReq: 38553.7
BusyWorkers: 1
IdleWorkers: 49
ConnsTotal: 0
ConnsAsyncWriting: 0
ConnsAsyncKeepAlive: 0
ConnsAsyncClosing: 0
Scoreboard: __________W_______________________________________....................................................................................................
TLSSessionCacheStatus
CacheType: SHMCB
CacheSharedMemory: 512000
CacheCurrentEntries: 0
CacheSubcaches: 32
CacheIndexesPerSubcaches: 88
CacheIndexUsage: 0%
CacheUsage: 0%
CacheStoreCount: 0
CacheReplaceCount: 0
CacheExpireCount: 0
CacheDiscardCount: 0
CacheRetrieveHitCount: 0
CacheRetrieveMissCount: 0
CacheRemoveHitCount: 0
CacheRemoveMissCount: 0

Most used metrics for System administrators

• server version
• total accesses
• total kBytes served
• CPU load
• uptime
• requests per sec
• bytes per sec
• bytes per request
• worker statuses

Monitoring with Zabbix

Now let’s see how this data can be obtained and parsed by Zabbix. To reuse the configuration it’s best to create Apache monitoring template. Obtaining the status data is easy with HTTP agent item type. Add it to the template, set URL to user macro {$APACHE.STATUS.URL} so it could be overridden by a host and add ‘auto’ to the query fields. The data will be parsed by dependent items, so the history can be set to 0, unless you want to keep history of the raw data.

Now the somewhat complicated part – data parsing. It’s possible to parse most of the data with regular expressions, however, parsing the worker statuses (scoreboard) is more problematic. That’s why we will use JavaScript preprocessing to convert the Apache status data to JSON format, which can be easily parsed by dependent items to extract specific metrics.

The data will be converted based on the following rules:

The conversion script:

// Convert Apache status to JSON.
var lines = value.split("\n");
var fields = {}, workers = {}, output = {};
 
// Get all "Key: Value" pairs as an object.
for (var i = 0; i < lines.length; i++) {
    var line = lines[i].match(/([A-z0-9 ]+): (.*)/);
    if (line !== null) {
        output[line[1]] = isNaN(line[2]) ? line[2] : Number(line[2]);
    }
}
 
// Parse "Scoreboard" to get worker count.
if (typeof output.Scoreboard === 'string') {
    for (var i = 0; i < output.Scoreboard.length; i++) {
        var char = output.Scoreboard[i];
        workers[char] = workers[char] ? (workers[char] + 1) : 1;
    }
}
 
// Add worker data to the output.
output.Workers = {
    waiting: workers["_"], starting: workers["S"], reading: workers["R"],
    sending: workers["W"], keepalive: workers["K"], dnslookup: workers["D"],
    closing: workers["C"], logging: workers["L"], finishing: workers["G"],
    cleanup: workers["I"], slot: workers["."]
};
 
// Return JSON string.
return JSON.stringify(output);

Now create dependent items to extract metrics from the converted JSON. The items will be similar, having corresponding value type and units. The items will have at least one JSONPath preprocessing step, with ‘Custom on fail’ set to ‘Discard values’ for optional metrics. Check the following table for specific details:

To test it create a host, link the template and add {$APACHE.STATUS.URL} host macro pointing at the apache mod status URL. After a while Zabbix should collect the Apache status data:

The Apache template is attached.

About Andris Zeila

Developer at Zabbix.

View all posts by Andris Zeila