Zabbix is a very complex software that takes weeks to fully understand. One interesting,complex and badly documented feature is escalations. They define what should happen when a certain event occurs. Like waking up the system administrator or doing some automatic emergency cleanup tasks.
The “why” of escalations
One of the reasons I suggest you use escalations instead of just standard actions are delayed notifications. If you are using trigger dependencies then chances are that in case of a network problem still get spammed with alerts until the dependencies step in. I recommend reading my wiki article on delayed notifications. (Seems that zabbix.com has removed my nice wiki article. I should have copied it.)
Further scenarios could be:
- if the SMS on your cell phone does not wake you up then a little later try to wake your backup admin or boss
- get an instant alert message via Jabber but if you did not read it send an additional SMS five minutes later
- if a certain process dies frequently then let Zabbix try to restart the service three times until it alerts you
- repeat an alert every 10 minutes in case you missed the first one
Configuring escalations
Take a look at this screenshot that I will use to explain how escalations work (taken from my wiki article I mentioned above)
There is a lot of information in this configuration screen. To enable escalations click on the checkbox in the upper left corner. Then the top right box with the escalation plan (Action operations) will appear.
The escalation plan
Here you will see what action is run when. As soon as the Action conditions defined on the lower left are met then the escalation plan will start to run – beginning at step 0. Every “period” number of seconds Zabbix advances to the next escalation step. In my example the period is 120 seconds. So two minutes after the action started step 1 runs. After another 120 seconds step 2 is run and so on. You can change the period to a new default value during a step but I find it confusing and would advise against it. To make things simple I recommend to set the period=60 so that the steps correspond to minutes.
Steps
If you add or edit an operation in the list of operations on the top right then you see the “From” and “To” fields. This is the range of steps that a certain action should be run at. In my example it’s from 2 to 2. Which means: this operation is supposed to be run at step 2 only. Other examples:
- from 0 to 2: run the operation at step 0 (immediately), step 1 (after 1 x period = 120 seconds) and step 2 (after 2 x period = 240 seconds)
- from 1 to 0: run the oepration at step 1 (after 1 x period = 120 seconds) and until infinity every “period” (=120 seconds)
- from 2 to 2: run the operation at step 2 only (after 2 x period = 240 seconds)
Only if not yet acknowleged
An extra goodie is the condition on the lower right. It means that this action is only run if the event has not been acknowledged yet. If anyone acknowledges the problem within the first 120 seconds then the condition would not match and nobody would get a notification. This is a workflow I can definitely recommend.
Stupid caveat
Caveat: you must always add an Action condition “Trigger value = PROBLEM” to the bottom left. Otherwise the escalation plan would not only start when this action starts but also after the recovery from the original network/server problem. This would lead to a delayed recovery message which is totally useless. (I have not yet understood why this is not an automatic/implicit condition.)
Recovery messages
Regarding the recovery: one the original problem that started the action is remedied then this is called recovery. If you tick the “Recovery message” checkbox then Zabbix sends out recovery messages. But only to the users that have been informed of the problem. If the escalation plan would have informed your boss 10 minutes later but you have fixed the problem then your boss will not get a (confusing) recovery message. Zabbix does the right thing.