Single points of failure
Monitoring, alongside backups and timely updates, is one of the fundamental principles of operational security. These areas are tightly connected, and monitoring is no exception.
A simple example: if there’s a monitoring service that watches all the resources you need, reports as required and does it well, it’s still no guarantee you will be alerted to whatever wrong could happen to your assets. Imagine connectivity loss, or monitoring service failure (everything breaks sooner or later).
Apart from backing up data, one should also back up every component used by monitoring service – including monitoring service itself.
Stopped monitoring service
Unless measures have been taken to ensure IPNetwork monitoring service to auto-start after failure, its absence may be incorrectly taken for “OK” state for all the monitors.
One of solution is to have another IPNetwork Monitor installation to monitor the one in question (and vice versa). The second installation may only monitor the main one, and be able to notify of its shutdown.
This situation is a typical pitfall; make sure you watch the monitoring service itself and render its absence a critical failure.
No valid SMTP server
By default, IPNetwork default settings do not contain valid SMTP server and proper credentials to send email. As a result, no notifications are sent (apart from default pop-ups, in case user sees the desktop of the system running IPNetwork).
Thus, we recommend that after IPNetwork Monitor has been set up and started for the first time, you configure at least one primary SMTP server and verify it using “Send test email alert”. The test window shows the SMTP conversation log, which helps diagnose authentication, connection, or delivery problems. After that, configuring the alternate SMTP server is strongly recommended, because it is used automatically if the primary server fails to send a message.
Connectivity loss
Regardless of how reliable are SMTP server(s), if there’s no connectivity at the system running IPNetwork Monitor, notifications might be not able to reach their destinations.
One backup option is to configure SMS delivery via GSM modem, which provides an additional notification path independent from the primary Internet connection. Push notifications to mobile devices can also be used, but they still require Internet connectivity. Please note that GSM-based messaging may incur extra mobile operator charges.
Setting up an alternate Internet connection, provided that outbound traffic is routed through it automatically when the primary link fails, is another useful solution outside IPNetwork Monitor itself.
Human errors
The $AdminMail, primary email address used by IPNetwork in variety of abnormal situations (set in “Settings > Email Settings”), might receive many messages from monitoring setup. If those are filtered out/otherwise automatically moved out of inbox, an important notification may be missed. It is recommended to create custom messages for email alerts, modifying subject line, so it can’t be confused with any other message – and ensure that messages related to “Down” state for critical monitors are paid attention to.
Another typical mistake is disabling alerting for maintenance and then forgetting to enable it again. In current versions, alerting can be disabled temporarily with automatic re-enable after a selected time interval, a warning indicator is shown while alerting is disabled, and regular reminder notifications while alerting is disabled can be configured on the Alerts settings page.
We would recommend making IPNetwork notifications as “noisy” as possible in the beginning; after you understand how well they are noticeable, you might wish to draw your attention to critical monitors, wherever marking all the rest as less important.
Do you know of any other typical human errors or other monitoring setup flaws preventing them from working as expected? We would be glad to know – feel free to email us a message.