Monitoring is a huge portion of my job and I’m perpetually seeking the utopian monitoring system. 

What makes for a perfect monitoring system you ask?  I’m glad you asked!  Offhand I would describe it as such:

1. Holistic — watches every conceivable metric, preferably from multiple vantage points
2. Corollary — distills data to provide concise, insightful results
3. Correct — suppresses false alarms through the use of intelligent collection, marked dependencies, etc

In my experience the base software can get you only so far and requires much tuning and adaptation to particular environments.

I have evaluated systems such as SolarWinds, Nagios, Zabbix, Shinken, and many more.  I check every few years to see if I can find someone who has built a Better Mousetrap but I always come back to Icinga.

As an offshoot of Nagios, Icinga is backward compatible with most (all?) Nagios monitoring agents (“plugins”).  Of all the plugins I have authored or modified, I have only uploaded a couple of them to the Nagios Exchange (https://exchange.nagios.org/directory/Owner/brentparish/1).

Perhaps eventually I will also include the whole collection here.

The most significant changes I made were to the “standard” plugins such as RAM, CPU, system load, disk space, etc.  My version of the plugin leverages SNMP first for lightweight speed, falling back to an SSH connection if SNMP is unavailable.