Look for typical traits of upcoming problems
Security monitoring is a constantly changing area: threats can appear at any moment, and true zero-day issues are difficult to prevent directly.
However, there are typical behavior patterns that can still be detected using familiar tools and monitor types.
Most examples below are related to Unix-like systems, but the same ideas can be applied on other platforms as well, even if the implementation differs — for example, Windows systems are often monitored through WMI or Event Log, while Linux systems commonly use SNMP, SSH, or Syslog.
Detect scanning and probing activity
When a known vulnerability gets known, all the services that can be potentially exploited are getting probed for possible weaknesses. In such cases, proactive logs analysis can provide much information on what is going on.
For example, typically crafted communication with mail servers; increased volume of brute force attacks and so on.
Talking of Web servers, that could result in raised amounts of calling possibly vulnerable URLs. Unless a Web Application Firewall (WAF) is used, watching Web server logs could help to detect a possible problem. If there are repeated attempt to pass data via known URLs, they should be inspected as soon as possible.
In such a case, Syslog monitor can help by turning suspicious log patterns into monitoring events. In larger environments, it is often useful to aggregate those messages first — either through a Syslog relay or through a Linux Remote Network Agent configured to receive Syslog inside the remote network — and then alert only on the patterns that really matter.
Use existing intrusion detection systems (IDS)
There are well-known intrusion detection systems: both passive (should be run to find a possible problem – such as Aide) and active ones (also known as host-based IDS, HIDS), which monitor I/O activity and can raise an alert if something out of pattern is encountered.
Host-based or network-based intrusion detection tools can be connected to IPNetwork Monitor through Syslog or other event-producing workflows, so suspicious activity can be surfaced in real time. Even if the exact nature of a breach is not yet known, event-driven monitors such as Syslog or SNMP Trap can be used to turn unusual activity into immediate alerts.
Passive IDS, such as Aide find changes in the system – i.e. new or changed files in areas not expected to host any changes – and report them. If there are new Cron jobs, new system services etc., they should be paid close attention to. Most known threats such as cryptominers often use self-reinstalling tricks of this kind, to stay active even if their main process has been stopped.
New or unusual process activity
When monitoring your systems, it’s a good idea to keep track of processes that can be using quite a lot of resources, even if for short duration.
Monitoring CPU load is a good starting point for detecting unusual activity, but it is rarely enough on its own. Current IPNetwork Monitor documentation also describes built-in resource checks for memory usage, free disk space, and process statistics over SNMP, SSH, or WMI, which makes it easier to watch for the combination of traits that often appears before a real problem becomes visible to users.
The same type of checks can be applied to higher-than-usual bandwidth usage, unexpected process counts, or other activity that is clearly out of pattern. Where short spikes are common, spike filters are worth using so that a brief anomaly does not immediately turn into a noisy alert.
React to possible threats
As earlier stated, active response is often preferable when it can be implemented safely. If a threat or possible intrusion is found, steps can be taken to isolate the system or make the problem harder to spread. In current IPNetwork Monitor versions, the practical building blocks for that are explicit: local programs or scripts, commands run via SSH, SNMP set actions, and HTTP(S) requests to external automation systems. These can be used to disable an interface, restart a service, call a firewall or orchestration API, or trigger an internal response workflow.
Do you have technique(s) of your own, to prevent and/or handle possible security threats, by means of monitoring software? If yes, please share it with us.