By which I mean that, unless you are a kernel developer or some other individual with esoteric purposes, having Linux up and running is not the point of your servers.  Your servers are there to DO something, whether that’s to serve web pages, answer database requests, or provide the best hosted monitoring service.

So…what do you monitor?

Well, in the case of a server whose main point is to be part of a cluster of Apache web servers – monitor Apache. Check that it’s running; returning the content you expect it to return; the number of busy server threads, throughput per second, etc.

Congratulations, you’ve monitored the most important thing about that server. But that doesn’t mean you are done. This is where the Linux monitoring comes in. If your server is to serve http requests with Apache, it needs CPU resources. It needs memory. It definitely should not be swapping. If there are any issues to resolve, that will be much easier if NTP has synch’d the server’s time.  You’ll want to know about hard drive failures; power supply issues, temperature.  You’ll want to know if the SSL certificate for your web server is approaching expiration (and you’ll want to know that a few weeks in advance.) You’ll want to know if file systems are filling up; if physical disks are approaching their maximum in terms of IO operations per second, increasing CPU wait time. If interfaces are having errors, drops or discards.  If the mail sent by the server is bouncing, being delivered, or rejected.

All these aspects are important to monitor so that you will know if your server faces any issues that will prevent it from fulfilling its destiny – of serving web pages.

And of course as this server is part of a cluster, you want to graph its throughput with the other servers, so you can see overall performance, and also compare different servers to each other:

And this is just for one server, with one application. If your server runs a java webserver, or a database, or memcached – there’s suddenly a lot more things to monitor, just to get “Linux Monitoring”.

So what’s the point? I’ve said it before, and will say it again, but automation is essential. If your monitoring is cumbersome to get new servers monitored completely, guess what – you won’t monitor them completely. Which will increase your outages, and increase time to resolution. The more you monitor, the quicker you can resolve issues, as issues are often not where you’d expect.

A quick query shows that a typical server in our lab has 382 datapoints monitored on it after a default install, just by entering its hostname into LogicMonitor. If we had to configure the monitoring ourselves, the number would more likely approach 1 for lab machines (“does it ping?”), especially as we fire up new virtual machines in the lab regularly. But that would mean our developers would miss all sorts of MySQL and Tomcat metrics that they routinely use to improve the next release of our product. And our Ops team would not have good history of performance metrics, like average request service time, so they can identify when issues were introduced on the QA servers. And QA would be held up as our QA systems would not be as highly available.

So for us, automation has both decreased work, and increased productivity in all sorts of unexpected ways. How has it benefited you? Or how could it?

LogicMonitor automates your monitoring setup.

It’s a quick (about 10 minutes) look at why it’s particularly important to have a unified monitoring system that covers virtualization infrastructure, the guest OS’s and the applications on the guest OSs, and the storage, all in one monitoring system – otherwise you can end up with people without sufficient information chasing all sorts of problems that would be immediately identifiable with a unified system.

Check it out, and if you have any questions, feel free to email us at info@ logicmonitor.com

There were quite a few questions at the end of the webinar, but I didn’t include them in the recording, else we would have had to post it somewhere else other than the Worlds Shortest Webinars.

Make sure you are loading the mod_status module.

If you are using a version of Apache that was installed by your OS’s package manager, there are OS specific ways to enable modules.

For Ubuntu/Debian:

/usr/sbin/a2enmod status

For Redhat/Centos: Just uncomment the line:

LoadModule status_module modules/mod_status.so

in /etc/httpd/conf/httpd.conf
For Suse derivatives:
add “status” to the list of modules on the line starting with APACHE_MODULES= in /etc/sysconfig/apache2

Configure the Mod_status module

You want the following to be loaded in your apache configuration files.

ExtendedStatus On
<Location /server-status>
 SetHandler server-status
 Order deny,allow
 Deny from all
#Add LogicMonitor agent addresses here
 Allow from localhost 192.168.10.10
</Location>

Where you set that configuration also changes depending on your Linux distribution.
/etc/apache2/mods-available/status.conf on Ubuntu/Debian
/etc/httpd/conf/httpd.conf on Redhat/CentOs
/etc/apache2/mod_status.conf on OpenSuse/SLES
Finally, restart apache using your OS startup script ( /etc/init.d/httpd restart or /etc/init.d/apache2 restart). Note that using the OS startup script is often necessary to allow the OS specific script files to assemble the final apache config. Sending apache signals, or using apache2ctl, does not do this.

3. Watch the monitoring happen.

If you are using LogicMonitor’s Apache monitoring, then you’re done.  LogicMonitor will automatically detect the Apache web server, and apply appropriate monitoring and alerting, as well as alerting and graphing on the rest of the system, so you can correlate CPU, interface and disk load to Apache load.

One thing you may want to customize is your dashboards – add a widget that collects all Apache requests/second, from all hosts, or all production hosts, and aggregates them into a single graph.  Using LogicMonitor’s flexible graphs, the graph will automatically include new servers as you add them.

Want to make your Apache monitoring simple? Check out LogicMonitor for monitoring that automates your monitoring setup. Free Trial

News reports detail allegations that some of the alarms on the failed rig were “inhibited”, because “they did not want people to wake up at 3 a.m. due to false alarms”.  The rig’s operator responded, “Repeated false alarms increase risk and decrease rig safety”.

We’ve said it before and we’ll probably say it again, but alert overload is dangerous, no matter what the occupation. Even if your field of responsibility is for ensuring the performance and availability of network switch monitoring, you need to ensure all alerts are meaningful.

This requires two things:

• processes that are followed (reviewing alerts, reports on top alert sources, etc)
• tools that allow you tune and route your alerts.  (One area I’d include in this is powerful alert calculations – it is often helpful to be able to express alerts such as “Warn if discards is more than 0.5% of traffic – but only if traffic is greater than 1 Mbps inbound”.)

If your monitoring system can’t do this kind of alerting, and easily let you tune it on group, host or instance level, you’re either wasting time dealing with false alerts, or have “inhibited” the alarms on your own oil rig. And we’ve all seen where that leads….

Want to make your monitoring less lethal?  Check out LogicMonitor for monitoring that automates your monitoring setup, and lets you only get the alerts that matter. Free Trial

Twitter is addressing issues, including their monitoring. (As we’ve noted before, if monitoring did not alert about an incident at the earliest possible time, the incident should never be closed until the monitoring is improved to warn of the issue in advance.)

In our view, the best monitoring approach is to have your monitoring software assume everything should be monitored to production levels, automatically, and not need your Ops team to tell it.

Repel your own flying whales by using data center monitoring automation as much as possible.  Let the humans go play soccer, or blow their vuvuzelas!

Putting processes will likely be the key to your monitoring deployment succeeding or failing, and is another area that a SaaS monitoring service is likely to be superior to a premise based system.  A SaaS service is invested in your continual use and satisfaction of monitoring – they don’t get the money up front.  At LogicMonitor, at least, we help our customers with the whole implementation, including processes.

Doesn’t all monitoring matter?  Well, yes it does.  But to badly paraphrase George Orwell, all monitoring matters, but some monitorings are more mattering than others.

What makes monitoring matter?  It’s whatever monitoring reduces your outages, reduces issues of unacceptable performance, or reduces time to resolution of these issues and outages.

We like to think of monitoring as similar to Maslow’s hierarchy – there are base levels that must be met first, but the higher up you go, the better.

1. The base level of monitoring is “is my host/site alive?”  Everyone needs this (but not everyone has it.) That gives you reactive monitoring.
2. “Is my host going to keep working in the near term?”  This means alerts about disks filling up, or swap space and memory used.  This helps with reducing some outages.
3. “How is my host performing?” Things like CPU load, and rate of swapping.  Alerts on these metrics warn of impending performance issues, that can be addressed.
4. “How is my application performing?”  A measurement of representational application performance. This may be things such as database transaction time, time for a web site to process a request, or even, for a storage array, latency of write requests. It’s really a more fundamental level of monitoring than level 3 – an alert about CPU load, in level 3, may not indicate anything amiss – in the case of NetApp monitoring, it could be a weekly raid scrub, and request latency, which is what really matters, is not affected at all. However, we rank it higher than level 3, simply as it’s easier (and more common) to monitor general purpose metrics (such as CPU) than application specific performance metrics (such as database transaction times.)
5. “Why is my application performing as it is?” This is where monitoring really starts to matter.  The more data you collect and trend, the better your monitoring system will be at helping you quickly identify and resolve issues. (Of course, the more data you graph, the better your monitoring system need to be at presenting and organizing that data in a meaningful way. It’s easy to show 4 graphs of a system, but when you may have 120 different graphs for one system, and you need to quickly scan them to see correlations – the UI challenges get more interesting.) In a database, an alert about response time is clearly significant, but doesn’t tell you what the cause is.  But if the monitoring system can quickly show you your database’s sequential table scans jumped after the last software release, or that latency of a storage array volume is high because another volume sharing the same physical disks is experiencing an abnormal rate of IO operations, you will be able to resolve your issues much quicker.
6. “How do I fix my issue?” This is the peak level, where the monitoring system not only shows all the data, but presents directions on how to resolve any issues.  LogicMonitor can do this in some cases (for example, using data from the number of select operations, query cache hits and cache prunes due to low memory to recommend enlarging, reducing or disabling the MySQL query cache).  But this is a much harder issue to generalize, especially across systems that interact. But we’re always improving.

How far up the hierarchy is your monitoring?  If you don’t have a wealth of data about all aspects of your system, that you can trend in real time and look at historically, it’s almost certain that your outages and performance issues are both too frequent and too long.

When the only alternatives were costly enterprise-class monitoring solutions, sweating it out with open source was understandable. But now that there are affordable tools that automate configuration and give you everything you need in 30 minutes, insisting on building your own doesn’t seem wise (especially in this era of understaffed data centers.) At the root of this DIY mentality is pride. With so many open source options available, Techies probably feel some sense of shame or embarrassment going to an IT director and asking for tools that cost money.

I’d suggest a better source of pride is being able to spend time on tasks that add value to the enterprise – writing Puppet scripts that automate machine and software deployments, and so greatly reduce the time to spin up machines; investigate cloud usage options; correlate resource expenses with revenue per business unit.  There are a lot of things that should be done in any enterprise, that are not because of lack of time.  A good systems administrator’s time is very valuable – much more valuable than going through a MIB to figure out which item is important to monitor.

And no matter how good a systems administrator you are, monitoring is not going to be your top priority (nor should it be).  You’ll get monitoring going “good enough” – but there will be lots of cases that it failed to alert on, when a comprehensive monitoring system would have.  Then after every outage, you’ll have to go back and extend the monitoring, adding in metrics that could have helped predict the specific case.

So given the cost of your time; the more in depth monitoring that you get immediately with LogicMonitor (a typical Nagios implementation may monitor 10 metrics on a linux host; a typical LogicMonitor deployment will monitor over 100); and the opportunity cost of the things you could be doing to add value with your time, if you weren’t configuring monitoring, then why not use an automated monitoring tool such as LogicMonitor that makes you a better system administrator, and doesn’t require a Fortune 500 budget to implement?

If you’d rather skip the tedious work, but want the peace of mind knowing that your infrastructure is properly monitored, and that you will be alerted of any issues early, it’s perfectly okay to go the automation route. You’ll feel a sense of satisfaction in preventing an outage, whether you wrote the code or not.  And your CFO may even thank you for spending the money.

One common observation that came up was that the conference was much smaller than previous years, both in attendees and exhibitors.  Some people lamented the new ownership of MySQL by Oracle, and the ensuing “corporatization”. I didn’t really see much evidence of that in the conference sessions or topics, however.  From the other exhibitors, however, the feeling was the set of attendees was smaller, but a better qualified set. (From the point of view of people trying to sell them .. something, but also from attendees who wanted to talk to other attendees that had tackled real problems.)  Most attendees did seem to be from not-seat-of-pants enterprises, but real companies with revenue models, and even revenue.  They also had real problems to solve, and were interested in products/solutions that addressed those problems, even if they cost money, as they realised that the costs of not addressing them were generally much higher. (In revenue costs, in staff time, in diverting resources from strategic development to solving already solved problems…)

There was still quite a lot of attendees who’s first question to us at the LogicMonitor booth was “Is it open source”?  But the fact that we are not was not a disqualification from them spending time with us – it just meant that they wanted to know what value we added over open source solutions, that would make it worth giving us money.  Something we could easily answer to (almost) everyone’s satisfaction, after we explained the time savings, the more comprehensive monitoring, the cross platform monitoring, the automation, the auto-aggregation and classification features, etc.

Perhaps MySQL being purchased by Sun, then Oracle, has to some extent meant that the young innovators that gathered around it have gone on to more Open pastures, but it may also mean that the solid base of users remaining and working with it will solve more real world problems, generate more revenue, and also be willing to spend more money to solve their own problems. Which may just lure back more of the wild innovators, once they see the opportunities are actually better than they were – cool things can still be done, but now people will pay them – which makes the doing of cool things sustainable, not just done for love. Nothing wrong with doing cool things for love AND money.

The people involved in SaaS think security is often better in SaaS systems that premise based systems.
Justin Pirie at “The Week in SaaS” (an essential blog for those in SaaS, I think), put it this way:

Reuvan Cohen at Elastic Vapor summarizes his view:

So what’s my opinion? Having managed IT operations for a variety of companies, and worked in SaaS companies, I think I can share a realistic view.
There are (simplistically) two aspects to application security – physical security and application level security.

If you are a small company, and have premise based applications, you probably don’t care much about application level security. The company is small, and everyone will be trusted to some degree. The fact that the application is behind a firewall, with no access from outside, does provide fairly good security. The SaaS advantage here is that small companies do not usually have physically secure premise based servers. They are typically in a small server room (or closet), without much in the way of alarms, 24 hour guards, and all the other touted features of datacenters. And if you can physically access a server, you can get to the data on the server. As a friend of mine, the head of sales at a SaaS property management software company, puts it “No Fortune 500 company would consider putting their servers in your SMB server room. Yet they do have them in the same datacenter as our SaaS servers.”

As a company grows, they will typically get on top of physical security, but then application security raises it’s head, as security sensitive applications will now be restricted to a subset of employees. Many premise based applications (especially open source ones or internally developed ones, it seems) are written without any access control designed in. And once a company reaches any size, the premise based application will need to be accessed by people outside the firewall (remote offices, teleworkers, etc). How is that access to be granted securely, without undermining the whole security premise of “Well, it doesn’t matter if it’s not terribly secure, as no one can access it.”

Yes, you can put reverse proxy firewalls or SSL VPNs to provide some sort of remote access, but now the “simple” choice of premise based software for security is getting more and more complicated (and expensive).

So I think the consensus above is correct – in a company of any size, you are more likely to have less security issues and expense with a SaaS solution than premise based software.
(FYI – LogicMonitor has its servers in Equinix datacenters.)

What are your thoughts?