Back to the issue!  This guy must have got bitten at some point or another with interface flapping because he needed to be alerted if an interface went down and back up again within the minute that we poll it.  I get it.  As a former admin myself, I know we all have that one quirky thing (ok more like 20 things) that happened to us one time when we got bit bad and we vow that we will never get bit by it again.  This type of attitude is what makes a good admin.  I think this was his one thing.  This one thing happened to be an alert that we didn’t provide by default and he definitely wasn’t going to trust our product if he couldn’t get it to provide this alert.  With his other product he accomplished it by collecting some trap info and doing some calculation which in turn let him know that the interface was flapping.  It sounded complicated and I give him kudos for making the most out of what he has.  I knew we could help him setup this same solution, but at LogicMonitor we prefer not to trust traps.  We would much rather poll snmp counters because traps tend to get lost in transit especially during times of duress, not to mention the configuration headache.  Anywho, I digress, what I didn’t know is how we could help him to make his solution better and more reliable while applying it to every interface he planned to monitor.  So I posed the problem to our support department (of which I am also a member).  Two days later I got a response from one of our smartest engineers a.k.a. “that tech Steve”.  He suggested that we collect the counter for the interface which displays the uptime of the last status change for the interface, but monitor it for changes, not for the time.  Genius!  We could now report alerts on if that value changes.  This is also a value that we can poll rather than catch a trap that is potentially lost in the ether and for good measure I graphed it.  Awesome, now we have ammo to use during the takeaway.

This scenario is one good example where we didn’t provide something by default but because of our flexible framework and expert support we were able to easily provide a solution quickly.  Consequently our potential customer came back with an additional problem he was hoping to see us solve before making a final decision on LogicMonitor.  We were able to get him a solution quickly again and are hoping that this will help close the deal as it has many times before.

Where was LogicMonitor when I was a Network Admin?

Cevin

Someone asked if that abstraction and inefficiency matters:  as in most operational issues, it matters only if it matters.  By which I mean if you are writing a system that is lightly used, or is on powerful hardware – it may not matter at all. But if you suddenly have an increased workload, it may matter a lot. (See the early occurrences of Twitter’s fail whale and RoR scaling.)

Then the question was asked, how can you know whether you are improving things when you change code?  Trend it, of course. You probably know what will constrain your application performance. (If not, you need better monitoring.)  For many sites, an obvious constraint is likely to be database queries per second.  So plot database queries per web request over time

For example, at LogicMonitor, we track database activity, correlated with monitoring data being reported back from customers, with a custom dashboard graph defined like the below. It simply divides Mysql questions per second by the number of datasets being reported back per second:

And the result is:

So it’s clear that in early October there was a code change that resulted in about a 50% reduction in the number of database lookups per reported dataset. Then another change that made things worse, that was reverted.

Graphs like these are essential for tracking the performance implications of code changes – given that this server has a constantly varying load, it’s harder to just look at the MySQL graphs – there may be a large change in Db Questions, but caused by changes to load, not code. By relating the load driver (web requests, or reported data) to the constraint (DB queries in this case), you can assess just the code impacts, not the load.

And the beauty of this approach is it lets you tie together your real inputs to various constraints.  You could chart:

• web requests to web server CPU
• web requests to DB operations (i.e. like relating to questions, but only questions that the query cache could not answer).
• web requests to NetApp disk operations for the volume holding the main DB
• any primary input to any constraint you can think of.

You’ll see if you’re code is making your requests more or less efficient, and thus impacting your scalability. You’ll see if infrastructure changes (deploying memcached, say) are making significant differences.

If you want to scale, and do it without wasting a ton of money on hardware, and have real actionable information to take to your development teams to help improve code – this approach is a great step.

And of course, if you’re a LogicMonitor customer, and want help setting up this kind of trending or dashboards – we’ll work with you to set  it up, free.

For some SaaS companies, this resulted in many hours of downtime. For others, there was a short impact. What was the difference?

Time to Notification

One of the main differentiators was where monitoring was running.  Some companies ran their own monitoring within EC2 – so when their instances were rebooted, so was their monitoring. If it did not automatically recover, they had several hours of outage before they even knew they had an issue. (As their monitoring was down, they had nothing to alert them to the rest of their infrastructure being down too.)

Time to Remediation

Once people knew their servers were down, those with monitoring hosted at EC2 knew they had an issue, but were blind as to the status of their other systems until they recovered their monitoring systems.  In some cases we heard about, this added several hours to the recovery time.  Those using monitoring as a service had immediate visibility into what systems and services had recovered, and which had not, and needed their attention – then they could quickly focus on recovering their databases, or redis systems, or whatever was needed.

The cloud or your datacenter?

Of course, exactly the same issues apply to outages of your own datacenter.  Even the best run datacenters can lose power or network connectivity. It should never happen with A/B power and redundant networks – but it does.  Having Monitoring as a service is just as important whether you have one datacenter (which is analogous to an EC2 region) or multiple.  You want your monitoring outside all your datacenters (otherwise no doubt the one hosting the monitoring will be the one that fails, following the law of toast falling jam side down).

And you want your monitoring to be available immediately, as soon as the power or network recovers, so you can know what to focus on to restore service.

Organize to speed remediation

Your infrastructure has dependencies. No point in trying to bring up your database if its data is stored on a NetApp that is in a cluster failure state.  So have a view in your monitoring that lets you see things by functional groups, and lets you assess whether the prerequisite systems are up.

Practice

Practice, as the saying goes, makes perfect.  Run DR drills.  Spin up a cloud based replica, and rudely shut it down. Make sure you know the order of your system’s dependencies.  See how long it takes you to recover. Get used to using your monitoring to guide you in what to address next.

Now imagine how long it would take to recover without monitoring visibility.

Key takeaway: Move your monitoring away from a premise based system, onto Monitoring as a Service, before your next datacenter or cloud impacting event.

The Tomcat process on http-443  on prod4 now has 96.2 %
of the max configured threads in the busy state.

These were SMS alerts, as that was close enough to exhausting the available threads to warrant waking someone up if needed.

The other alert we got was that Tomcat was taking an unusual time to process requests, as seen in this graph:

Normally we process requests in about 7ms, so 40 ms is a significant degradation.

Nothing was obviously wrong – our internal metrics showed the system processing no more data feeds, nor an unusual number of requests, nor were the disks or CPU any higher utilized than normal.

But as we trend all sorts of metrics, one did jump out, even though it was not alerting:

Turns out there was some minor packet loss between this datacenter and some customers in an upstream ISP. Nothing enough to trigger any network alarms. No loss or discards on our hosts or networks. But enough that some connections to Tomcat were subject to some loss, which meant re-transmissions, which meant connections were taking longer to complete than usual, which meant the server had to deal with more concurrent connections simultaneously, and was in danger of running out.

So, the simple fix: increase the supported max connections in Tomcat. We have plenty of resources available. You can see the max threads was increased at 16:48 to deal with the longer sessions, giving a bit of headroom. (It was a conservative increase, as it hadn’t been  thoroughly tested – we’ll be testing and rolling out bigger max limits soon, just to give more space.)

That allowed Tomcat enough spare threads to remove our alerts and any danger of refusing connections, and soon enough the upstream ISP resolved the loss issue, so session response time – and thus active threads – dropped to normal for this server.

Without LogicMonitor watching our own servers – well, we probably would have been ignorant of Tomcat getting close to exhausting the maximum available threads, and not alerted at all. Which may have led to a more peaceful Christmas Eve, but may have led to angry customers as they couldn’t have accessed their monitoring, or reported gaps in data after the fact – when we would have had no idea what the issue was, or how to prevent it.  Even if we had been alerted by customers while the issue occurred, without the TCP retransmissions indicating what the underlying cause was – we’d still have been in the dark.

Just an example of why we call LogicMonitor disruptive InTelligence.  With LogicMonitor, we avoided an outage, and had the information to diagnose the issue within a few minutes, which let us get back to family time.  That’s disruptive – in the best way.

Happy Holidays.

They provide migration of their customers’ physical servers onto VMware infrastructure – either the customer’s own VMWare infrastructure, or the MSP’s own infrastructure if the customer is using their hosted offering.

One of the issues they had faced in the past was right-sizing the infrastructure that they were to migrate onto. If they were migrating 100 servers onto a VMWare infrastructure with NetApp storage – how much memory would be needed? How much CPU resources? How many disk IOps (input/output operations/second) would be needed to support the combined servers’ workload?
Getting the combined metrics from all these servers, covering peak and average usage, was something that used to take them a long time, and be done very imperfectly.

But they use LogicMonitor’s flexible graphs to make this a simple process. They add the 100 servers to monitoring (if they are not already monitored – a netscan or import from a script makes this easy). They create 3 flexible graphs, that show the aggregate of each of the three statistics (broken down by servers so they can see the big consumers, or just in the aggregate, as they prefer). This takes about a minute, using wildcards. Come back in a day, or a week, and they have their maximum workload that they need the infrastructure to support.

Graph of all windows systems disk IOps

In the above summary graph of 84 servers, you can see that the combined disk IO’s (read + write) of all logical disks on all volumes on all servers peaks at around 2000 IO operations per second (at least for this day.)

Knowing this, it makes it easy to size the performance of the NetApp that these servers will be migrated onto. Without this data, it becomes a guessing game, which can result in over spending on capital, or underspending and poor performance.

So just another way LogicMonitor can save you time, or money.

I found this a bit dismaying at first, as we try to engage in sustainable actions both personally (we have solar panels, two Prius cars, our own chickens – the typical urban hippie) and LogicMonitor as a corporate entity (we recycle, encourage non-car transport, etc). I didn’t like to think that our servers were undoing all the other environmental actions we were taking.

But on reflection, I realized that LogicMonitor is an environmental net positive. Our servers are not only fairly energy efficient (using SSDs, which use about half the energy of rotational disks), but they are very heavily leveraged. Each of our servers is collectively replacing about 100 individual servers that our customers would have deployed if they ran their own monitoring servers. (More, if they ran redundant monitoring servers like we do.)

So from that point of view, LogicMonitor is taking hundreds of SUV’s off the road. All while freeing them from having to run, patch, and backup those servers, or extend their software to deal with new things like MongoDB monitoring, and so on – we do all that for them. So that means they can get out of the office earlier, and avoid the peak traffic times – saving more driving impact. Unless they have something else to do at work.

Happy thanksgiving.

The article mentions that “enterprises have confused (with vendor help) the notion of monitoring the resources that an application uses with its performance”.  The way I put it in my lecture was that:

• Systems are limited by Disk IO, memory and less commonly CPU and network.
• Users dont care about Disk/memory/CPU/network… They care about web pages, and speed.

So… how to tie one to the other?

Monitor both.

Monitor what users care about (page load times, response per request, etc)



Also monitor all the limiting resources (CPU, Disk IO – or more importantly what percentage of the time a drive is busy, network, memory):

And monitor the performance of the systems that affect the limiting resources:

So while monitoring InnoDB file sytem reads does not tell you anything that an end user cares about, if your monitoring of Tomcat request time shows that users are experiencing poor performance, and your logical drives are suddenly 100% busy and request service time increasing, it’s good to know why that is. It may be because of InnoDB buffer misses, or it may be because of something else – but having this intermediate data will drastically reduce your time to correct the issue that users care about – response time.

Another point to note: the “user” in the phrase “monitor what users care about” may not be a human.  If a server is a memcached server – the users for this server are web servers, who care about memcached response time, availability and hit rates.  So on this class of machines, that is the thing to monitor to determine if the service is meeting the needs of users.

In short, for every machine, identify the “thing(s) to care about” for it; monitor those things; monitor the constrained resources; and monitor all aspects of the systems on that server that inmpact the constrained resources.

In adding some more Exchange Monitoring we ran into some issues, and solutions, that may help others.  Some things in recent Exchange versions can only be monitored by Powershell. (Perfmon, WMI, Powershell, all needed for different versions of Exchange…. I wish they’d make up their mind…)

So the first issue was that Powershell scripts, when called from a LogicMonitor agent, never returned. This wasn’t too hard – simply pass the parameter -inputformat with the (undocumented) option “none”, and the agent can successfully run Powershell commands:

powershell -inputformat none dbstatus.ps1

(Why? The Microsoft.PowerShell.ConsoleHost class constructs a M.PS.WrappedDeserializer passing the STDIN TextReader as one of the parameters. By default, the WrappedDeserializer will call ReadLine() on this STDIN TextReader and wait indefinitely, effectively hanging PowerShell and the calling process. That’s why.)

So past that hurdle, but the next one:
>> powershell -inputformat none dbstatus.ps1
Add-PSSnapin : No snap-ins have been registered for Windows PowerShell version 2.

Yet running the exact same command from the command shell on the host running the agent resulted in the output we were expecting. And we could see the Exchange snap in, called by the Powershell script, was correctly registered, and in fact worked fine.

But.. our agent was running on a 32 bit JVM and Exchange 2010 (in our lab, at least) is installed on 64 bit Windows. The Powershell snap in was only visible when powershell was started from a 64 bit app. When I started powershell from the cmd.exe in SysWOW64, I got the same error about missing snap-ins as our agent reported.

The solution – it doesn’t matter that our agent was installed as a 32 bit app, in Program files (x86). What mattered was that the Java virtual machine launched by the agent, that ultimately launched Powershell, be a 64 bit JVM, not the default 32 bit JVM installed from Java.com. (At least, a 32 bit JVM is the default when you browse to Java.com with a 32 bit browser.)

So, running the LogicMonitor agent with a 64 bit JVM, and Powershell started with “-inputformat none” gives us full access to Powershell output and all its snap ins, so expect some datasources released very shortly to take advantage of that.

They contacted us two days before the hurricane, and were completely monitoring all their infrastructure that same day, with the alerts coming from our datacenters, not theirs.

They and their infrastructure came through Irene unscathed – and they knew that they did, because they used a hosted monitoring system. Had they used a premise based monitoring system, they would not have known if  their lack of alerts was because their monitoring system had been flooded or cut off from the internet.

So while enhanced disaster preparedness is not usually the way we sell our value, it’s certainly a nice bonus.

i.e. we assume you want to monitor everything, always. (You can of course turn off monitoring or alerting for specific groups, hosts or objects.)  This serves us well almost always – we will detect a new volume on your NetApp once you create it, and start monitoring it for read and write latency, number and type of operations, etc – this means that when you have an issue on that volume (or other groups are blaming storage performance for an issue), you already have the monitoring of all sorts of metrics in place before the issue – so you have the data and alerts to know whether the storage was or was not the issue.

However, we have found some cases where this doesn’t work so well. We have been monitoring NetApp disk performance by default, too, tracking the number of operations and the busy time for each disk.  However, on customers with larger NetApps, there are often hundreds of disks, each of which we would monitor via API queries.  This is useful for identifying when disks need to be rebalanced (if the top 10 and bottom 10 disks by busy time are wildly different.) And while we only monitor the performance of a disk every 5 minutes (as opposed to volumes and PAM cards and things that are monitored more frequently), this apparently overloads the API subsystem of NetApp devices.

We’d see that when we’d restart the collection process, and the only monitoring by the API was for the volume performance, things worked great – the response to an API request from a NetApp was around 100 ms.

When the disk requests started getting added in (and we stagger and skew the requests, so they are not all hitting at once) – the API response time for a single query climbed up to 40 seconds.

This started causing a backlog of monitoring, and was causing data to be missed in the more important volume performance metrics.

So… while we’ll open a case with NetApp, in the interim, we’ll probably disable the monitoring of physical disk performance by default to avoid this issue.