Recently, we updated some servers from CentOS 5.6 to CentOS 6.2. Of course, we carefully monitored the performance of the hosts, so we can make a comparison about the performance. The one big change we applied with the application of Cent OS 6 was switching from ext3 to ext4 (although as yet without TRIM enabled.)
Nothing else changed on the hosts – same hardware (Sun X4170’s, in this case), same disks (Intel SSDs in RAID), same workload, same application version.
Different performance. Better performance, which is good – although the reason as to why is unclear.
This first thing that caught our eye was the amount of time the SSDs were busy was significantly less (The load is on the machine with the dark line to start with – then the same load is transferred to the machine with the green line). This is a plot of the amount of the time the SSD was busy:
Looking at the disk operations on the SSDs both systems are doing the same amount of writes – but the physical disk reads have dropped from about 700 per second to 340 per second. (Gotta love SSDs!)
These systems don’t swap, don’t have InnoDB buffer cache misses, and have the same hardware, application and load in both cases – the only variable is Centos 5 to 6.
So while we can’t explain why there should be less disk reads, presumably some changes in the virtual memory system made .. something.. more efficient.
The important thing is that we were measuring it, so we could see it. That way we know about it – and given that it could have gone the other way and made performance worse, it’s essential to be able to know about the impact changes make to systems. (Of course, we saw this change on non-production systems, before we rolled it out.)
Are you tracking the impacts of the changes you make to your infrastructure?
Sometimes the truth hurts. Well the truth is what we didn’t find at DevOps Days was a throng of adoring fans waiting to throw their undergarments at us. Come to think of it, that would be kind of gross anyway, especially with the DevOps crowd…no disrespect.
What we did find was:
a) our marketing table nestled so close to our competitor’s that…if our tables had been teenagers, we would have sent them to the Principal’s office (see PHOTO below…with competitor’s name shamelessly Photoshopped out and replaced with ours) … and,
b) a lot of companies and DevOps teams that were fairly embedded in their custom-rigged, hard-fought and hard-won monitoring solutions.
In our last blog post we talked about the “suck” factor in monitoring. Well, maybe for some, blessed with sizable IT budgets and IT brains, monitoring doesn’t suck so bad at all. In fact maybe for those who take pride in their ability to cobble together a patchwork of complex solutions into one grand “comprehensive” solution, it’s sort of a way of life… a job within a job, a golden chalice, a worthy opponent for any Real Mensa up to the task.
When I was a kid I entered a Soapbox Derby – a racing event where the entrants spend the better part of a year (usually with their dads) making, honing, tweaking, and polishing their own motorless downhill race cars. Well I was new in town and my dad was busy with a new job, so I saved up and bought a Soapbox Derby Car from an enticing ad in the back of Popular Mechanics. The car was amazing. It was beautiful, took me fifteen minutes to put together, and with very little time, effort, or expense I placed an easy second in the popular Derby out of more than three dozen entrants. I loved it.
When, on the trophy stand, I told everyone I’d bought the car, they called an emergency meeting and, despite having no written rule to back up their judgement…took the trophy right out of my hands and disqualified me from the race. My car was arguably better, faster, sleeker and more attractive than most of the others in the field, but I hadn’t spent hundreds of hours and piles of money and put the requisite amount of blood, sweat and tears into it… so it didn’t count.
Sometimes the truth hurts. Well the truth is I just completely made up that story. Sorry, but I was searching for something analogous to what we didn’t find at DevOps Days and that fake memory seemed to kind of fit. It seemed more rich (and fun) than just coming straight out and saying, “When I was out last week I went to DevOps Days – an event where the participants spend a good part of their year (usually with their team) searching, honing and tweaking a multitude of products like Nagios, Cacti, collectd + graphite + pnp4nagios, Muni, etc. etc. to create their own monitoring solution…” and so on.
Plus, admit it, it conjured up a nice little twinge of boyhood nostalgia for a few seconds, didn’t it? Oh well, it did for me. It also caused me to realize what to do with the rest of this quarter’s marketing & event budget – we’re taking out a full page ad in the back of Popular Mechanics.
Performance monitoring for all your infrastructure & applications. In minutes, not hours.
Questions? Call Us!
(888) 415-6442 or +1 (805)-617-3884