{"id":2202,"date":"2018-10-31T20:13:00","date_gmt":"2018-10-31T20:13:00","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw92\/?p=2202"},"modified":"2018-11-02T10:22:48","modified_gmt":"2018-11-02T10:22:48","slug":"cloudwatch-is-of-the-devil-but-i-must-use-it","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2018\/10\/31\/cloudwatch-is-of-the-devil-but-i-must-use-it\/","title":{"rendered":"CloudWatch Is of the Devil, but I Must Use It"},"content":{"rendered":"

Let’s talk about Amazon CloudWatch.<\/em><\/p>\n

For those fortunate enough to not be stuck in the weeds of Amazon Web
\nServices (AWS), CloudWatch is, and I quote from the official
\nAWS description<\/a>, “a monitoring and
\nmanagement service built for developers, system operators, site reliability
\nengineers (SRE), and IT managers.” This is all well and good, except for the
\npart where there isn’t a single named constituency who enjoys working with
\nthe product. Allow me to dispense some monitoring heresy.<\/p>\n

Better, let me describe this in the context of the 14 Amazon
\nLeadership Principles<\/a> that reportedly guide every decision Amazon makes.
\nWhen you take a hard look at CloudWatch’s complete failure across all
\n14 Leadership Principles, you wonder how this product ever made it out
\nthe door in its current state.<\/p>\n

“Frugality”<\/h3>\n

I’ll start with billing. Normally left for the tail end of articles like
\nthis, the CloudWatch billing paradigm is so terrible, I’m leading with
\nit instead. You get billed per metric, per month. You get billed per
\nthousand metrics you request to view via the API. You get billed per
\ndashboard per month. You get billed per alarm per month. You get charged for
\nlogs based upon data volume ingested, data volume stored and “vended logs”
\nthat get published natively by AWS services on behalf of the customer. And,
\nyou get billed per custom event. All of this can be summed up best as
\n“nobody on the planet understands how your CloudWatch metrics and logs get
\nbilled”, and it leads to scenarios where monitoring vendors can inadvertently
\ncost you thousands of dollars by polling CloudWatch too frequently. When the
\nAWS charges are larger than what you’re paying your monitoring vendor, it’s
\nnot a wonderful feeling.<\/p>\n

“Invent and Simplify”<\/h3>\n

CloudWatch Logs, CloudWatch Events, Custom Metrics, Vended Logs and Custom
\nDashboards all mean different things internally to CloudWatch from what you’d
\nexpect, compared to metrics solutions that actually make some fathomable
\nlevel of sense. There are, thus, multiple services that do very different
\nthings, all operating under the “CloudWatch” moniker. For example, it’s not
\nparticularly intuitive to most people that scheduling a Lambda function to
\ninvoke once an hour requires a custom CloudWatch Event. It feels overly
\ncomplicated, incredibly confusing, and very quickly, you find yourself in a
\nsituation where you’re having to build complex relationships to monitor
\nthings that are themselves far simpler.<\/p>\n

“Think Big”<\/h3>\n

All business people, when asked what they want from a monitoring platform,
\nwill respond with something that resembles “a dashboard” or “a
\nsingle pane of glass view”. CloudWatch offers minutia up the wazoo, but
\nit categorically offers no global view, no green\/yellow\/red status
\nindicator that gives you even a glimmer of the overall health of your site.
\nWant a graph of each core in your instance’s CPU for the past 30
\nseconds? Easy! Want to know if your entire company should be putting out the
\nburning fire that is the current production state of your website? Keep
\nlooking\u2014CloudWatch has nothing to offer you.<\/p>\n

“Insist on the Highest Standards”<\/h3>\n

By its very nature, CloudWatch feels like small thinking. The entire
\nexperience, start to finish, smacks of “what’s the absolute least we
\ncould do and get away with it?” They built their MVP, and then just
\nsorta…stopped, frozen in amber. They created a set of building blocks,
\nexcept they didn’t solve the problem of “how do I monitor my AWS resources?”
\nInstead, it feels like the entire team phoned it in and let a large market
\nof monitoring vendors develop as a result. None of those vendors have the
\nlevel of access to the raw data that CloudWatch does; all of them have built
\nbetter products. You’d think the CloudWatch team would take a clue from
\nthe innovation that’s rapidly happening in this space, but that’d
\nrequire someone to Learn and Be Curious.<\/p>\n

“Are Right, a Lot”<\/h3>\n

Recent data is “eventually consistent”, so you always get graphs like the
\none shown in Figure 1.<\/p>\n

\"CloudWatch<\/p>\n

Figure 1. Example CloudWatch Graph<\/em><\/p>\n

Here in reality, that would be a terrifying thing to see on an accurate
\ndashboard<\/em>\u2014something is obviously very wrong with your site! For better or
\nworse, the “accurate” description doesn’t apply to CloudWatch, and that’s
\njust how your graphs always look. “Your metrics will be eventually
\nconsistent” is very nearly the last thing you want to hear about your
\nmonitoring platform, second only to “what metrics?” This ties directly
\nto…<\/p>\n

“Earn Trust”<\/h3>\n

Let me be very clear here\u2014the real issue isn’t the ingestion problem.
\nAbsolutely every vendor on the planet has the same issue\u2014you can’t
\ndisplay data you don’t have. Where CloudWatch drops the ball is in
\nexposing this behavior to the end user without explanation as to what’s
\ngoing on. Thus, until you grow accustomed to it, you have a heart-stopping
\nmoment of “what the hell just happened to the site” whenever you
\nglance at a dashboard. This conditions you to be entirely too calm when
\nlooking at sensible dashboards when a disaster just happened. If you trust
\nwhat the CloudWatch dashboards show you, you’re making a terrible
\nmistake.<\/p>\n

“Dive Deep”<\/h3>\n

If you’re using Lambda or Fargate, you have no choice but to use CloudWatch
\nLogs, wherein searching for everything is absolutely terrible. If you’re
\nusing CloudWatch Logs to diagnose anything, congratulations: you’re
\ndiving so deep, you may drown before making it back to the surface.
\nFor example, if I have a Lambda function that throws an error, in order to
\ndiagnose the problem, I must:<\/p>\n