Gears linked together

Automate Setup of EC2 Disk and Memory Metrics using AWS CloudWatch Agent and Systems Manager

- Updated August 24, 2020

Amazon Web Services offers a plethora of services that easily customize an environment while remaining cost-effective for a business. Usually, the tricky part is piecing together the different services to form a fully functioning system. In this blog post, I use IAM, EC2, CloudWatch, and AWS Systems Manager (SSM) to create a scalable system that monitors disk and memory usage of EC2 instances. The AWS CloudWatch Agent and SSM Agent are preinstalled on the Amazon Linux 2 AMI so no additional installation is needed!

Getting Started – Set up our Permissions

In AWS, navigate to the IAM service. 

  1. Select Roles in the left side nav.
  2. Click the Create Role button.
  3. Select AWS service for your trusted entity. 
  4. Choose EC2 for your use case.
  5. In Permissions, search for and select AmazonSSMManagedInstanceCore and CloudWatchAgentServerPolicy
  6. If you wish, tag the role using best practices.  
  7. Review the role and give it a name like SSM or DefaultEC2Role
  8. Create the role. This role allows automatic management of EC2 instances.
  9. Create one or two EC2 instances and assign the SSM role from earlier. You may optionally use EC2 instances that already exist in your environment. Make sure this role is attached to your running EC2 instances.
  10. After a few minutes, the instances register themselves with SSM and become managed instances. Now, SSM can manage these instances automatically with Run Documents and Associations.

Configure AWS Systems Manager to Automate Configuration

Navigate to the Systems Manager service 

  1. Select Parameter Store in the side nav. 
  2. Click, Create Parameter. 
  3. Give the parameter a name like CloudWatchMemDisk, and a description. 
  4. Select the standard tier and string for the type. 
  5. The JSON we’ll use for the value field is:
    {
       "agent":{
          "run_as_user":"root"
       },
       "metrics":{
          "append_dimensions":{
             "AutoScalingGroupName":"${aws:AutoScalingGroupName}",
             "ImageId":"${aws:ImageId}",
             "InstanceId":"${aws:InstanceId}",
             "InstanceType":"${aws:InstanceType}"
          },
          "metrics_collected":{
             "mem":{
                "measurement":[
                   "mem_used_percent"
                ]
             },
             "disk":{
                "measurement":[
                   "used_percent"
                ],
                "resources":[
                   "*"
                ]
             }
          }
       }
    }
    
  6. After creating the parameter, navigate to State Manager in the left side nav.
  7. Click, Create Association. 
  8. Give your association a name, like CloudWatch-MemDisk. 
  9. For the corresponding document, search for AmazonCloudWatch-ManageAgent and select it. The default version is fine. 
  10. For the parameters, select Action: configure; Mode:ec2; Optional Configuration Source: ssm; Optional Configuration Location: CloudWatchMemDisk (this is our string we created in Parameter Store); Optional Restart: yes (to restart the Cloudwatch Agent on the EC2 instance). 
  11. Managing your EC2 instances by Resource Groups or by Tags is the preferred approach, but, for the sake of simplicity, I’m going to manually target a single EC2 instance in my environment. Although SSM allows you to run our Association on a schedule or only once. If you plan to add additional tagged EC2 instances or instances to resource groups, set a schedule to ensure this association is applied to new instances. 
  12. Otherwise, choose “Run Once.” In this use case, I won’t write to S3 or change advanced options and rate control. 
  13. Click Create Association.

Viewing the New Metrics in CloudWatch

Next, navigate to the CloudWatch service. CloudWatch is the AWS service that monitors instances, collects logs, and displays metrics via its CloudWatch Agent installed on EC2 instances. In CloudWatch, select Metrics in the left side nav. In the all metrics tab, select CWAgent and look by instance name for  mem_used_percent metrics. Select these metrics to display the instance’s memory used percentage. Now, you can create a CloudWatch Dashboard with this information to share with your team or set up Alarms to notify when memory usage is too high.