Automate Monitoring for Stopped ECS Container using Event Driven System (AWS)

In a cloud environment, monitoring the state of your resources is critical for maintaining the health and performance of your applications. This blog will guide you on how to monitor your stopped Amazon Elastic Container Service (ECS) containers using event-driven systems, specifically using Amazon EventBridge.

High Level Approach

1. Configure Event Bridge

2. Create the Target

3. Handle the Event

4. Event Details

Amazon ECS and EventBridge

If you are reading this then you are already familiar with ECS. So, before we delve into the nuts and bolts, let's take a moment to understand the Amazon EventBridge.

Amazon EventBridge is a serverless event bus service that makes it easy for your applications to communicate with each other. It does this by connecting different parts of your application, called "services," and allowing them to send and receive information, called "events."

Amazon ECS sends the following types of events to EventBridge:

• Container instance state change events,

• Task state change events,

• Service action, and

• Service deployment state change events

If these resources change, an event is generated. These events are a powerful tool for monitoring the state of your ECS resources.

Let's dive in to AWS Console..!!

STEPS :

1. Open the Event Bridge Console

2. Click on Create Rule, enter the Rule Details and click Next.

3. Choose Event Source, let it be 'AWS Events' and you can ignore Sample Event (optional) details.

4. Choose Custom Pattern (JSON Editor) as a Creation Method.

5. Provide the below Event Pattern and click on next. (This is the most imp step)

- This Pattern will capture only ECS Events, when there is a change in ECS Task State. Also, We are only capturing events when container gets stopped.

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Task State Change"],
  "detail": {
    "containers": {
      "lastStatus": ["STOPPED"]
    }
  }
}

6. Select Target Types as 'AWS Service' and from the drop down, choose 'Lambda'

7. Currently, we have not created any Lambda yet. So let's open a another Tab and create a Lambda with below details and :

8. Add the below Code to Lambda

def lambda_handler(event, context):
    try:
       print(event)
        
    except Exception as e:
        raise e
        

9. Go back to the Previous Tab (Event Bridge Page) and choose the newly created Lambda (ecs-monitoring-system) and create a Rule.

Up to this point, the major part has been completed. We have successfully created a rule so that whenever an 'STOPPED' event is generated by ECS, Lambda will be triggered. Now we need to determine the action we want to take. In this example, for sake of simplicity, we will send an email using SNS to the responsible team so that they will be alerted.

10. Go to SNS Service and Create a SNS Topic (Standard):

11. Create a subscription, choose Email Protocol and provide your email address in the Endpoint field as below:-

12. Go your email inbox and confirm the subscription.

Now, we have to update our Lambda Code with SNS Logic so that whenever Lambda receives Stopped Events then we should send an email.

13. As we will be using SNS to send an email, add SNS Permissions to your Lambda IAM Role.

14. Update the Lambda code as below and also replace the TopicArn in the Code with your SNS ARN;

import boto3

def lambda_handler(event, context):
    try:
       print(event)
       send_email()
       
    except Exception as e:
        raise e
        
def send_email():
    try:
        
        client = boto3.client('sns')
        
        response = client.publish(
            TopicArn='arn:aws:sns:<region-id>:<sns-topic-name>',
            Subject= 'Need Attention | ECS Container got stopped',
            Message='Your Custom Message')
                
    except Exception as e:
        raise e

15. Testing your Lambda Function. Click on Configure Test Event

16. Configure the below dummy ECS Stopped Event in the Lambda to test it out.

{
  "version": "0",
  "id": "f8a990c2-2f93-4713-8b5d-d5b96f35bfd7",
  "detail-type": "ECS Task State Change",
  "source": "aws.ecs",
  "account": "123456789012",
  "time": "2016-09-15T21:57:35Z",
  "region": "us-east-1",
  "resources": ["arn:aws:ecs:us-east-1:123456789012:task/3102878e-4af2-4b3c-b9c1-2556b95b2bbf"],
  "detail": {
    "clusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/cluster1",
    "containerInstanceArn": "arn:aws:ecs:us-east-1:123456789012:container-instance/04f8c17d-29e0-4711-aa74-852654e477ec",
    "containers": [{
      "containerArn": "arn:aws:ecs:us-east-1:123456789012:container/40a3b4bd-79ae-4472-a0be-816e5e0044a0",
      "lastStatus": "STOPPED",
      "name": "test",
      "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/3102878e-4af2-4b3c-b9c1-2556b95b2bbf"
    }],
    "createdAt": "2016-09-15T21:30:33.3Z",
    "desiredStatus": "RUNNING",
    "lastStatus": "STOPPED",
    "overrides": {
      "containerOverrides": [{
        "command": ["command1", "command2"],
        "environment": [{
          "name": "env1",
          "value": "value1"
        }, {
          "name": "env2",
          "value": "value2"
        }],
        "name": "test"
      }]
    },
    "updatedAt": "2016-09-15T21:30:33.3Z",
    "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/3102878e-4af2-4b3c-b9c1-2556b95b2bbf",
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:123456789012:task-definition/testTD:1",
    "version": 1
  }
}

17. Save it and click on Test Button and voila.!!

Conclusion

Above is just a basic implementation which can be improved. But in Conclusion, Amazon EventBridge is a powerful service that lets you build highly scalable, event-driven applications. It provides a simple and consistent way to ingest, filter, transform, and deliver events so your applications can respond in real time to changes that occur within your AWS environment.

Remember, the power of EventBridge lies not just in its ability to handle a high volume of event data, but also in its flexibility. You can customize your rules and targets to meet your specific needs, making it a versatile tool in the arsenal of any cloud engineer.

As we move towards increasingly dynamic and distributed systems, services like EventBridge will only grow in importance. It's crucial to understand and leverage these tools to build resilient, responsive applications. Happy building!