Set up Data Pipeline Error Alerts with Amazon EventBridge (CloudWatch) (OPTIONAL)

Note: Using Amazon EventBridge together with CloudWatch is optional. If the system does not require event-driven architecture or complex event routing, CloudWatch can still be used independently for monitoring, logging, and alerting as usual.

Objectives

Set up an automated event-driven mechanism and end-to-end monitoring for the Data Pipeline lifecycle. Specifically, the system will perform two core tasks:

1. Automatic Trigger: Detect when new data files (Raw Data) are uploaded to Amazon S3 and immediately trigger AWS Step Functions (State Machine) to run the ETL workflow.
2. Centralized Error Alerting: Monitor the status of the entire AWS Step Functions workflow. If any step (Crawler/Glue Job) fails and causes the pipeline to fail (FAILED, TIMED_OUT, ABORTED), the system will extract the error and send an alert email through Amazon SNS.

Architecture Logic Overview

• S3 Upload Event (Rule 1): The S3 bucket acts as the event source. When a new object is created (Object Created), EventBridge captures the event stream and invokes the target service, AWS Step Functions, so the system can run automatically without manual intervention.
• Step Functions Error Event (Rule 2): Instead of handling small errors in each Glue Job separately, Step Functions acts as the conductor. If an error occurs in any node, Step Functions changes the overall status to FAILED. EventBridge detects this status, uses an Input Transformer to extract the executionArn, and formats a clear alert message before sending it to Amazon SNS.

Detailed Amazon EventBridge (CloudWatch) Setup

We will set up the two EventBridge rules sequentially based on the two architecture logics above.

Part 1: Create Rule 1 - Monitor Step Functions Errors

Step 1: Open Amazon EventBridge

First, use the search bar to open Amazon EventBridge

Step 2: Create the EventBridge Rule (Rule 1)

1. From the left menu, choose Rules -> Create rule.

2. Builder mode: Choose Advanced builder
3. Name: Enter monitor-stepfunctions-errors.
4. Description: Monitors AWS Step Functions state machine execution changes and triggers an SNS alert if the orchestration pipeline fails, times out, or is aborted.
5. Event bus: Keep default.
6. Click Next

Step 3: Configure the Event Filter (Rule 1)

1. Events: Choose AWS events or EventBridge partner events.

2. Scroll down to the Event pattern section:

• Creation method: Choose Use pattern form
• Event source: Choose AWS services.
• AWS service: Choose Step Functions.
• Event type: Choose Step Functions Execution Status Change.
• Event Type Specification 1: Select Specific state(s) and choose FAILED, TIMED_OUT, and ABORTED.

3. Click Next.

Step 4: Select the Target and Configure the Input Transformer

1. Target types: Choose AWS service.
2. Select a target: Search for and choose SNS topic.
3. Topic: Open the list and select your SNS-Alerts topic.
4. Expand Additional settings

In Configure target input, choose Input transformer -> click Configure input transformer.

5. Part 1: Input Path - Copy and paste the following JSON:

{
	"pipelineName": "$.detail.stateMachineArn",
	"executionArn": "$.detail.executionArn",
	"status": "$.detail.status"
}

6. Part 2: Template - Copy and paste the following text:

"WARNING: The Step Functions Data Pipeline execution has failed! \n\n- State Machine: <pipelineName>\n- Execution ID: <executionArn>\n- Status: <status>\n\nPlease review the logs in the AWS Step Functions Console to identify which Glue Job encountered an issue."

7. Click Confirm -> Next -> skip Tags -> click Create rule.

Part 2: Create Rule 2 - Automatic Trigger from S3

Step 5: Create the EventBridge Rule (Rule 2)

1. Return to the Rules page and click Create rule.
2. Name: Enter trigger-on-s3-upload.
3. Description: Automatically triggers the AWS Step Functions data orchestration pipeline whenever a new raw data object is uploaded to the landing S3 bucket
4. Event bus: Keep default.
5. Click Next.

Step 6: Configure the Event Filter (Rule 2)

1. Events: Choose AWS events or EventBridge partner events.

2. Scroll down to the Event pattern section:

• Event source: Choose AWS services.
• AWS service: Choose Simple Storage Service (S3).
• Event type: Choose Amazon S3 Event Notification.
• Specific event(s): Select Object Created.
• Specific bucket(s) by name: Enter the exact name of your raw data S3 bucket: yellow-taxi-trip-demo-fcaj

3. The JSON format in the validation box on the right will look similar to the following:

{
	"source": ["aws.s3"],
	"detail-type": ["Object Created"],
	"detail": {
		"bucket": {
			"name": ["your-s3-bucket-name"]
		}
	}
}

4. Click Next.

Step 7: Select the Target (Invoke Step Functions)

1. Select a target: Choose AWS service.
2. Select a target: Search for and choose Step Functions state machine.
3. State machine: Select DataPipeline-Orchestrator.
4. Expand Additional settings

In Configure target input, choose Input transformer -> click Configure input transformer.

5. Part 1: Input Path - Copy and paste the following JSON:

{
	"pipelineName": "$.detail.stateMachineArn",
	"executionArn": "$.detail.executionArn",
	"status": "$.detail.status"
}

6. Part 2: Template - Copy and paste the following text:

"WARNING: The Step Functions Data Pipeline execution has failed! \n\n- State Machine: <pipelineName>\n- Execution ID: <executionArn>\n- Status: <status>\n\nPlease review the logs in the AWS Step Functions Console to identify which Glue Job encountered an issue."

7. Click Confirm -> Next -> skip Tags -> click Create rule.

8. Click Next -> skip Tags -> click Create rule.

Done!

Your Data Pipeline is now fully automated. As soon as you upload a data file to S3, Step Functions will automatically run the entire ETL workflow. At the same time, any errors that occur in the system chain will be collected and delivered neatly to your inbox.