💡 Summary
A comprehensive guide and code library for interacting with AWS Simple Queue Service (SQS) for message queuing in decoupled architectures.
🎯 Target Audience
🤖 AI Roast: “It's a well-written SQS manual that forgot to put on its 'AI Agent Skill' costume.”
The skill requires AWS credentials with SQS permissions, posing a secret management risk. Code examples perform network calls to AWS APIs. Mitigation: Use temporary credentials (e.g., AWS IAM Roles) and never hardcode secrets in the agent's environment.
name: sqs description: AWS SQS message queue service for decoupled architectures. Use when creating queues, configuring dead-letter queues, managing visibility timeouts, implementing FIFO ordering, or integrating with Lambda. last_updated: "2026-01-07" doc_source: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/
AWS SQS
Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling and scaling microservices, distributed systems, and serverless applications.
Table of Contents
Core Concepts
Queue Types
| Type | Description | Use Case | |------|-------------|----------| | Standard | At-least-once, best-effort ordering | High throughput | | FIFO | Exactly-once, strict ordering | Order-sensitive processing |
Key Settings
| Setting | Description | Default | |---------|-------------|---------| | Visibility Timeout | Time message is hidden after receive | 30 seconds | | Message Retention | How long messages are kept | 4 days (max 14) | | Delay Seconds | Delay before message is available | 0 | | Max Message Size | Maximum message size | 256 KB |
Dead-Letter Queue (DLQ)
Queue for messages that failed processing after maxReceiveCount attempts.
Common Patterns
Create a Standard Queue
AWS CLI:
aws sqs create-queue \ --queue-name my-queue \ --attributes '{ "VisibilityTimeout": "60", "MessageRetentionPeriod": "604800", "ReceiveMessageWaitTimeSeconds": "20" }'
boto3:
import boto3 sqs = boto3.client('sqs') response = sqs.create_queue( QueueName='my-queue', Attributes={ 'VisibilityTimeout': '60', 'MessageRetentionPeriod': '604800', 'ReceiveMessageWaitTimeSeconds': '20' # Long polling } ) queue_url = response['QueueUrl']
Create FIFO Queue
aws sqs create-queue \ --queue-name my-queue.fifo \ --attributes '{ "FifoQueue": "true", "ContentBasedDeduplication": "true" }'
Configure Dead-Letter Queue
# Create DLQ aws sqs create-queue --queue-name my-queue-dlq # Get DLQ ARN DLQ_ARN=$(aws sqs get-queue-attributes \ --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-dlq \ --attribute-names QueueArn \ --query 'Attributes.QueueArn' --output text) # Set redrive policy on main queue aws sqs set-queue-attributes \ --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \ --attributes "{ \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\" }"
Send Messages
import boto3 import json sqs = boto3.client('sqs') queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue' # Send single message sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps({'order_id': '12345', 'action': 'process'}), MessageAttributes={ 'MessageType': { 'DataType': 'String', 'StringValue': 'Order' } } ) # Send to FIFO queue sqs.send_message( QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo', MessageBody=json.dumps({'order_id': '12345'}), MessageGroupId='order-12345', MessageDeduplicationId='unique-id-12345' ) # Batch send (up to 10 messages) sqs.send_message_batch( QueueUrl=queue_url, Entries=[ {'Id': '1', 'MessageBody': json.dumps({'id': 1})}, {'Id': '2', 'MessageBody': json.dumps({'id': 2})}, {'Id': '3', 'MessageBody': json.dumps({'id': 3})} ] )
Receive and Process Messages
import boto3 import json sqs = boto3.client('sqs') queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue' while True: # Long polling (wait up to 20 seconds) response = sqs.receive_message( QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=20, MessageAttributeNames=['All'], AttributeNames=['All'] ) messages = response.get('Messages', []) for message in messages: try: body = json.loads(message['Body']) print(f"Processing: {body}") # Process message... # Delete on success sqs.delete_message( QueueUrl=queue_url, ReceiptHandle=message['ReceiptHandle'] ) except Exception as e: print(f"Error processing message: {e}") # Message will become visible again after visibility timeout
Lambda Integration
# Create event source mapping aws lambda create-event-source-mapping \ --function-name my-function \ --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \ --batch-size 10 \ --maximum-batching-window-in-seconds 5
Lambda handler:
def handler(event, context): for record in event['Records']: body = json.loads(record['body']) message_id = record['messageId'] try: process_message(body) except Exception as e: # Raise to put message back in queue raise return {'batchItemFailures': []}
CLI Reference
Queue Management
| Command | Description |
|---------|-------------|
| aws sqs create-queue | Create queue |
| aws sqs delete-queue | Delete queue |
| aws sqs list-queues | List queues |
| aws sqs get-queue-url | Get queue URL by name |
| aws sqs get-queue-attributes | Get queue settings |
| aws sqs set-queue-attributes | Update queue settings |
Messaging
| Command | Description |
|---------|-------------|
| aws sqs send-message | Send single message |
| aws sqs send-message-batch | Send up to 10 messages |
| aws sqs receive-message | Receive messages |
| aws sqs delete-message | Delete message |
| aws sqs delete-message-batch | Delete up to 10 messages |
| aws sqs purge-queue | Delete all messages |
Visibility
| Command | Description |
|---------|-------------|
| aws sqs change-message-visibility | Change timeout |
| aws sqs change-message-visibility-batch | Batch change |
Best Practices
Message Processing
- Use long polling (WaitTimeSeconds=20) to reduce API calls
- Delete messages promptly after successful processing
- Configure appropriate visibility timeout (> processing time)
- Implement idempotent consumers for at-least-once delivery
Dead-Letter Queues
- Always configure DLQ for production queues
- Set appropriate maxReceiveCount (usually 3-5)
- Monitor DLQ depth with CloudWatch alarms
- Process DLQ messages manually or with automation
FIFO Queues
- Use message group IDs to partition ordering
- Enable content-based deduplication or provide dedup IDs
- Throughput: 300 msgs/sec without batching, 3000 with
Security
- Use queue policies to control access
- Enable encryption with SSE-SQS or SSE-KMS
- Use VPC endpoints for private access
Troubleshooting
Messages Not Being Received
Causes:
- Short polling returning empty
- All messages in flight (visibility timeout)
- Messages delayed (DelaySeconds)
Debug:
# Check queue attributes aws sqs get-queue-attributes \ --queue-url $QUEUE_URL \ --attribute-names All # Check approximate message counts aws sqs get-queue-attributes \ --queue-url $QUEUE_URL \ --attribute-names \ ApproximateNumberOfMessages,\ ApproximateNumberOfMessagesNotVisible,\ ApproximateNumberOfMessagesDelayed
Messages Going to DLQ
Causes:
- Processing errors
- Visibility timeout too short
- Consumer not deleting messages
Redrive from DLQ:
# Enable redrive allow policy on source queue aws sqs set-queue-attributes \ --queue-url $MAIN_QUEUE_URL \ --attributes '{"RedriveAllowPolicy": "{\"redrivePermission\":\"allowAll\"}"}' # Start redrive aws sqs start-message-move-task \ --source-arn arn:aws:sqs:us-east-1:123456789012:my-queue-dlq \ --destination-arn arn:aws:sqs:us-east-1:123456789012:my-queue
Duplicate Processing
Solutions:
- Use FIFO queues for exactly-once
- Implement idempotency in consumer
- Track processed message IDs in database
Lambda Not Processing
# Check event source mapping aws lambda list-event-source-mappings \ --function-name my-function # Check for errors aws lambda get-event-source-mapping \ --uuid <mapping-uuid>
References
Pros
- Well-structured documentation covering core concepts and patterns.
- Provides practical CLI and Python (boto3) code examples.
- Includes best practices and troubleshooting guidance for real-world use.
Cons
- Primarily a documentation/guide, not a novel 'skill' with unique logic.
- Utility is dependent on external AWS credentials and permissions.
- No abstraction or automation beyond standard AWS SDK usage.
Disclaimer: This content is sourced from GitHub open source projects for display and rating purposes only.
Copyright belongs to the original author itsmostafa.
