Pipe Docker container logs to AWS Cloud Watch

There comes a time when you can’t simply write all your logs to log files or inside a docker container. Many platforms provide inbuilt CloudWatch agents inside them. But what if the logging library doesn’t provide it by default? Don’t need to worry about it, that said every problem comes with a solution 😃.

Drawbacks of writing logs to files and container:

1. While writing our application logs to files, we’ve to maintain it consistently(no doubt one can rotate it if want, on certain time period)
2. If one writes logs to the docker container instead of writing it to files, there will always be a case when we need history logs for tracking our application’s working flow or maybe bugs sometimes. At that time docker won’t have the long-lasting history logs.

Advantage of writing logs to Cloud Watch console:

1. AWS CloudWatch persists logs for our defined retention period(1 day, 3 days, 5 days. 7 days, 1 month…) that totally depends on us, once the retention period expired it will vanish logs automatically.

Photo by Sigmund on Unsplash

Here, we’ll see how to bypass application logs to AWS CloudWatch console with an assumption that you’re already familiar with docker deployment on AWS EC2 instance and have a basic knowledge about AWS CloudWatch(if not then you can get more info here). You can use the same trick with any of your projects which is deployed on AWS EC2 using docker.

Note : Verify that your AWS EC2 instance has an IAM role attached with it has a permission of accessing AWS CloudWatch services, if not you can see how to allow IAM user to access cloudwatch logs.

When running docker you can add a logging driver awslogs with following.

$ docker run \
    --log-driver=awslogs \
    --log-opt awslogs-region=eu-central-1 \
    --log-opt awslogs-group=myLogGroup \
    --log-opt awslogs-stream=myLogStream
    ...

where,
awslogs-region = AWS REGION in which EC2 instance is deployed
awslogs-group = created log group in AWS CloudWatch
awslogs-stream = created logs stream inside AWS CloudWatch log group
you can find details of creating log group and log streams here.

Or you can specify, which logging driver you’re going to use in your docker-compose.yml like the following.

docker-compose.yml:

myService:
logging:
  driver: awslogs
    options:
      awslogs-region: eu-central-1
      awslogs-group: myLogGroup
      awslogs-stream: myLogStream

Voila! you can see AWS CloudWatch has a log group named myLogGroup and inside which log-stream myLogStream is recording your application logs.
We can also set extra options such as aws-datetime-format .

Example — Rails application logging to CloudWatch

Let’s write production logs into CloudWatch.

If you’re currently writing your logs to file, then replace logging configuration with the following code to write logs directly to the docker container.

config/environments/production.rb

if ENV[“RAILS_LOG_TO_STDOUT”].present?
logger = ActiveSupport::Logger.new(STDOUT)
logger.formatter = config.log_formatter
config.logger = ActiveSupport::TaggedLogging.new(logger)
end

Here, STDOUT will write logs directly into docker container instead of log files and then it will be piped to AWS cloudwatch.

Hope you get some crunchy clues about AWS CloudWatch logging directly from the docker container.

Thanks for your support!
If you like what you read, be sure to 👏 it below — as a writer it means the world!
Follow Canopas Software to get updates on interesting tech articles!

Happy coding!!