Recommand · October 22, 2021 0

Cloudwatch not parsing multiline logs with multiple JSON objects

Our API (.Net Core) running in AWS creates multi-line log entries like this:

2021-10-21 15:02:22.968|Error|{
  "Message": "An error has occurred."
}|Product.Api.Exceptions.ApiException: {
  "Message": "An error has occurred."
}
   at Product.Api.Utils.Http.ResponseValidationHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\BuildAgent\work\Product.Api\Utils\Http\Response
ValidationHandler.cs:line 26

These are fed directly into AWS Cloudwatch but are split into multiple log entries, like this:

2021-10-21T15:02:25.123+01:00   2021-10-21 15:02:22.968|Error|{ "Message": "An error has occurred."
2021-10-21T15:02:25.123+01:00   }|Product.Api.Exceptions.ApiException: { "Message": "An error has occurred."
2021-10-21T15:02:25.123+01:00   } at Product.Api.Utils.Http.ResponseValidationHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\BuildAgent\work\Product.Api\Utils\Http\Response

We then send them directly to DataDog, so we need them correct in Cloudwatch before we do.

If the multiline log was a single valid JSON string I believe it would parse correctly, but because there is a combination of string, JSON and pipes(|) it seems to just split on what it thinks are the separate JSON objects.

Is there anyway to keep it as one log entry in Cloudwatch?

I’ve seen a lot of solutions, but they all seem to relate to Cloudwatch Insights/Metrics, but we send these logs straight to DataDog, so don’t analyse them in Cloudwatch itself.