Spring Boot on Fargate slow

Hi,

We have recently upgraded our application for a monolithic app running on JDK7 in a single Docker container on ECS with EC2. We’ve upgraded to JDK 11 and split the services into microservices funning as Fargate tasks. The web frontend is using the Tomcat Docker image, and the microservices are running on OpenJDK Alpine Docker images. The adatabase is Postgres on RDS.

Since this upgrade the entire system is running a less then half the speed it used to before the upgrade.
If I run all the services on my laptop they work perfectly, but once deployed to Fargate they are very slow.

I have enabled all the ClouWatch monitoring, but nothing looks out of the ordinary.

I have also tried many combinations of JAVA_OPTS, with no success.

Has anyone had this experience or can suggest any solutions?

Thanks
Nikolai

Hi @nikolaiv

Can you share the task definition with us (don’t forget to remove sensitive information, e.g. in environment variables?

Thanks
Michael

You need to identify where the bottleneck is.

  • What exactly is now slower than before and what “slower” even means?
  • Is that the end-user latency?
  • Is it related to specific user transactions/operations or it just feels generally slower?
  • Do you have an easy reproducer that you can use to measure response time (or throughput if that’s the concern) against both old and new approaches?

Because you migrated your monolith to microservices, it’s likely that you suffer from extra latency induced by network calls since now the end-user transactions require coordination of multiple services.
If yo have latency issues, then you need to break down the total response time into its components and find out what operations take longer - adding some logging around potentially long-running operations can be easy and enough to start.

Hi,

Every service is running slower… I can imagine a bit of network latency but not to this extent.
I tried to implement AWS XRay, but it crashed the application with a servlet error… I see there are many different ways to use XRay - not sure what the correct configuration is?

Here is an example of one of the task definitions…

{
“ipcMode”: null,
“executionRoleArn”: “arn:aws:iam::334203084188:role/ecsTaskExecutionRole”,
“containerDefinitions”: [
{
“dnsSearchDomains”: null,
“environmentFiles”: null,
“logConfiguration”: {
“logDriver”: “awslogs”,
“secretOptions”: null,
“options”: {
“awslogs-group”: “/ecs/assessment-qa”,
“awslogs-region”: “ap-southeast-2”,
“awslogs-stream-prefix”: “ecs”
}
},
“entryPoint”: null,
“portMappings”: [
{
“hostPort”: 8081,
“protocol”: “tcp”,
“containerPort”: 8081
}
],
“command”: null,
“linuxParameters”: null,
“cpu”: 0,
“environment”: [
{
“name”: “JAVA_OPTS”,
“value”: “-XX:+UseG1GC -XX:MinRAMPercentage=40.0 -XX:MaxRAMPercentage=75.0 -XX:+HeapDumpOnOutOfMemoryError”
}
],
“resourceRequirements”: null,
“ulimits”: null,
“dnsServers”: null,
“mountPoints”: [],
“workingDirectory”: null,
“secrets”: null,
“dockerSecurityOptions”: null,
“memory”: null,
“memoryReservation”: null,
“volumesFrom”: [],
“stopTimeout”: null,
“image”: “334203084188.dkr.ecr.ap-southeast-2.amazonaws.com/assessment:qa”,
“startTimeout”: null,
“firelensConfiguration”: null,
“dependsOn”: null,
“disableNetworking”: null,
“interactive”: null,
“healthCheck”: null,
“essential”: true,
“links”: null,
“hostname”: null,
“extraHosts”: null,
“pseudoTerminal”: null,
“user”: null,
“readonlyRootFilesystem”: null,
“dockerLabels”: null,
“systemControls”: null,
“privileged”: null,
“name”: “assessment-qa”
}
],
“placementConstraints”: [],
“memory”: “4096”,
“taskRoleArn”: “arn:aws:iam::334203084188:role/ecsTaskExecutionRole”,
“compatibilities”: [
“EC2”,
“FARGATE”
],
“taskDefinitionArn”: “arn:aws:ecs:ap-southeast-2:334203084188:task-definition/assessment-qa:8”,
“family”: “assessment-qa”,
“requiresAttributes”: [
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “com.amazonaws.ecs.capability.logging-driver.awslogs”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “ecs.capability.execution-role-awslogs”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “com.amazonaws.ecs.capability.ecr-auth”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “com.amazonaws.ecs.capability.docker-remote-api.1.19”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “com.amazonaws.ecs.capability.task-iam-role”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “ecs.capability.execution-role-ecr-pull”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “com.amazonaws.ecs.capability.docker-remote-api.1.18”
},
{
“targetId”: null,
“targetType”: null,
“value”: null,
“name”: “ecs.capability.task-eni”
}
],
“pidMode”: null,
“requiresCompatibilities”: [
“FARGATE”
],
“networkMode”: “awsvpc”,
“cpu”: “2048”,
“revision”: 8,
“status”: “ACTIVE”,
“inferenceAccelerators”: null,
“proxyConfiguration”: null,
“volumes”: []
}

Thanks
Nikolai

@nikolaiv What’s the instance type/sizing of your EC2 instances in your “old” ECS cluster?

If you define 2048 CPU units (2 cores) and 4 GB memory on Fargate/ECS, you get exactly this amount of resources and nothing more.
If you use the same constraints on EC2/ECS and the host has more resources available, your container can use those resources as well.

The thing is my EC2 host was 2 CPU with 4 GB running 6 services… now I’ve split those services onto 6 Fargate tasks each running 2CPU and 4GB. It should have way more power, but its just crawling along.

I even took the old container and ran it on Fargate instead of EC2 and it runs like a dream… has to be something with the Java configuration (i.e. no longer in Tomcat but using Spring Boot embedded Tomcat), or network speed, but I don’t know how I could improve the network when they’re all running in the same VPC the same subnet and same availability zone?!?

Someone must have configured Java on Fargate and got it working!
I tried setting up AWS XRay for logging, but it just crashes the app… there seem to be many ways to set it up - anyone had any luck with t?

Any help is much appreciated!

Thanks
Nikolai

I see. I have no experience with “Spring Boot embedded Tomcat” so let’s see if someone else can add anything to your thread.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.