What to do if the AWS default Route is being modified after reboot?

Author : Farhan Ali

If your AWS default route is being modified after reboot, there could be several reasons for this behavior. Here are some common causes and troubleshooting steps:

1. Internet Connectivity:The reason default route is getting modified could be if the Netscaler is not able to contact the meta-data server during boot up. To check if the internet connectivity is present/not, execute the below commands:

show route

Now, check the route for 169.254.169.254 and verify the route is proper via management 1/1 interface.If the route is missing, then create a static route for 169.254.169.254 via management interface gateway (normally it will be VPC subnet ending with .1 i.e x.x.x.1 where x.x.x is VPC subnet). Verify if the instance is created with metadata option IMDSv2 only. If yes, then stop the instance and change the setting to imdsv1 and v2 both OR upgrade to 13.1.46.x and above release.

2. Automation Scripts or Configuration Management Tools: Check if you have any automation scripts or configuration management tools (e.g., AWS CloudFormation, AWS OpsWorks, Ansible) running that might be modifying the route during bootstrapping or configuration updates.

3. User Actions: Ensure that no one with appropriate permissions is manually modifying the default route. AWS IAM (Identity and Access Management) policies should be reviewed to see who has permission to modify route tables.

4. Instance Metadata Service: Make sure that there are no scripts or services running on the instance that are fetching instance metadata and modifying routes based on that information.

5. Incorrect Route Table Association: Ensure that the instance is associated with the correct route table. If it's associated with multiple route tables, there might be conflicts in route configurations.

6. Network ACLs and Security Groups: Check if there are any network ACLs or security group rules that might be affecting the routing behavior.

7. VPC Peering or VPN Connections: If you have VPC peering or VPN connections set up, verify that the routes and configurations are correct in those connections.

8. AWS Managed Services: Some AWS managed services might automatically modify routes based on certain conditions. For example, AWS Direct Connect or AWS VPN might modify routes based on BGP advertisements.

9. Ephemeral Storage: If your instance is using ephemeral storage for configuration or scripts, ensure that the changes are persisted or re-applied after reboot.

10. Check Route Table Logs: AWS CloudTrail logs can provide insights into who made changes to route tables and when they were made. Check the CloudTrail logs for any relevant events.