How to Fix Out Of Storage issue in AWS EC2 Instance

The issues of EC2 instance with mysql 8.0 growing out of storage is growing more and more but one cannot be certain of the issue without looking at it.

There are factors which needs to be considered and checked before jumping into any such conclusion :

• Check if there is a memory leak at the application layer.
• Check your server logs apache/nginx and looks for any red flags.
• Check application cron logs if any.
• Check if the debug modules and debug code is present on the live site.

Take the following steps to start looking at the server level, login to the application using SSH and follow along :

• Execute the following command to check what is the current status of your server and if they are running or not.

  • sudo service nginx status
  • sudo service apache2 status

   

• If the above is working as expected, check the current status of storage and which volume is consuming how much of the storage using the following command

  • df -h

• Check if the application volume in this instance "/dev/nvme0n1p1" is consuming a lot of storage. If this is taking too much or more than 80% there is a possible issue with the storage at this point.

• If the above is consuming a lot of memory we need to drill down to see which parts are consuming the most storage space. Execute the below command to list down

  • sudo du /var/* -shc

  

• If the above looks similar where the /var/lib is using a lot of storage space, execute the below command to find out more about which library is using the most storage space.

  • sudo du /var/lib/* -shc

• If the above results in mysql using most of the storage, there is a problem with mysql configuration.

• Check if the mysql service is running using the below command

  • sudo service mysql status

• If the mysql service is running without any error, the problem is mysql is not cleaning up the mysql binary logs that needs to be auto-cleared. Mysql 8.0 enables the binary log by default, hence it needs to be set to auto purge.
  • Log into the mysql terminal using the below command

    • mysql -u root -p

  • Execute the following command to check binary logs

    • SHOW BINARY LOGS

  • Remove old binary logs by executing the command with the binary log name from the execution of the above command

    • PURGE BINARY LOGS TO 'binary-log.000349'

  • Next, set the auto timeout to clear binary logs using the below command to clear out every 5hours (60sec x 60min x 5hour) = 18000

    • SET GLOBAL binlog_expire_logs_seconds = 18000;
    • SET PERSIST binlog_expire_logs_seconds = 18000;

     

• Check the storage again 

  • df -h