The Linux server's hard disk space is insufficient and cannot be written, but there is still some space left on the hard disk. It turns out that Inodes are to blame!

Ubuntu system when you encounter the problem of insufficient hard disk space and unable to write

exist Linux On a server, even if the hard disk seems to have enough space, you may still encounterUnable to writeThis is usually due to inodes Exhausted.

When the hard disk cannot be writtenDisk spaceIf there are still some remaining, it is likely that inodes are exhausted. You can clean up small files, adjustFile SystemOr archiving storage and other methods to solve the problem, and cooperate with monitoring and optimization design to prevent similar problems from happening again.

What are inodes?

Inodes are data structures in the file system that store meta-information about files (such as file permissions, owner, creation time, etc.).
Each file occupies an inode, no matter how big the file is. If the inodes are exhausted, even if the hard disk still hasRemaining space, and new files cannot be created.

How to check the usage of hard disk and inodes?

Check Disk Space

df -h

This will show you the disk usage in a human readable format.

Check inodes usage

df -i

Sample output:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1000000 999999 1 100% /

IUse% indicates the usage rate of inodes. If IUse% reaches 100%, it means that inodes are exhausted.

• Too many small files will consume a large number of inodes, especially log files and temporary files.
• File system type. Some file systems (such as ext4) will allocate a fixed number of inodes when they are created and cannot be adjusted dynamically.
• Application problem: The application may continuously generate small files, causing inodes to be quickly exhausted.

Solution to inodes occupying 100%

First, delete the system log files

sudo rm -rf /var/log/*.log

After deleting the system log files, it is found that the usage rate of inodes is still 100% or has only decreased a little. At this time, it is necessary to find out which directory files in the system occupy too much space.

Generally in Linux, /home stores website program files, and system program files are generally in the /var directory.

Next, check which directories are taking up too much space, check the size of the folders in the root directory, and sort them by size.

du -h --max-depth=1 / | sort -hr

Further check which directories in the var directory occupy a large amount of files based on the directory size. Follow this method to check other directories.

du -h --max-depth=1 /var | sort -hr

During the investigation, I found that /usr/local/lsws/logs occupied 9.4G. When I entered the directory, I saw an auditmodsec.log file that occupied almost 8G.

auditmodsec.log is the audit log file of ModSecurity, which records the HTTP requests and response data that trigger security rules.

Continuing to investigate, we found that the /var/lib/lsphp/session/lsphp83 directory took up too much space and had too many files.

The /var/lib/lsphp/session/lsphp83 directory is the storage directory related to LiteSpeed PHP (lsphp) sessions. This directory is usually related to LSAPI (LiteSpeed SAPI), which is the API used by LiteSpeed Web Server to handle PHP scripts.

/var/lib/lsphp/session is the directory where LiteSpeed PHP session files are stored. PHP session files are stored here and named according to the session ID.

If there are too many or too large files in the session directory, it may cause insufficient storage space. Clean up invalid session files regularly.

The inodes mentioned above occupy 100%. Most of the time, it is a problem with this directory. Due to the huge number of files in the /var/lib/lsphp/session folder, inodes occupy 100% and the hard disk cannot write files. In addition to the hard disk being full, another reason is that inodes occupy 100%

Both of these problems will cause the hard disk to be unable to write content. When the hard disk cannot write files, the web or database needs to write temporary files during operation. Failure to write temporary files will cause website paralysis, database paralysis and other problems.

Cleaning and managing session files

You can use cron to regularly clean up expired session files to prevent the directory from taking up too much space. Usually, expired session files are automatically cleaned up according to system settings, or you can control it by setting session.gc_maxlifetime.

First enable recycling management in the PHP configuration file.

In PHP, Session Garbage Collection (GC) is a mechanism for cleaning up expired session data. Depending on your environment requirements (such as development environment, high-traffic production server, etc.), you can optimize the configuration of session.gc_probability, session.gc_divisor and session.gc_maxlifetime.

The following are recommendations for optimal settings to enable recycling, depending on your scenario.

Key Parameter Optimization Explanation

session.gc_probability

Purpose: Controls the probability of triggering garbage collection each time a session is initialized.
Recommended settings:
Development environment: 1. It is convenient to frequently clean up session data to reduce debugging interference.
Production environment: 1. Make sure the garbage collection mechanism is turned on.

session.gc_divisor

Function: Together with session.gc_probability, it determines the probability of triggering garbage collection. The probability formula of garbage collection is:
Copy code
Garbage collection probability = gc_probability / gc_divisor
Recommended settings:
Development environment: 100 (i.e., 1/100 probability of triggering garbage collection).
High-traffic production environment: 1000 (i.e., a 1/1000 probability of triggering garbage collection, reducing performance overhead).

session.gc_maxlifetime

Purpose: Set the maximum survival time (in seconds) of session files. Sessions that are not used after this time will be considered "junk".
Recommended settings:
Development environment: Adjust as needed, usually set a shorter time (such as 1440 seconds = 24 minutes).
Production environment: Longer time, set up according to application requirements and user habits.
For example:
Ordinary e-commerce platform: 3600 (1 hour)
Enterprise backend system: 7200 (2 hours) or higher, such as 14400 (4 hours).

; Enable garbage collection mechanism. For production environment, it is recommended to set session.gc_probability = 1 session.gc_divisor = 1000 session.gc_maxlifetime = 14400 ; Set to 4 hours, which can be adjusted according to business needs

Alternatively, use a cron job or other tool to periodically clean up session data instead of relying on PHP's garbage collection mechanism.

find /var/lib/lsphp/session/lsphp83 -type f -mmin +240 -delete

The above command cleans up session files that were last modified more than 240 minutes ago.

Okay, here comes the Ubuntu systemInsufficient hard disk spaceThe problem of not being able to write has been solved!