Space not being freed from disk after deleting a file in centos

Issue

1. 晚饭过后，同事找过来看个问题，机器磁盘满了，问有什么方案找出大文件，我说这个就只能遍历目录找了，推荐了 ncdu 这个工具给他去扫描目录
2. 同事找了一圈，没有找到大文件。我也看了下数据盘空空如也，系统 USED 97%，/tmp 、/var/log、/root 这些容易藏污纳垢的地方看了一眼，居然一切正常
3. lsof | grep delete 一看，哇咔咔[Doge]。 rsylogd 写入 /var/log/message 快 90G 了，而且 /var/log/message 是 deleted 状态的

Resolution

1. 90G 的 /var/log/message 没什么保留的必要了，直接重启了 rsylogd 进程，磁盘空间笋间释放

借鉴一下 RHEL 的排版和说明：https://access.redhat.com/solutions/2316

Graceful shutdown of relevant process

First, obtain a list of deleted files which are still held open by applications:

$ lsof | egrep "deleted|COMMAND"
COMMAND       PID    TID TASKCMD     USER   FD  TYPE  DEVICE    SIZE/OFF      NODE NAME
ora         25575   8194 oracle    oracle   33   REG   65,65  4294983680  31014933 /oradata/DATAPRE/file.dbf (deleted)

Note: check either the filesystem path within NAME field or the device number under DEVICE to match the filesystem of interest.

The lsof output shows the process with pid 25575 has kept file /oradata/DATAPRE/file.dbf open with file descriptor (fd) number 33.

After a file has been identified, free the file used space by shutting down the affected process. If a graceful shutdown does not work, then issue the kill command to forcefully stop it by referencing the PID.

Truncate File Size

Alternatively, it is possible to force the system to de-allocate the space consumed by an in-use file by forcing the system to truncate the file via the proc file system. This is an advanced technique and should only be carried out when the administrator is certain that this will cause no adverse effects to running processes. Applications may not be designed to deal elegantly with this situation and may produce inconsistent or undefined behavior when files that are in use are abruptly truncated in this manner.

$ echo > /proc/pid/fd/fd_number

For example, from the lsof output above:

$ file /proc/25575/fd/33
/proc/25575/fd/33: broken symbolic link to `/oradata/DATAPRE/file.dbf (deleted)'
$ echo > /proc/25575/fd/33

The same reason will cause different disk usage from du command and df command, please refer to Why does df show bigger disk usage than du?

To identify the used file size (in blocks), use the command below:

lsof -Fn -Fs |grep -B1 -i deleted | grep ^s | cut -c 2- | awk '{s+=$1} END {print s}'

Root Cause

On Linux or Unix systems, deleting a file via rm or through a file manager application will unlink the file from the file system’s directory structure; however, if the file is still open (in use by a running process) it will still be accessible to this process and will continue to occupy space on disk. Therefore such processes may need to be restarted before that file’s space will be cleared up on the filesystem.

Diagnostic Steps

Log Reaper will allow you to visualize and quickly narrow down the lsof data to exactly the subset you want to see