File System aparece lleno pero no tiene ficheros

En algunas ocasiones nos damos cuenta que en un servidor tenemos problemas de espacio y al realizar un du dentro del file system indicado realmente no esta ocupando lo que dice. Para solventar esto vamos a realizar una serie de comprobaciones para ver que proceso esta ocasionando el problema .

 

Entramos al servidor  y comprobamos al realizar un df que esta ocuado gran cantidad del espacio.

rokitoh@redorbita:/# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/dm-0 12G 2.0G 9.3G 18% /
/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp
/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt
/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home
/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr
/dev/mapper/vg_display-lv_var 13G 12G 1G 16% /var

Accedemos al file system y comprobamos que realmente no tenemos 12GB ocupados tal y como nos indica el DF

rokitoh@redorbita:/var# du -hs *

4.0K agentx
1.4G app
9.9M backups
202M cache
4.0K crash
339M lib
4.0K local
0 lock
14M log
16K lost+found
4.0K mail
4.0K opt
0 run
36K spool
12K tmp

Vemos si se ha quedado algun proceso colgado de borrado en cache:

rokitoh@redorbita:/var# ls -l /proc/*/fd/* | grep deleted | grep /var | awk ‘{print $9}’ | xargs -i ls -lhL {}

-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/12673/fd/7
-rw-r–r– 0 tomcat7 tomcat7 2.7K Jul 24 13:09 /proc/1521/fd/12
-rw-r–r– 0 tomcat7 tomcat7 0 Jul 24 13:09 /proc/1521/fd/13
-rw-rw-rw- 0 root root 11G Aug 4 11:02 /proc/15247/fd/7
-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/16
-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/3
-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/5
-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/7
-rw-rw-r– 0 root root 22K Jul 24 21:14 /proc/1680/fd/9
-rw-r—– 0 root root 151 Jul 24 13:09 /proc/1/fd/11

 

mediante el comando lsof vemos cuantos procesos de borrado se han quedado colgados:

rokitoh@redorbita:/var#lsof | grep «/var» | grep deleted
init 1 root 11w REG 252,4 151 651557 /var/log/upstart/systemd-logind.log.1 (deleted)
java 1521 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1522 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1522 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1523 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1523 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1524 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1524 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1525 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1525 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1526 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1526 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1527 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1527 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1528 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1528 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1529 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1529 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1530 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1530 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1531 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1531 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1532 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1532 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1533 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1533 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1534 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1534 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1535 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1535 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1536 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1536 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1541 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1541 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1762 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1762 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1763 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1763 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
java 1521 1764 tomcat7 12w REG 252,4 2720 651535 /var/log/tomcat7/catalina.2016-07-24.log (deleted)
java 1521 1764 tomcat7 13w REG 252,4 0 655268 /var/log/tomcat7/localhost.2016-07-24.log (deleted)
ruby 12673 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby-time 12673 12675 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12676 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12677 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12678 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12679 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12702 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12726 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12727 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12728 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12729 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12730 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12731 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12732 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12733 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 12673 12734 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby-time 15247 15249 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15251 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15252 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15253 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15254 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15271 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15286 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15287 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15288 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15289 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15290 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15291 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15292 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15293 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)
ruby 15247 15294 root 7w REG 252,4 10806130794 521464 /var/app/log/production.log (deleted)

 

Como podemos observar el tomcat y una aplicación estan bloqueando el borrado de multiples logs.

Reiniciamos el servicio tomcat y el aplicativo y comprobamos que se libera el espacio.

rokitoh@redorbita:/# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/dm-0 12G 2.0G 9.3G 18% /
/dev/mapper/vg_display-lv_tmp 5.4G 12M 5.1G 1% /tmp
/dev/mapper/vg_display-lv_opt 10G 9.0G 520M 95% /opt
/dev/mapper/vg_display-lv_home 5.4G 991M 4.2G 20% /home
/dev/mapper/vg_display-lv_usr 9.3G 3.2G 5.7G 36% /usr
/dev/mapper/vg_display-lv_var 13G  2.0G   11G  16%

 

un saludo.

:wq!