Today,while working in cloud server.I faced an issue.I was copying some files to s3bucket,which I have mounted by using s3fs package. One every copy command(i.e
cp),I was getting error – fatal error: error writing to /tmp/ccwAjc9Z.s: No space left on device.
The problem was not new for me.I have seen this problem earlier also.The problem occur when any service/program uses the /tmp file.And the /tmp is not writable.
Reason for this fatal error
(1) The Disk space of /tmp partition might be full
(2) /tmp got a new mounted partition, which was not actually earlier. Means,/tmp itself get its partition.
Solution : For this problem ,follow the given below troubleshooting steps
Step 1: First run the
df -h command. The command will show,disk space to mounted partition.
In case /tmp is full (100%). Remove the files/directory from /tmp .
rm -fr /tmp/*
After this, check again. If still issue persist go for Step 2 as given below
Step 2: Sometimes the server do very unusual behavior. Because of continuous file writing in /tmp and because of some other partition issue/ memory issue.
The /tmp will mount to spool partition overflow.
See the below given detail from my server
root@server:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/vda 20G 13G 6.5G 66% / udev 494M 4.0K 494M 1% /dev tmpfs 200M 212K 199M 1% /run none 5.0M 0 5.0M 0% /run/lock none 498M 0 498M 0% /run/shm overflow 1.0M 0 1.0M 0% /tmp root@server:~#
Here,I found that /tmp is mounted to spool partition overflow .
In this case,simply unmount the /tmp .
Use given below two command :
umount /tmp mount -a
Now run again
df -h command.To check,all mounted partition are correctly matched . Now the problem should be solved.
In case,even after removing file and directory from /tmp . Your partition is still showing full disk space.
lsof command and get the DELETED item PID no. and kill it.
lsof | grep deleted|grep tmp|while read abc;do kill -9 $abc;done
Thank you for this post. I had this exact issue today, and every other post talked about iNodes, this is the right answer though, not sure what could’ve caused this wrong mounting, but that was it.
sharad chhetri says
Thank you for providing feedback. It really helps other to try this method too.