fatal error: error writing to /tmp/ccwAjc9Z.s: No space left on device

Today,while working in cloud server.I faced an issue.I was copying some files to s3bucket,which I have mounted by using s3fs package. One every copy command(i.e cp),I was getting error – fatal error: error writing to /tmp/ccwAjc9Z.s: No space left on device.

The problem was not new for me.I have seen this problem earlier also.The problem occur when any service/program uses the /tmp file.And the /tmp is not writable.

Reason for this fatal error
(1) The Disk space of /tmp partition might be full
(2) /tmp got a new mounted partition, which was not actually earlier. Means,/tmp itself get its partition.

Solution : For this problem ,follow the given below troubleshooting steps

Step 1: First run the df -h command. The command will show,disk space to mounted partition.
In case /tmp is full (100%). Remove the files/directory from /tmp .

rm -fr /tmp/*

After this, check again. If still issue persist go for Step 2 as given below

Step 2: Sometimes the server do very unusual behavior. Because of continuous file writing in /tmp and because of some other partition issue/ memory issue.
The /tmp will mount to spool partition overflow.

See the below given detail from my server

root@server:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         20G   13G  6.5G  66% /
udev            494M  4.0K  494M   1% /dev
tmpfs           200M  212K  199M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M     0  498M   0% /run/shm
overflow        1.0M     0  1.0M   0% /tmp
root@server:~#

Here,I found that /tmp is mounted to spool partition overflow .
In this case,simply unmount the /tmp .

Use given below two command :

umount /tmp

mount -a

Now run again df -h command.To check,all mounted partition are correctly matched . Now the problem should be solved.

NOTE:

In case,even after removing file and directory from /tmp . Your partition is still showing full disk space.
Run lsof command and get the DELETED item PID no. and kill it.

lsof | grep deleted|grep tmp|while read abc;do kill -9 $abc;done