CloudBolt unavailable and Apache won't start with error_log 'No space left on device: Couldn't create accept lock'

Rarely, CloudBolt's apache httpd service will fail to start with the symptoms below. This can occur for a few reasons, such as the system reaching the semaphore limit or the apache service being improperly terminated. This will result in stale semaphores being left behind after apache stops/dies, and can be resolved by purging those semaphores, increasing the semaphore limit, and ensuring that you always stop apache httpd with the service httpd stop command.

Symptoms

1. CloudBolt is unavailable with a 'Connection Refused' error in your browser, e.g.:
   This site can’t be reached
   10.0.0.100 refused to connect.
   ERR_CONNECTION_REFUSED
2. Error message in /var/log/httpd/error_log:
   

   [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.28028) (5)

3. Disk usage/free is normal
   

   # df -h

4. Apache says it's dead:
   

   # service httpd status
   "httpd dead but pid file exists"

5. Apache [re]start says 'OK' but dies with the error message in (1):
   

   service httpd [re]start

Root Cause

1. A semaphore limit has been reached.
2. If main httpd thread should receive KILL signal instead of TERM, httpd won't remove the semaphores using semctl(IPC_RMID).
3. Semaphore limits can be adjusted by echoing new values into /proc/sys/kernel/sem.

Source: https://access.redhat.com/solutions/55914
Source: https://access.redhat.com/solutions/89103

Diagnosis

1. Confirm the symptoms listed at the top.
2. Ensure apache is stopped with `ps`.
3. Check the semaphore table while apache is stopped:
   

   # ipcs -s

4. If there are a bunch of apache semaphores, purge them:
   

   # ipcs -s | awk '{ if ( $3 ~ /apache/ && $1 ~ /[0-9]/ ) system("ipcrm -s "$2) }'

5. Verify semaphore table for apache is empty.
   

   # ipcs -s

6. Restart apache, check for symptoms:
   

   # service httpd restart

7. If not fixed, read the Red Hat guide for further help or contact CloudBolt Support.
8. If fixed, increase your semaphore limits, as described below. 

Resolution

1. Check the semaphore limits
   

   # cat /proc/sys/kernel/sem
   250 32000 32 128
                ^^^128=too low

2. Edit the output from the previous command, increasing the last entry (128 to 256 in this case) and reconfigure the system with the new limit:
   

   # echo "kernel.sem=250 32000 32 256" >> /etc/sysctl.conf

3. Apply your changes
   

   # sysctl -p​

4. Verify your changes were applied
   

   # cat /proc/sys/kernel/sem
   250 32000 32 256