Requests are failing - a SovLabs workflow is timing out – CloudBolt Software

Problem

SovLabs Custom Naming, or other SovLabs integrations that use locks are suddenly unable to obtain a lock and provisioning is stopped.

See the bottom of this article for a list of all the SovLabs integrations that use locks and can be affected by an abandoned one.

The following error may be seen in the vRO logs.

Error

Attempting LOCK with lock ID: 'sovlabs.generateHostname.main.lock'
[E] Error in (Workflow:Custom Naming / Lock (item16)#2) Wrapped java.lang.InterruptedException: sleep interrupted

Attempting LOCK with lock ID: 'sovlabs.generateHostname.main.lock'
[E] Error in (Workflow:Custom Naming / Lock (item16)#2) Wrapped java.lang.InterruptedException: sleep interrupted

Cause

If a previous lock didn't get cleaned up due to a failure, the next workflow run will hang, waiting to obtain a new lock. This will cause provisioning of VMs whose requests include the SovLabs module in question to stop because locks cannot be obtained.

You will see the SovLabs workflow stuck on the Lock scriptable task:

Affected Versions

Sovlabs Plugin v2019.8 and older

Workaround or Solution

Overview

If this happens, you can run the Force Unlock - Hostname Generation workflow to clear the abandoned lock. Restarting vRO will not clear a lock.

If that doesn't clear it, you can run the workflows to Display all locks and Release all locks.

If you use Release all locks, be aware that it might cause unwelcome side effects if Display all locks shows a lot of active locks. It's best to wait until Display all locks shows only the lock in question before releasing.

If you're experiencing a hung up SovLabs workflow due to a lock on something other than Custom Naming, see the list of Manual unlock workflows for other SovLabs modules below:

DRS
F5
Snapshot Management
SovLabs IPAM
Veeam (Baas)
BlueCat DNS
VM Tagging
Ansible Tower (Remove hosts)

Additional information

n/a

Requests are failing - a SovLabs workflow is timing out