ATCMF - Unable to create a vCO endpoint of type 'atcmf'. Reason: '400'

Problem 

You are unable to create an ATCMF endpoint and you receive the following error:

Cause

Unable to create a vCO endpoint of type 'atcmf'. Reason: '400'

• You have installed the Ansible Tower CM Framework (ATCMF) plugin in vRO but have not installed the license
  • When you create the endpoint the Username and Password fail to validate, but you delete the username and enter it again and it validates
  • There are several times you go back and forth to get a username on different pages to validate
• You go to finish and get the error
  
• You have already installed the license but only did a services restart
  • You may need to do a complete restart of your vRO servers
• In a clustered vRO environment you may need to restart each server for this to take affect
  • If the restart does not resolve this issue, you can install the license against each vRO server

Affected Versions

• Only affects the Ansible Tower CM Framework plugin (not the SovLabs Ansible Tower module)

• All versions are affected as this is a configuration error

Workaround or Solution

Overview

The following instructions provide the steps on installing the license against the ATCMF plugin in vRO.  If you have already installed the license and restarted services and still have this error then I would restart your vRO servers.

Installing the license in vRO

1. Log into your vRO designer and go to Ansible Tower → Add SovLabs Ansible Tower License
   
   
   
2. Right click the workflow and select Start Workflow
   
   
   
3. On the screen, enter your tenant name (1), In License Key, Paste in your SovLabs License key and click out of the License Key area (2). Ensure that License Status shows "Valid License, click Submit to install" (3) and click Submit (4)
   
   
   
4. Once licensed, you may have to restart your vCO Services or the vRO Server (NOTE:  Please be aware that restarting your vRO services / Server may impact builds / in the process builds and will require some planning / downtime to complete, especially if using vRO embedded in vRA)
   1. Restarting vCO Server service via ssh

      1. service vco-server restart

   2. Restarting vCO Configurator services

      1. service vco-configurator restart

5. After restarting the services, wait up to 10 minutes (Depending on your setup ie; stand alone / clustered and test).  If you still receive the 400 error, restart your vRO servers and wait up to 10 minutes. 
6. You should now be able to create your vRO endpoint without any errors.

Restarting your vRO servers

1. If using external vRO servers, perform reboots of all your vRO servers. 
2. If using  the Embedded vRO services (embedded in vRA) follow VMware's restart procedures in the Product Guide for your vRA/vRO version.  See link under Additional Information below.
3. For standalone wait a few minutes before testing ATCMF
4. For clustered vRO, wait till nodes are all in sync (Generally wait 10 - 15 minutes)
   1. If the sync does not bring across the license you will have to go through the license installation process on each vRO server.

NOTE:  Please be aware that restarting your vRO services/Servers will impact builds in progress and will require some planning / downtime to complete

Additional information

SovLabs adding license to Ansible Tower : http://docs.sovlabs.com/latest/vRA/7.6/modules/configuration-mgmt/ansible-tower-cm-framework/install/#add-license

VMWare product documentation for vRA/vRO 7.5 (restart procedure):  https://docs.vmware.com/en/vRealize-Automation/7.5/com.vmware.vra.prepare.use.doc/GUID-1297E02C-DCE5-4AA0-800B-4E24CD8BCE0B.html