DNS Verification times out/fails in the VRO workflow

Problem

The DNS machineBuilding or Manual Create DNS Entries workflow in the DNS module will time out after a DNS entry has been created properly. That scenario is easily remedied by setting and / or increasing the SovLabs_preDnsValidationSleepSeconds property. In some instances, a customer might note that from the command line of the vRO appliance that an nslookup will show the forward and reverse DNS entry as resolving but the workflow may be timing out saying the host is not resolvable.

Cause

A negative cache of the failed lookup can cause the JVM to believe that the newly provisioned VM is not there when it is and should be resolvable. 

Affected Versions

• 2018.3.5 and above (likely previous versions too; unverified)

Workaround or Solution

Overview

The solution is to disable caching in name resolution in the JVM.

Disable it for the entire JVM

1. 1. Log into your vRO server via console or SSH
   2. Run the following command to confirm your current settings
      1. cat java.security | grep networkaddress.cache*
      2. By default, the cache settings for JVM on vRO is set as follows:

      3. Setting name	Value
         #networkaddress.cache.ttl	-1
         networkaddress.cache.negative.ttl	10

   3. Amend the default cache settings to the following:

      1. Setting name	Value
         networkaddress.cache.ttl	0
         networkaddress.cache.negative.ttl	0

   4. Restart your vco-server

   NOTE:  A setting of -1 means cache forever, a setting of 1 →  <number> means it will cache for that amount of time in minutes. A setting of 0 means it will never cache.
   Also be aware that future updates to your vRO environment may change these settings back to their defaults.

Disable it for just the vco-server application

1. Since these options are security options passing them as an argument via the -D option is a bit different. I believe (but haven't confirmed) that adding -Dsun.net.inetaddr.negative.ttl=0 -Dsun.net.inetaddr.ttl=0 as JVM arguments defined in usr/lib/vco/app-server/bin/setenv.sh and then bouncing vco-server should do it.

Set Pre DNS Validation sleep seconds

Set the number of seconds to pause before validating DNS entries. This helps if negative caching is causing problems with DNS resoultion.

1. Login to the vRA tenant
2. Click on the Design tab > Blueprints
3. Hover over the desired blueprint name and click Edit
   1. Click on the vSphere machine component on the Blueprint Design Canvas
   2. Click on the Properties tab
   3. In the Custom Properties section:
      1. Click on the  New Property button
      2. Name field: Type in SovLabs_preDnsValidationSleepSeconds
      3. Value field: Type the number of seconds to sleep
      4. Click on the  button
   4. Click OK

Repeat Step 3 for all desired blueprints

This information has been taken from: http://docs.sovlabs.com/latest/vRA/7.5/modules/dns/infoblox-dns/usage/#set-pre-dns-validation-sleep-seconds

Amend your infrastructure DNS Negative cache settings

Most organisations run Microsoft DNS infrastructure.  The default values for a Microsoft DNS server are as follows (running powershell command: Get-DnsServerCache)

More information on DNS Negative cache from Microsoft:

Change DNS replication intervals

https://blogs.technet.microsoft.com/askpfeplat/2013/03/22/mailbag-how-often-does-the-dns-server-service-check-ad-for-new-or-modified-data/

Changing MaxNegativeTTL for your DNS

https://docs.microsoft.com/en-us/powershell/module/dnsserver/set-dnsservercache?view=win10-ps

Additional information

Discovered in ticket 1162