High volume deployments duplicating IP addresses when using Infoblox IPAM

Problem

When doing high volume deployments we are experiencing conditions where calls to Infoblox IPAM are causing an IP address to be assigned multiple times. The occurrence level is low, but when it happens on large deployments the entire deployment can get rolled back. When resubmitting the same request it generally completes successfully.

When reading through your IPAM Debug logs you may see an error similar to the following

[E] getNextIp error reserving IP. initInfobloxIpAndProperties Error getting next IP from Infoblox. Did not get new IP address from Infoblox host <HOSTNAME> Error : unexpected Infoblox reservation status "Bad Request, response: { "Error": "AdmConDataError: None (IBDataConflictError: IB.Data.Conflict:The record ' <ENDPOINT NAME> ' already exists.)", "code": "Client.Ibap.Data.Conflict", "text": "The record ' <ENDPOINT NAME> ' already exists." }

Cause

The infoblox IP allocation is a two step process

Get the next IP Address
Reserve the next IP Address

For fast moving environments with large scale deployments, it may be possible for multiple calls to get the next IP Address before the reserve the next IP address has run causing a conflict and assignment of the same IP address to multiple machines/F5 VIPs.

Affected Versions

up to SovLabs version 2019.12.0

Workaround or Solution

Overview

The solution is to update to module version 2019.14.0 or above AND to use WAPI version 2.0 for your deployments.

Upgrade process

Request through your account manager or by raising a support ticket for a link to download module 2019.14.0 or above
Follow the upgrade steps from the SovLabs document site (See Additional information at the end of this document for links)

Selectin WAPI 2.0 for Infoblox Endpoint

In the Infoblox endpoint, make sure that the WAPI version selected is 2.0.

NOTE: This should be set by Default to 2.0, however, please check the version of WAPI supported by your Infoblox Grid (https://<infoblox address>/wapidoc/ and look at the title of the page, this should match the WAPI version running on the Infoblox device) or speak to your Infoblox administrator before making changes.
Click on Submit to apply the changes