Problem
If a VM in vRA 8 has a very large properties stack, it can cause the deployment to become stuck during the Delete process in vRA. This is a rare case, but it could possibly happen. From our testing the limit seems to be around 500,000 characters for Keys and Values combined
Cause
If the limit is exceeded, vRA is unable to load the properties stack and the deployment will become “stuck” during Deletion or Updates. The solution is to go into the PostgreSQL OneFuse Database and remove those specific Properties on the VM.
Affected Versions
vRA 8.x
OneFuse - ALL
Considerations
This procedure has the potential to be disruptive/damaging to the database. Use extreme caution and be sure to back up your database before starting this process. If you’re uncomfortable with updating databases, contact support before attempting this.
Solution
In the normal vRA web GUI, go to Deployments and click the problematic deployment.
Ir the URL bar at the top of the browser window, copy the Deployment ID
URL bar:https://vra81-onefusetest-001.vra81cs01.sovlabs.net/automation-ui/#/deployment-ui;ash=%2Fdeployment%2Fb98e8965-a491-49d7-a379-e7f4cb9a8f62
Deployement ID:
b98e8965-a491-49d7-a379-e7f4cb9a8f62
SSH into the vRA 8.x Appliance
Run this command to gain a BASH command line on the postgres-0 container (that holds the PostgreSQL DB for vRA)
kubectl exec --namespace prelude -it postgres-0 -- /bin/bash
Switch to postgres user
su postgres
Access PostgreSQL Database “catalog-db”
psql catalog-db
Find VM id based on the Deployment id
SELECT id,name,deployment_id FROM dep_resource WHERE deployment_id = 'b98e8965-a491-49d7-a379-e7f4cb9a8f62';
Result:catalog-db=# SELECT id,name,deployment_id FROM dep_resource WHERE deployment_id = 'b98e8965-a491-49d7-a379-e7f4cb9a8f62'; id | name | deployment_id --------------------------------------+--------------------+-------------------------------------- 5eefa229-7d3b-4843-ad5b-be27420a1801 | Cloud_Machine_1[0] | b98e8965-a491-49d7-a379-e7f4cb9a8f62 (1 row)Output Properties for specific VM, based on id
SELECT properties FROM dep_resource WHERE id = '5eefa229-7d3b-4843-ad5b-be27420a1801';
Result for a normal VM:catalog-db=# SELECT properties FROM dep_resource WHERE id = '5eefa229-7d3b-4843-ad5b-be27420a1801'; properties ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ {"id": "/resources/compute/5eefa229-7d3b-4843-ad5b-be27420a1801", "name": "Cloud_Machine_1", "zone": "Cluster1", "count": "1", "image": "CentOS7", "flavor": "Small", "osType": "LINUX", "region": "SovLabs", "vcUuid": "634f89cd-bcf8-421b-bfa3-65f09557abca", "__moref": "VirtualMachine:vm-664685", "account": "vcenter01.sovlabs .net", "address": "10.30.29.252", "storage": {"disks": [{"vm": "VirtualMachine:vm-664685", "name": "boot-disk", "type": "HDD", "vcUuid": "634f89cd-bcf8-421b-bfa3-65f09557abca", "bootOrder": 1, "encrypted": false, "capacityGb": 16, "persistent": false, "independent": "false", "provisionGB": "16", "diskPlacementRef": "Storage Pod:group-p627707", "provisioningType": "thin"}, {"name": "CD/DVD drive 1", "type": "CDROM", "bootOrder": null, "encrypted": false, "capacityGb": 0, "persistent": false}]}, "cpuCount": 1, "hostName": null, "networks": [{"dns": ["10.30.0.11", "10.30.0.12"], "name": "dvs_SovLabs_329_10.30.29.0_24", "domain": "infoblox851.sovl abs.net", "address": "10.30.29.252", "gateway": "10.30.29.1", "netmask": "255.255.255.0", "assignment": "static", "deviceIndex": 0, "mac_address": "00:50:56:a5:50:e5", "resourceName": "dvs_SovLabs_329_10.30.29.0_24", "ipv6Addresses": ["fe80::250:56ff:fea5:50e5"], "dnsSearchDomains": ["infoblox851.sovlabs.net"], "assignPubli cIpAddress": null}], "__noNICVM": "true", "countIndex": "0", "endpointId": "90ee4ee7-0b83-49c0-8f6b-683f9cde7986", "folderName": "VRM-BACKUPEXCLUDED/onefusedev-001", "powerState": "ON", "primaryMAC": "00:50:56:a5:50:e5", "resourceId": "5eefa229-7d3b-4843-ad5b-be27420a1801", "constraints": [{"tag": "Cluster1"}], "provisionGB ": "16", "__dcSelfLink": "/resources/groups/9e636b5f-d9c1-452e-bc77-e5f7d1d6575b", "__flavorLink": "/provisioning/resources/instance-types/ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "endpointType": "vsphere", "resourceLink": "/resources/compute/5eefa229-7d3b-4843-ad5b-be27420a1801", "resourc eName": "malbrechtslund-885", "resourcePool": "/resources/pools/a41c1d135ff67c75-7f703c5265a63d87", "softwareName": "Red Hat Enterprise Linux 7 (64-bit)", "__computeHost": "true", "__computeType": "VirtualMachine", "componentType": "Cloud.vSphere.Machine", "datastoreName": "Nimble-DS2", "totalMemoryMB": 1024, "__blueprint_i d": "1846bc0f-e916-405a-bca4-8f6d384f4bb6", "__endpointLink": "/resources/endpoints/90ee4ee7-0b83-49c0-8f6b-683f9cde7986", "__endpointType": "vsphere", "computeHostRef": "ClusterComputeResource:domain-c26", "__deployment_id": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__imageOsFamily": "LINUX", "__placementLink": "/resources/ compute/724d7c83-039d-4a42-b602-d963a8176abb", "computeHostType": "ClusterComputeResource", "environmentName": null, "__bootDiskSizeMB": "16384", "__deploymentLink": "/resources/deployments/b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__namingTemplate": "${userName}-${###}", "resourceDescLink": "/resources/compute-descriptions/d 50fddf5-2114-4697-b4b2-b6547839ca4f", "OneFuse_Ipam_Nic0": "{\"_links\":{\"self\":{\"href\":\"/api/v3/onefuse/ipamReservations/824/\",\"title\":\"10.30.29.252\"},\"workspace\":{\"href\":\"/api/v3/onefuse/workspaces/2/\",\"title\":\"Default\"},\"policy\":{\"href\":\"/api/v3/onefuse/ipamPolicies/2/\",\"title\":\"infoblox851_i pampolicy\"},\"jobMetadata\":{\"href\":\"/api/v3/onefuse/jobMetadata/4779/\",\"title\":\"Job Metadata Record id 4779\"}},\"id\":824,\"ipAddress\":\"10.30.29.252\",\"hostname\":\"malbrechtslund-885\",\"primaryDns\":\"10.30.0.11\",\"secondaryDns\":\"10.30.0.12\",\"dnsSuffix\":\"infoblox851.sovlabs.net\",\"dnsSearchSuffixes\": \"infoblox851.sovlabs.net\",\"nicLabel\":null,\"subnet\":\"10.30.29.0/24\",\"gateway\":\"10.30.29.1\",\"network\":\"dvs_SovLabs_329_10.30.29.0_24\",\"netmask\":\"255.255.255.0\",\"endpoint\":\"onefuse\"}", "__computeHostLink": "/resources/compute/39d93f54-8bed-41db-936a-bb5731a254d8", "customizationSpec": "Linux_infoblox851 ", "resourceGroupName": "VRM-BACKUPEXCLUDED/onefusedev-001", "__cpuHotAddEnabled": "false", "__imageMappingLink": "/provisioning/resources/image-profiles/ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "__operationTimeout": "7200", "__storageReference": "Datastore:datastore-627705", "OneFuse_Trac king_Id": "9ebc9799-4123-458c-a212-35f0f1d86e85", "__isStorageReserved": "true", "__resolvedImageLink": "/resources/images/2a26420693c567970fad4cf271f7117a02a0317e", "__allocation_request": "true", "__memoryHotAddEnabled": "false", "__blueprint_request_id": "ef0ba19e-4280-4d0c-a228-2c2c5ba40311", "__vmw:provisioning:org": " ba5d937d-f531-4aa7-b8c6-b5458f68eabc", "_clusterAllocationSize": "1", "OneFuse_IpamPolicy_Nic0": "onefuse:infoblox851_ipampolicy", "__blueprint_resource_id": "5eefa229-7d3b-4843-ad5b-be27420a1801", "__imageDisksTotalSizeMB": "16384", "__composition_context_id": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__isDefaultPlacementZo ne": "true", "__blueprint_resource_name": "Cloud_Machine_1[0]", "__blueprint_resource_type": "Cloud.Machine", "__attachedDisksTotalSizeMB": "0", "__vmw:provisioning:project": "c08c5266-9131-41a5-8302-199667ddb248", "__groupResourcePlacementLink": "/provisioning/resources/group-placements/c08c5266-9131-41a5-8302-199667ddb248 -2046ba94-72f5-458c-9ecf-ab36aadd1df0", "__vmw:provisioning:blueprint": "1846bc0f-e916-405a-bca4-8f6d384f4bb6", "__vmw:provisioning:cloudZone": "2046ba94-72f5-458c-9ecf-ab36aadd1df0", "__vmw:provisioning:requester": "malbrechtslund", "__vmw:provisioning:deployment": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__vmw:provisionin g:imageMapping": "ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "OneFuse_NetworkConfigure_Logging": "true", "__blueprint_resource_constraints": "[{\"tag\":\"Cluster1\"}]", "__vmw:provisioning:flavorMapping": "ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "__bluepri nt_resource_dependencies": "[]", "__blueprint_resource_last_operation": "create", "__vmw:provisioning:blueprintResourceName": "Cloud_Machine_1", "__vmw:provisioning:constraints.placement": "Cluster1", "__computeConfigContentPhoneHomeShouldWait": "false", "__blueprint_resource_dependent_resource_ids": "[]", "__blueprint_reso urce_dependency_resource_ids": "[]", "__blueprint_resource_allocation_dependent_ids": "[]", "__ext:ComputeReservationTaskState:STARTED:SELECTED": "true", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Naming": "", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Ipam_Nic0": "", "Request_ef0ba19e-4280-4d0c-a228-2 c2c5ba40311_OneFuse_Tracking_Id": "", "__ext:ComputeAllocationTaskState:STARTED:START_COMPUTE_ALLOCATION": "true", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Naming_DnsSuffix": ""} (1 row)
For a VM with super large properties, the output will properly be handled by “more”, which will be shown in the lower left hand corner of the SSH session.
Click space (properly many times) to scroll thru the output, until you get the start of the JSON.
Copy the start of the JSON, and Paste into a good Text Editor (NotePad++, BBEdit etc etc).
And click space (properly many times) until you get to the end of the JSON. Copy the end of the JSON and paste that into the Text Editor as well.
Now clean up the super large property in the middle of the JSON in the Text Editor.
If the JSON spans multiple lines (which it properly will), clean all new lines/line changes, so that the JSON is all on one line.Update the Properties column of the Table for that specific VM
In the SSH session (still inside the PostgreSQL session), run the following command:
Descriptive command:UPDATE dep_resource SET properties = 'json value string from Text Editor' WHERE id = '5eefa229-7d3b-4843-ad5b-be27420a1801';
Actual command:UPDATE dep_resource SET properties = '{"id": "/resources/compute/5eefa229-7d3b-4843-ad5b-be27420a1801", "name": "Cloud_Machine_1", "zone": "Cluster1", "count": "1", "image": "CentOS7", "flavor": "Small", "osType": "LINUX", "region": "SovLabs", "vcUuid": "634f89cd-bcf8-421b-bfa3-65f09557abca", "__moref": "VirtualMachine:vm-664685", "account": "vcenter01.sovlabs.net", "address": "10.30.29.252", "storage": {"disks": [{"vm": "VirtualMachine:vm-664685", "name": "boot-disk", "type": "HDD", "vcUuid": "634f89cd-bcf8-421b-bfa3-65f09557abca", "bootOrder": 1, "encrypted": false, "capacityGb": 16, "persistent": false, "independent": "false", "provisionGB": "16", "diskPlacementRef": "StoragePod:group-p627707", "provisioningType": "thin"}, {"name": "CD/DVD drive 1", "type": "CDROM", "bootOrder": null, "encrypted": false, "capacityGb": 0, "persistent": false}]}, "cpuCount": 1, "hostName": null, "networks": [{"dns": ["10.30.0.11", "10.30.0.12"], "name": "dvs_SovLabs_329_10.30.29.0_24", "domain": "infoblox851.sovlabs.net", "address": "10.30.29.252", "gateway": "10.30.29.1", "netmask": "255.255.255.0", "assignment": "static", "deviceIndex": 0, "mac_address": "00:50:56:a5:50:e5", "resourceName": "dvs_SovLabs_329_10.30.29.0_24", "ipv6Addresses": ["fe80::250:56ff:fea5:50e5"], "dnsSearchDomains": ["infoblox851.sovlabs.net"], "assignPublicIpAddress": null}], "__noNICVM": "true", "countIndex": "0", "endpointId": "90ee4ee7-0b83-49c0-8f6b-683f9cde7986", "folderName": "VRM-BACKUPEXCLUDED/onefusedev-001", "powerState": "ON", "primaryMAC": "00:50:56:a5:50:e5", "resourceId": "5eefa229-7d3b-4843-ad5b-be27420a1801", "constraints": [{"tag": "Cluster1"}], "provisionGB": "16", "__dcSelfLink": "/resources/groups/9e636b5f-d9c1-452e-bc77-e5f7d1d6575b", "__flavorLink": "/provisioning/resources/instance-types/ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "endpointType": "vsphere", "resourceLink": "/resources/compute/5eefa229-7d3b-4843-ad5b-be27420a1801", "resourceName": "malbrechtslund-885", "resourcePool": "/resources/pools/a41c1d135ff67c75-7f703c5265a63d87", "softwareName": "Red Hat Enterprise Linux 7 (64-bit)", "__computeHost": "true", "__computeType": "VirtualMachine", "componentType": "Cloud.vSphere.Machine", "datastoreName": "Nimble-DS2", "totalMemoryMB": 1024, "__blueprint_id": "1846bc0f-e916-405a-bca4-8f6d384f4bb6", "__endpointLink": "/resources/endpoints/90ee4ee7-0b83-49c0-8f6b-683f9cde7986", "__endpointType": "vsphere", "computeHostRef": "ClusterComputeResource:domain-c26", "__deployment_id": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__imageOsFamily": "LINUX", "__placementLink": "/resources/compute/724d7c83-039d-4a42-b602-d963a8176abb", "computeHostType": "ClusterComputeResource", "environmentName": null, "__bootDiskSizeMB": "16384", "__deploymentLink": "/resources/deployments/b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__namingTemplate": "${userName}-${###}", "resourceDescLink": "/resources/compute-descriptions/d50fddf5-2114-4697-b4b2-b6547839ca4f", "OneFuse_Ipam_Nic0": "{\"_links\":{\"self\":{\"href\":\"/api/v3/onefuse/ipamReservations/824/\",\"title\":\"10.30.29.252\"},\"workspace\":{\"href\":\"/api/v3/onefuse/workspaces/2/\",\"title\":\"Default\"},\"policy\":{\"href\":\"/api/v3/onefuse/ipamPolicies/2/\",\"title\":\"infoblox851_ipampolicy\"},\"jobMetadata\":{\"href\":\"/api/v3/onefuse/jobMetadata/4779/\",\"title\":\"Job Metadata Record id 4779\"}},\"id\":824,\"ipAddress\":\"10.30.29.252\",\"hostname\":\"malbrechtslund-885\",\"primaryDns\":\"10.30.0.11\",\"secondaryDns\":\"10.30.0.12\",\"dnsSuffix\":\"infoblox851.sovlabs.net\",\"dnsSearchSuffixes\":\"infoblox851.sovlabs.net\",\"nicLabel\":null,\"subnet\":\"10.30.29.0/24\",\"gateway\":\"10.30.29.1\",\"network\":\"dvs_SovLabs_329_10.30.29.0_24\",\"netmask\":\"255.255.255.0\",\"endpoint\":\"onefuse\"}", "__computeHostLink": "/resources/compute/39d93f54-8bed-41db-936a-bb5731a254d8", "customizationSpec": "Linux_infoblox851", "resourceGroupName": "VRM-BACKUPEXCLUDED/onefusedev-001", "__cpuHotAddEnabled": "false", "__imageMappingLink": "/provisioning/resources/image-profiles/ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "__operationTimeout": "7200", "__storageReference": "Datastore:datastore-627705", "OneFuse_Tracking_Id": "9ebc9799-4123-458c-a212-35f0f1d86e85", "__isStorageReserved": "true", "__resolvedImageLink": "/resources/images/2a26420693c567970fad4cf271f7117a02a0317e", "__allocation_request": "true", "__memoryHotAddEnabled": "false", "__blueprint_request_id": "ef0ba19e-4280-4d0c-a228-2c2c5ba40311", "__vmw:provisioning:org": "ba5d937d-f531-4aa7-b8c6-b5458f68eabc", "_clusterAllocationSize": "1", "OneFuse_IpamPolicy_Nic0": "onefuse:infoblox851_ipampolicy", "__blueprint_resource_id": "5eefa229-7d3b-4843-ad5b-be27420a1801", "__imageDisksTotalSizeMB": "16384", "__composition_context_id": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__isDefaultPlacementZone": "true", "__blueprint_resource_name": "Cloud_Machine_1[0]", "__blueprint_resource_type": "Cloud.Machine", "__attachedDisksTotalSizeMB": "0", "__vmw:provisioning:project": "c08c5266-9131-41a5-8302-199667ddb248", "__groupResourcePlacementLink": "/provisioning/resources/group-placements/c08c5266-9131-41a5-8302-199667ddb248-2046ba94-72f5-458c-9ecf-ab36aadd1df0", "__vmw:provisioning:blueprint": "1846bc0f-e916-405a-bca4-8f6d384f4bb6", "__vmw:provisioning:cloudZone": "2046ba94-72f5-458c-9ecf-ab36aadd1df0", "__vmw:provisioning:requester": "malbrechtslund", "__vmw:provisioning:deployment": "b98e8965-a491-49d7-a379-e7f4cb9a8f62", "__vmw:provisioning:imageMapping": "ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "OneFuse_NetworkConfigure_Logging": "true", "__blueprint_resource_constraints": "[{\"tag\":\"Cluster1\"}]", "__vmw:provisioning:flavorMapping": "ba5d937d-f531-4aa7-b8c6-b5458f68eabc-9020080b-c106-4d10-a8c9-e5b353bc513a", "__blueprint_resource_dependencies": "[]", "__blueprint_resource_last_operation": "create", "__vmw:provisioning:blueprintResourceName": "Cloud_Machine_1", "__vmw:provisioning:constraints.placement": "Cluster1", "__computeConfigContentPhoneHomeShouldWait": "false", "__blueprint_resource_dependent_resource_ids": "[]", "__blueprint_resource_dependency_resource_ids": "[]", "__blueprint_resource_allocation_dependent_ids": "[]", "__ext:ComputeReservationTaskState:STARTED:SELECTED": "true", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Naming": "", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Ipam_Nic0": "", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Tracking_Id": "", "__ext:ComputeAllocationTaskState:STARTED:START_COMPUTE_ALLOCATION": "true", "Request_ef0ba19e-4280-4d0c-a228-2c2c5ba40311_OneFuse_Naming_DnsSuffix": ""}' WHERE id = '5eefa229-7d3b-4843-ad5b-be27420a1801';Now the deployment should be updated without the problematic Property, and can now be deleted in vRA web GUI.
0 Comments