Back in the saddle

Well, I took a brief hiatus, and 4 years later I'm back. I'm going to attempt to keep this up to date with tips, tricks and general palaver...

Introduction:
I've never had a reason to run VMware vSphere from a professional perspective. Several years ago, I became very interested in it, and began running a small embedded vSphere lab in our company's lab. Aside from simply messing around, I learned slowly the how / what / why things work the way they do. Fast forward to today, and I'm running a full-on hyperconverged data center in my basement. It makes my wife very happy...

I have come to rely on a few extremely smart friends for advice, help, and so on - and they have been amazing with the amount of knowledge they're willing to share. One of them is zsoldier, a friend and colleague I'm blessed to know. Find him here: https://tech.zsoldier.com/ - and I'm very thankful for his wisdom and friendship.

Problem: vCenter Server failed with a disk capacity issue for its' database.

I had been away on business, and when I came back, I checked in on my vSphere cluster, and vCenter was up but not really. When I started digging in (https://vcenter.local:5480), I discovered that the database (seat) was out of space, and the vSphere Server couldn't start.

I started researching this, and found that this issue is pretty well documented (here: https://kb.vmware.com/s/article/2145603 and here: https://kb.vmware.com/s/article/2126276).

It literally took longer to restart services and back up vCenter than it did to fix the issue. I had considerable trouble identifying the exact volume to expand, because the default vCenter with embedded PSC installation process creates a bunch of 10GB volumes. Nothing is labeled, so seat could be on any of these volumes.

Here are the steps I took to fix the issue:

1. SSH to the vCenter VM and enabled the shell
2. Ran df -h to determine which mount point seat is on - completely useless except to see that it was, indeed, full - but helpful later
3. Opened the vSphere UI on the host I know vCenter and the embedded PSC are running on. I then edited every one of the 10GB vDisks as follows: 11GB, 12GB, 13GB, etc etc until they were all completely unique.
4. There is a shell script to autogrow the changed LUNs: "/usr/lib/applmgmt/support/scripts/autogrow.sh"
5. Ran df-h again to see which of the now unique 10GB LUNs are /dev/mapper/seat_vg-seat (it was 13GB)
6. Back at the vSphere node that is running the vCenter VM and increased the 13GB volume to 55GB (thin so who cares right?)
7. Ran the autogrow shell script again
8. Ran df -h again to confirm that seat did, in fact, grow to 55GB
9. reboot

Voila! Healthy again!

/finis