Saturday, May 09, 2020

Wow

Has it really been 2 weeks since I updated this? Has anyone even noticed?

Homelab
I added a node to my homelab. A reasonably powerful PowerEdge R340. It has a single CPU socket, 64GB RAM, and. 2x 1Gb/sec ethernet interfaces. I added an older Mellanox 10Gb NIC and a Dell EMC BOSS card with 2 mirrored 240GB SSD's. The machine also has a pair of 2TB 7.2k SATA drives. 

I've installed ESXi 6.7 on the machine and it was great. Peppy performance, although because it's not the same as my HCI cluster, it can't participate in vSAN (well, not THAT vSAN anyway). I literally spun up a FoH client on the machine, which proceeded to demonstrate the drawbacks of a Xeon E 2124. Friends don't let friends crush all four cores of a 4 core CPU.

I've decided that this class of machine is probably a little too lightweight as a "tech refresh" target for my HCI cluster, but makes a really nice management and TIG stack host.

Not much of a story but it's a slow news day.

Work Hacks
I have to say that working from home with zero airport visits for 75 consecutive days has my "honey do" list well sorted at home.  For work, I've focused on optimizing the "sit in a chair with back to back zoom calls with zero time in between for 14 hours a day for 5 consecutive days every week" (read that without spaces - you'll begin to understand the insanity). I've taken the following steps:

1. Bought and installed a standing desk. (Editor's note: WHAT TOOK SO LONG FOR ME TO DO THIS). I now stand for at the very least every other meeting.  Huge difference.

2. Decided that I am only attending 25 minutes of a 30 minute call - and 50 minutes of an hour long call. I'm also ending regular calls that I host 5 or 10 minutes early depending on the length. If we're not finished, we will schedule a follow-up call. I strongly encourage you to consider this. It's helping tremendously.

3. Declining meta-meetings. Think about the words there, and you'll understand what these are. If someone must have a meeting ABOUT a meeting, I'm not coming. The exception here is preparatory meetings for important client interactions (like being added to an existing conversation thread that I had not previously been a part of) - BUT - those meetings should really only be 15 minutes or so. Give me the cliff's notes backstory - I'll figure out the rest.

4.  Take time - block time - and make it immutable. Calendars like mine mean that I'm either forced to work all weekend to produce whatever it is I agreed to during the week, or work from 6am-8am and then 7pm-10pm on some days. Mine is a global role, so I'm frequently on calls at odd hours, so that plan (early and late work) doesn't really work -  and it's horrible for work/life balance. So -  I've blocked an hour each day for work. And that hour is sacrosanct. I don't answer the phone, decline invitations, and will not "give up" that hour for anything or anyone. I may move the hour based on demands, but out of my work day, I will have one hour of uninterrupted work time.

I would love to hear any work hacks you've come up with. Work hacks are part of a global weekly call I lead - we spend at least a few minutes talking work hacks and how to navigate this new reality we're all in.

/finis

Thursday, April 16, 2020

Technical debt? Or just bored?

Since the world has gone to complete lockdown, I found myself wanting to correct some technical debt in my lab (clean up cabling, etc). However, because of the lockdown, amazon took its' sweet time delivering the cool Monoprice slim cat6 cables. I finally got everything I need, and installed it today. These cables are awesome!

Instead of having a bunch of heavy cat6a cables poking out of a brushstrip, I created a patch panel on each end of the rack, and short patch cables to the switches. Can't do much about the DAC cables, but removing the cat6 cables and using the patches really cleaned it up. I also replaced the 16 port secondary management switch with a small Ubiquiti 5 port POE switch and put it in the back.

Take a look and see if you agree:

BEFORE

AFTER

/finis


Wednesday, March 18, 2020

vmware-statsmonitor service times out on VCSA 6.7u2

In my homelab, I've made some changes, rolled back some changes, blown up a bunch of stuff, rebuilt servers, tried Terraform to see if I could make it work (I would call it a 58% success); tried Ansible (which I am MUCH more familiar with so it was a smashing success)... All of that to say I've been mean to my infrastructure, so I decided it was time to give it a fresh start.

I used the iDRAC (Integrated Dell Remote Access Console) to mount the vSphere install ISO and reloaded vSphere on all four nodes from scratch. I also re-initalized all of my  Once that completed, it was off to the races with vCenter and Update Manager.

This is probably a really old issue, but hopefully this helps someone :)

The VCSA I installed was older, so I backed it up to my Drobo and upgraded to 6.7u2. That's where the fun begins. Before that, though - if you're new to vSphere and vCenter, take my advice (especially if you're using a homelab to learn) and enable SSH when you install the VCSA. You'll thank me later.

On reboot, the PSC appeared to struggle (well, to be frank, it failed) to start all of the services. Most troubling was that the vapi-endpoint service continually failed upon start. Here's where SSH comes in handy. I connected to the appliance and checked the services status. You COULD do the same thing in the :5480 VCSA management console, but not nearly as quickly as with SSH.

Running "service-control --status" shows what is running and what is not. As you can see from the screenshot, many of the services were not running. The quickest way to restart is to use service-control.

This KB article is where I found help.

https://kb.vmware.com/s/article/68149

This describes a different problem, but it seems that the vmware-statsmonitor service was taking more than 60 seconds to load, which caused all kinds of downstream havoc. I changed the timeout values as suggested and rebooted. 3 days later (IM KIDDING), all services were running again and all is well.

Once I changed the .json mentioned in the article, everything seems to be ok. Now I can backup vCenter again and upgrade to 6.7.u3... stay tuned...

/finis

Saturday, February 08, 2020

Rethink the possible - SAP HANA at home

The first thing I did with my new lab was connect it to my old lab. (yawn - and well, the power bill at some point becomes a problem)

CAUTION: This was an experiment. It involves mostly commercially supported configurations. I'm specifically "detail light" in this post because I don't want any assumptions being made about suitability or supportability for production workloads. Nothing about my homelab is suitable for production supported workloads.

I'm involved in a pretty cool project for work around SAP HANA virtualized on VMware vSphere. As a VMUG Advantage member, I'm loving the fact that I can get non-production licenses for just about anything I can think of when it comes to building out virtual datacenters. While I am not licensed for HANA at home, I did have 30 days to mess around with it, and mess around I did.

I wish I had taken screenshots, but here's the hot take. I connected my homelab with a 64GB HANA instance (and 3 application servers) to a 64GB HANA instance I built in Walldorf, Germany. This was a mostly functional landscape, with some bogus data I generated in my homelab - and used HANA System Replication to copy the data, and Dell EMC RecoverPoint for VMs for the application servers to Germany. This copy (with my exceptional cable modem service) took almost a week to complete, but just before I shut down my HANA landscape in my homelab, I was able to literally push all of the functions of my tiny little application instance over to Germany.

VMware vSphere as my virtualization engine provided stable, consistent infrastructure. I was using vSphere 6.7 in my homelab and 6.5u3 in the Germany VxRail cluster. Of course, consideration had to be made for a variety of configuration details (HW versions for the 6.7 VMs, etc). Let me tell you how incredibly simple VMware NSX virtual networking made this - especially since the test was across such great distances.

In the projects I'm on at work, this is a very similar scenario to what we're working on, but with much bigger landscapes. For me personally, provided a proof point as to how simple SAP HANA can be when it's built on top of intelligent virtualization. The great thing about this to me is, I proved several important concepts in my own mind that before this, I had only read about in whitepapers.

I'm a very pragmatic technologist, and the ability to perform a "cloud to on-premise and back" landscape migration / replication / copy is hard. Systems must be 100% perfect across potentially hundreds of them. Complex application integrations must remain established on either side of the landscapes. Data integrity is, of course, sacrosanct. Intelligent virtualization helps customers with a consistent view, regardless of the "iron underneath". There is no better than VMware vSphere, vSAN  and NSX - especially on VxRail.  My little experiment has given me a high degree of confidence that with the right tools, time and smart people, enormous projects at enormous scale can be accomplished with the right tools. A proof point in my own mind on the work I do every day.

Interested in learning more? Find out more about VMUG Advantage here. Learn about SAP infrastructure for free here.

/finis