Musings from "the Field"

Hello

This update has been a LONG time coming, but as with most technologists in this new "zoom world", I've stared at my computer with a relentless conference call workday (which is now somehow 14-17 hours a day).

This leads me to an interesting discussion and thought process that I've been sharing with colleagues. Let's take a journey WAY back to 2019 when the world was "normal" (whatever that means). In 2019, I flew almost 200k miles for work. I was on multiple itineraries most weeks, jetting from meeting to meeting. In my professional career, we had, by all measures, a banner year. 2020 - a good year for sure, but a tremendous amount of disruption in the way work happens. Still, there was time to get "work done" between meetings, and it wasn't as bad as it could have been.

Fast forward to today, and my 30+ hours of scheduled, re-occurring meetings per week. Looking at the way we work, I'm finding that there are people who are not able to work without meeting about it. That's their thing - it's how they associate their value, and that's ok. But - when does one have time to actually accomplish work - NOT meeting about work - but real, meaningful work that results in something tangible?

In 2019, during a very hectic schedule, I would still have time in airport lounges to do expenses, administrative work, or simply reading / learning / preparing for the day or week ahead. I had time to collect my thoughts before and during flights. I landed - ready for the meetings that were ahead of me. 

A Seismic Shift

Looking back at those times, I wouldn't fly to the west coast for just one meeting. I typically would fill up my calendars by having 2-3 meetings a day scheduled. Those were heady days indeed. Only 2-3 meetings per day. Think about the times when a 2-3 or 4-5 meeting day would be a crazy schedule. Either driving to different client sites, flying to a client location, or just having a conference call (or that great new conferencing service called Zoom that everyone loved). 

In those days, I had time to prepare for my 10am meeting. We met, for 1,2,3 or even 8 hours. During extended meetings, we would have breaks throughout our time together. We would use restrooms, check email, or gather in a coffee room for casual discussion. Often times, we would, in the course of our casual discussions, come upon a solution to a problem that was raised during the formal part of the meeting. 

These casual conversations; the human connections; the on-the-fly problem solving: they're gone.

Think about what you just read - and look at the way you work today. 

The problem with problems

The real danger in the way we work today is that everything has moved from casual / sidebar conversations to scheduled 30 or 60 minute intervals of relentless discussion. Many times, it's people invading your digital space because you're 18" away from your monitor, staring at a digital meeting with cameras on, etc. It used to be a novel approach to meeting. Now, it's becoming dangerous. 

Before this new way of working, where did innovation occur? Of course, it occurred with very smart people working in labs, solving problems, etc - but where did innovation occur in your life? For me, innovation occurred in the white space between the meetings or in the breaks in the meetings. Innovation occurred in meeting new people, learning about them, learning from them, and applying that learning to your own ethos. 

The problem with the problem of incessant back to back meetings isn't the fact that we're meeting. It's the fact that all we're doing is meeting. We're not really innovating. We're not establishing new organic relationships with other people.

Conclusion

This era of technology has led to an incredible amount of change with the rapid shift by business, education, government - almost every sector there is - of people working and learning from home. Business has benefitted from this shift, because despite an increase in capital expense to support this shift, that expense was more than offset by the lack of T&E for what was a very mobile organization. 

Consider this though: 

- How does one assign a value to the loss of innovation?
- How do you quantify interactions that won't happen now because people aren't working the way they used to?
- What's the dollar value in the damage caused by creating calendar anxiety in double / triple / quadruple booking time slots?
- Has your process of personal innovation changed? How? 

I don't think business will know the full impact of this for years to come, but there will be impact in my opinion.

I would love to hear yours.

/finis

Wow

Has it really been 2 weeks since I updated this? Has anyone even noticed?

Homelab
I added a node to my homelab. A reasonably powerful PowerEdge R340. It has a single CPU socket, 64GB RAM, and. 2x 1Gb/sec ethernet interfaces. I added an older Mellanox 10Gb NIC and a Dell EMC BOSS card with 2 mirrored 240GB SSD's. The machine also has a pair of 2TB 7.2k SATA drives. 

I've installed ESXi 6.7 on the machine and it was great. Peppy performance, although because it's not the same as my HCI cluster, it can't participate in vSAN (well, not THAT vSAN anyway). I literally spun up a FoH client on the machine, which proceeded to demonstrate the drawbacks of a Xeon E 2124. Friends don't let friends crush all four cores of a 4 core CPU.

I've decided that this class of machine is probably a little too lightweight as a "tech refresh" target for my HCI cluster, but makes a really nice management and TIG stack host.

Not much of a story but it's a slow news day.

Work Hacks
I have to say that working from home with zero airport visits for 75 consecutive days has my "honey do" list well sorted at home.  For work, I've focused on optimizing the "sit in a chair with back to back zoom calls with zero time in between for 14 hours a day for 5 consecutive days every week" (read that without spaces - you'll begin to understand the insanity). I've taken the following steps:

1. Bought and installed a standing desk. (Editor's note: WHAT TOOK SO LONG FOR ME TO DO THIS). I now stand for at the very least every other meeting.  Huge difference.

2. Decided that I am only attending 25 minutes of a 30 minute call - and 50 minutes of an hour long call. I'm also ending regular calls that I host 5 or 10 minutes early depending on the length. If we're not finished, we will schedule a follow-up call. I strongly encourage you to consider this. It's helping tremendously.

3. Declining meta-meetings. Think about the words there, and you'll understand what these are. If someone must have a meeting ABOUT a meeting, I'm not coming. The exception here is preparatory meetings for important client interactions (like being added to an existing conversation thread that I had not previously been a part of) - BUT - those meetings should really only be 15 minutes or so. Give me the cliff's notes backstory - I'll figure out the rest.

4.  Take time - block time - and make it immutable. Calendars like mine mean that I'm either forced to work all weekend to produce whatever it is I agreed to during the week, or work from 6am-8am and then 7pm-10pm on some days. Mine is a global role, so I'm frequently on calls at odd hours, so that plan (early and late work) doesn't really work -  and it's horrible for work/life balance. So -  I've blocked an hour each day for work. And that hour is sacrosanct. I don't answer the phone, decline invitations, and will not "give up" that hour for anything or anyone. I may move the hour based on demands, but out of my work day, I will have one hour of uninterrupted work time.

I would love to hear any work hacks you've come up with. Work hacks are part of a global weekly call I lead - we spend at least a few minutes talking work hacks and how to navigate this new reality we're all in.

/finis

Technical debt? Or just bored?

Since the world has gone to complete lockdown, I found myself wanting to correct some technical debt in my lab (clean up cabling, etc). However, because of the lockdown, amazon took its' sweet time delivering the cool Monoprice slim cat6 cables. I finally got everything I need, and installed it today. These cables are awesome!

Instead of having a bunch of heavy cat6a cables poking out of a brushstrip, I created a patch panel on each end of the rack, and short patch cables to the switches. Can't do much about the DAC cables, but removing the cat6 cables and using the patches really cleaned it up. I also replaced the 16 port secondary management switch with a small Ubiquiti 5 port POE switch and put it in the back.

Take a look and see if you agree:

BEFORE

AFTER

/finis

vmware-statsmonitor service times out on VCSA 6.7u2

In my homelab, I've made some changes, rolled back some changes, blown up a bunch of stuff, rebuilt servers, tried Terraform to see if I could make it work (I would call it a 58% success); tried Ansible (which I am MUCH more familiar with so it was a smashing success)... All of that to say I've been mean to my infrastructure, so I decided it was time to give it a fresh start.

I used the iDRAC (Integrated Dell Remote Access Console) to mount the vSphere install ISO and reloaded vSphere on all four nodes from scratch. I also re-initalized all of my  Once that completed, it was off to the races with vCenter and Update Manager.

This is probably a really old issue, but hopefully this helps someone :)

The VCSA I installed was older, so I backed it up to my Drobo and upgraded to 6.7u2. That's where the fun begins. Before that, though - if you're new to vSphere and vCenter, take my advice (especially if you're using a homelab to learn) and enable SSH when you install the VCSA. You'll thank me later.

On reboot, the PSC appeared to struggle (well, to be frank, it failed) to start all of the services. Most troubling was that the vapi-endpoint service continually failed upon start. Here's where SSH comes in handy. I connected to the appliance and checked the services status. You COULD do the same thing in the :5480 VCSA management console, but not nearly as quickly as with SSH.

Running "service-control --status" shows what is running and what is not. As you can see from the screenshot, many of the services were not running. The quickest way to restart is to use service-control.

This KB article is where I found help.

https://kb.vmware.com/s/article/68149

This describes a different problem, but it seems that the vmware-statsmonitor service was taking more than 60 seconds to load, which caused all kinds of downstream havoc. I changed the timeout values as suggested and rebooted. 3 days later (IM KIDDING), all services were running again and all is well.

Once I changed the .json mentioned in the article, everything seems to be ok. Now I can backup vCenter again and upgrade to 6.7.u3... stay tuned...

/finis

Rethink the possible - SAP HANA at home

The first thing I did with my new lab was connect it to my old lab. (yawn - and well, the power bill at some point becomes a problem)

CAUTION: This was an experiment. It involves mostly commercially supported configurations. I'm specifically "detail light" in this post because I don't want any assumptions being made about suitability or supportability for production workloads. Nothing about my homelab is suitable for production supported workloads.

I'm involved in a pretty cool project for work around SAP HANA virtualized on VMware vSphere. As a VMUG Advantage member, I'm loving the fact that I can get non-production licenses for just about anything I can think of when it comes to building out virtual datacenters. While I am not licensed for HANA at home, I did have 30 days to mess around with it, and mess around I did.

I wish I had taken screenshots, but here's the hot take. I connected my homelab with a 64GB HANA instance (and 3 application servers) to a 64GB HANA instance I built in Walldorf, Germany. This was a mostly functional landscape, with some bogus data I generated in my homelab - and used HANA System Replication to copy the data, and Dell EMC RecoverPoint for VMs for the application servers to Germany. This copy (with my exceptional cable modem service) took almost a week to complete, but just before I shut down my HANA landscape in my homelab, I was able to literally push all of the functions of my tiny little application instance over to Germany.

VMware vSphere as my virtualization engine provided stable, consistent infrastructure. I was using vSphere 6.7 in my homelab and 6.5u3 in the Germany VxRail cluster. Of course, consideration had to be made for a variety of configuration details (HW versions for the 6.7 VMs, etc). Let me tell you how incredibly simple VMware NSX virtual networking made this - especially since the test was across such great distances.

In the projects I'm on at work, this is a very similar scenario to what we're working on, but with much bigger landscapes. For me personally, provided a proof point as to how simple SAP HANA can be when it's built on top of intelligent virtualization. The great thing about this to me is, I proved several important concepts in my own mind that before this, I had only read about in whitepapers.

I'm a very pragmatic technologist, and the ability to perform a "cloud to on-premise and back" landscape migration / replication / copy is hard. Systems must be 100% perfect across potentially hundreds of them. Complex application integrations must remain established on either side of the landscapes. Data integrity is, of course, sacrosanct. Intelligent virtualization helps customers with a consistent view, regardless of the "iron underneath". There is no better than VMware vSphere, vSAN  and NSX - especially on VxRail.  My little experiment has given me a high degree of confidence that with the right tools, time and smart people, enormous projects at enormous scale can be accomplished with the right tools. A proof point in my own mind on the work I do every day.

Interested in learning more? Find out more about VMUG Advantage here. Learn about SAP infrastructure for free here.

/finis

Homelab: Update

I promised an update when the rework was complete, and here it is.

First, the rack and stack. This took a bit of time, and a bit of patience as I sourced the best deals on my favorite auction site. My patience paid off, and here's the finished product. From top down:

Ubiquiti UniFi USW-XG (16 port 10Gb ethernet)
AC Infinity sensor based intelligent fan
Ubiquiti UniFi USW-16 (18 port 1Gb ethernet)
Leviton commercial power conditioner
Dell EMC PowerEdge R620 x4
2014 Mac Mini (home media server)
Drobo 5c (connected to Mac) with 5x 1.5TB SSD
Drobo 5n (clients, vCenter backups) with 5x 8TB HDD

All of this is connected to my Ubiquiti UniFi based system. The only component of the network that is not UniFi is a SonicWall TZ350 just before my Comcast Business CPE.

This home lab is surprisingly quiet and consumes (again, surprisingly) much less power than I originally planned for or had available when I built it.

It's all virtualized using vSphere (of course), managed by vCenter and vRealize Log Insight (VMUG Advantage is awesome by the way), and I recently installed a TIG stack (Telegraf, InfluxDB and Grafana) to see what kind of metrics I could get out of it. Questions / comments welcome.

/finis

Homelab: connectivity

I've posted a little bit about my home lab, and have recently consolidated everything into a single rack. I'm still waiting on a few components for the rack to address recirculation and aesthetics, so I'll wait to publish pictures until that's complete.

I'm a big fan of Ubiquiti UniFi products, and they suit a lab's needs well. If you're looking for a managed solution of simple layer 2 switches, go check them out. I'll publish links to the products as I describe them.

My lab is connected to my home network via the default VLAN. That's the only way into the lab. Everything else is isolated within the lab networks. My home network consists of a pretty robust firewall, and everything behind it is Ubiquiti.

I have a 1Gb fiber connection between the switch that serves my home and the lab "core" switch. The lab core is a UniFi Switch 16 XG (link) that offers (12) 1/10 Gb/sec SFP+ capable ports, and (4) 1/10 Gb/sec RJ45 ports. It is connected to a UniFi Switch 16 (link) and a UniFi Switch 8 (link).

The VLAN configuration simplifies everything in the network. Rather than worry about port assignments, VLAN to port tagging, etc; I decided to create my distributed virtual switches with the VLAN tag in the Distributed Port Groups. This way, I can maintain flexibility and simplicity. The only exception in this scheme is in the connections to the NAS platform which is connected to the Default VLAN and is accessible from both the home network and the lab network.

The server connectivity is shown to the right.

I didn't have a 24 port switch, so I decided to separate management and provisioning. There's not really a need to do that for a small environment, but I could - so I did.

The vMotion and vSAN ports are separate, and the DVS' are using separate VLANs in the connections. I could have used a LACP connection on these ports but in the interest of simplicity, these connections are separate 10Gb/sec using SFP+ DACs.

Hopefully if you're building a server based home lab, you find this helpful. Comments / questions welcomed below.

/finis

Homelab: The quest for the circle of trust

NOTICE: This contains some advanced and potentially dangerous configuration steps. If you're at all uncertain on this, please don't do it. I cannot assume any responsibility for your system or information security. This worked for me, and may introduce serious risk to your own system. Know what you're doing, and how to undo it - or don't read this.

I would like to address an issue that has come up with Mac OS Catalina (10.15.x). Besides the rapid release of fixes, etc associated with iOS 13 and Catalina, one other issue has arisen that I found the workaround for. It truly is a workaround, and appears to affect ONLY Chrome on Catalina.

NET::ERR_CERT_REVOKED

SSL certificates are a pain by any measure, and self-signing isn't working anymore on Chrome / Catalina. SO, you can either get / create your own (a massive pain), or follow the steps below.

The NET:ERR_CERT_REVOKED message can't be bypassed like some SSL errors that Chrome reports. In the case where you're on the internet or looking into a system that you're not completely familiar with, this is a good thing. However, in the case where you KNOW the system (home labs are a perfect example), this is a royal pain.

So, upon connecting to my lab post-upgrade (to Mac OS Catalina), I received this message on all of my "home" systems. Connecting via Safari worked, as did connecting via Firefox - so I knew it was (1) a certificate issue, and (2) Chrome. Here's the workaround:

1. Open the URL in Safari ex: 192.168.1.200. You will receive the usual SSL message. 

Select "Show Details"

2. Here's a little known Mac OS trick. Once you view the details of the offending certificate in Safari, you can drag the certificate to your desktop by click / hold / drag the image. You'll then have your certificate on your desktop.

3. Once it's there, open "Keychain Access" and drag the certificate into your certificate store.  Once there, you need to expand the "Trust" section at the top and then select "Always Trust". This will then allow you to connect via Chrome. PLEASE NOTE: If you are at all unsure about what you're doing here, please do not do it. This bypasses a VERY significant security feature of Mac OS and Chrome. I am only doing this because I trust these systems.

I hope this works for you. I would also STRONGLY state that this process should NEVER be used on any SSL protected connection that you are not 100% responsible for, and definitely not for something outside of your own network and control.


/finis