wires wires wires

Troubleshooting Short: Prompting NFS Reconnect with vmkping

I run a small partner validation lab these days and came across a hack worth sharing.

My NFS datastores were unpredictably disconnecting from one of three of my ESXi hosts. We’re running ESXi 5.5 and it was a new behavior.

My instinct — coming from a block protocol background — was to click “Rescan All,” but it has no effect on NFS based storage.

What my colleague Dan Perkins point me toward was an alternative route he discovered: if you run a vmkping pointed out the NFS vmkernel interface, it seems to prompt a reconnection if one is possible.

Sure enough, I ran:

vmkping -I vmk2 10.10.10.100

And the ping showed an ICMP response. Low and behold, the datastore immediately showed as mounted in vCenter!

Mind you, this did not fix the problem.

Dan ended up discovering a faulty port on the storage controller after further isolation. It was handy along the way to run vmkping as a means of isolation between no connectivity and dropped connectivity at load.

For more syntax that may be of help, check out this KB.

chutes-box

A Personal & Professional Pivot

 

I was asked a few weeks ago whether I was “all in on Sales or not,” and chose the latter.

Answering that question with honesty was terrifying and brought about a journey of career reflection, as well as a needed shift in how I saw myself.

The Plan

I joined Infinio as a Sales Engineer (SE).

While the role was a new one for me, the idea of new work has never been a concern of mine. It was actually part of the attraction.

I’ve always thought of my career aspirations as a map that will give me the “big picture” view of running an IT business. After the success of launching EMC Elect, I knew a startup was next for me. Transitioning to a SE role was a bonus: two checkboxes with one move.

Mapping out my career planning. I've always thought of this stage of my career as gathering up the big picture. Infinio gave me two checkboxes for one move.
I’ve always thought of this stage of my career as gathering up the big picture. Infinio gave me two checkboxes for one move.

It made sense from a business angle for Infinio as well.

Infinio launched with the idea of a “download and go” model of sales. It was perfect to have someone with social media marketing experience involved. That is, it made sense 7 months ago.

The Pivot

Remember, startups change all the time. The Lean Startup terminology for this situation is a pivot - a decision by leadership to shift how a product is positioned in order to gain momentum. The team behind Buffer offers a textbook example of this behavior in the wild.

Seamless installation was achievable, but results were totally dependent on sufficient workload.

This fact is no strike against the Infinio Accelerator product — I’ve spoken with half a dozen storage vendors who face this fact everyday. Performance testing is difficult and a single instance of IOmeter doesn’t cut it.

Technical details aside, the sales process required more engagement. More engagement meant longer sales cycles and thus a shift. What the pivot meant to the Sales team, and me as part of it, is that our process now looks different.  The shift made my SE role more core Sales and less cross-functional into Marketing.

Who I Am

Back to our original story.

When I was asked if I was all in on Sales, I had the think about the difference between what I could be and who I am. Here are a the two facts that made the choice clear:

  • My top three concerns related to the business are longer-term brand loyalty, communication strategy, and measuring meaningful data
  • I believe an uncomfortable amount of public honesty keeps us all improving

These made one more detail a truth. If I have to fit into a strict org chart, my career will continue in Marketing (That’s the first time I’ve written it down).

What now?

The conclusion – the scary part of this honesty – was that I would have to find a new job.

I reached out to parts of our community that may know of openings that would fit my recent discoveries. My management team at Infinio was incredibly considerate as I searched out options.

I was near a decision on a few incredible companies when I was surprised by a new offer.

The team at Infinio had reconsidered the marketing budget to allow room for me to join. When the pitch was made to me, my gut told me everything I needed to know: I still had much to build as part of this team.

Today is my second full day as a Technical Marketer for Infinio, running a partner validation lab, owning some content creation and parts of the social media strategy.

Due Thanks

The hardest part of this process was that my first reaction was one of being a failure. It has only been through the incredible mentors I’m fortunate enough to have that I pushed pass this falsity and got to the part that matters.

Thank you to the incredible friends in this community that help me think through these types of priorities. I love how we all make each other better in our careers.

 

Now, it’s time to get back to work. I have a startup to help build.

 

Moar Neckbeards

Introducing Neckbeard Influence

I outlined my content strategy in a recent post. It included a known gap in where I explore a big part of my passion for technology: the connection of people and the measurement of their influence.

where-I-blog-mjbrender

Well, I figured out what’s next for my strategy. It’s called Neckbeard Influence.

content-strategy-plus-neckbeards

I’ve thought about the scope of this project for a while now. I see Influence Marketing, from the Influencer Diet that Amy Lewis blogs about to the evolution of Community influence that John Mark Troyer continues to develop, to be the future.

My time on the Geek Whisperers has reenforced my love for Community building. There are tips on how to be an influencer, how to measure influence and how to live as an influencer that not many are covering just yet. I’d like to be one of them.

This site will remain my go-to for sharing technical content — from VMware expertise to my exploration of GitHub. You can read more on my experiences in Influence Marketing at Neckbeard Influence.

I hope you continue to enjoy both.

Time Melting

Quick Post: vCenter “Troubleshooting”

I ran into one of those vCenter Server Appliance (VCSA) situations that can only be called “curious.” The 5.5 server had been left alone for some time. Logging in was a no-go:

Damn you vSphere. Damn you.

As any reasonably respectable admin would do, I googled it. David Hill has an insightful post out there from a few years ago that’s still relevant. I ran through it. I ran into the same error afterwards.

I began toggling services from the UI at :5480, but nothing seemed to make sense.

Then I looked up. I noticed time melting off the clock as I investigated why I couldn’t administrate over this cluster. Doing some quick math, I accepted my fate and clicked the VM reset button. The Web Client worked like a charm after that.

A Disappointing & Important Conclusion

There’s nothing like vCenter to teach me important lessons of administration.

There are only so many hours in our days. As much as I would love to tell you that XYZ service had hung and you can run ABC command to clear it, I can’t. I just reboot the system.

Knowing that a vCenter reboot is reasonably non-disruptive to the data center, I accepted that time was more important than exact answers.

It’s not a satisfying observation, but it’s an example of the time management we all have to make throughout the day.

Soylent Ops

My Latest Obsession: Admins becoming Developers

More and more of what we do in Enterprise IT can be expressed as code.

While I won’t lead with click bait like Systems Administration is Dead, I do believe we’re not far from a fundamental skill set shift. It involves coding, but has less and less to do with the code.

Stay with me here. 

If you’re a ‘Technologist’ or ‘Engineer’ in anyway that connects to operations and administration, you’ve written code. You learn enough BASH shell to get the job done. You read up on PowerCLI to make tomorrow a little easier than yesterday. You hack together some HTML on a wiki to make your documentation pretty. (Slight aside: please write documentation… even if it’s not pretty.)

Ergo, you’re a coder.

What you — and I, to make this story personal — are not yet is a developer. The core differentiator, as I’ve found it, is a learned set of skills around socialization.

It’s all about how code is shared.

As I play with Chef, dig deeper into Vagrant and get more curious about how DevOps has lead such change in our industry, I notice one commonality: sharing code.

Keeping It Personal

I get more certain everyday that my core skill is Community building (with a capital ‘C’). My people are going on this journey from calculating IOPS to the automation behind DevOps. I spend my evenings and weekends making sure we’re on the same journey so I can continue to be part of the Community we’ve made.

If you’d like a tactical takeaway from this observation, learn Git. The best resource for that need is Git Immersion.

The more I explore, the more I find branching strategy as crucial to success as well.

Lastly, find a project worth forking. As a VMware vExpert, I am inspired to give back to the pyVmomi project. I hope you do as well.

It’s all a start to something bigger. I hope you’re join me for the ride.

everything-in-its-place

Where I Blog: A New Content Strategy

 

Every time I’m about to write a long post, a few mental steps are taken automatically. They help me decide whether the idea will see the light of day or whether I spend my time elsewhere.

I take this process for granted given that I’ve been posting here and there on the internet since high school.  Well, I’ve had enough side conversations about why I post here and what I do there to want to explore it visually.

So here it is: my mental map to publishing content.

where-I-blog-mjbrender

Here’s the breakdown:

  • Infinio – Contributing to my company’s blog at Infinio helps me articulate value and understand the server-side caching industry in a way I enjoy. I keep that fact in mind and aim to post every two weeks.
  • Industry – This blog has become a home base for me. The more my career evolves, the more I see the value of this site as being a technical reference as I continue to grow my expertise. My posts come in waves at times as I explore, though I am to keep it to once a week.
  • Marketing – There’s another part of my brain that I think deserves a proper place to live. It’s inspired by Geek Whisperers and continues to be a bigger part of my mind. How do businesses engage customers and employees in meaningful ways? How does storytelling and metrics impact business? There’s a big story to tell here and I plan to focus on it soon.
  • Other – Sometimes writing is like plumbing: you can’t get one idea out until you clear out the others. I find there are times when I need to dig into a concept via public dialogue and that conversation doesn’t fit nicely into my available places. Thankfully I was introduced to Medium by my friend and old colleague Diego. It’s design and tagging makes it perfect for my potpourri of interests.

There’s one major omission — I contribute on The Geek Whisperers as we take turns writing up the podcast notes. I don’t think of that process going through the same chain of events however.

Unlike Twitter, where I stand by a unified self, I find longer-form content is most meaningful when well organized. I believe every conversation has its place.

 

kant-means-to-an-end

Infrastructure as a Means, not an End

In philosophy, the term means to an end refers to any action (the means) carried out for the sole purpose of achieving something else (an end)

I run two small clusters of ESXi hosts for the SEs at Infinio.

They act as  a microcosm of real infrastructure: shared amongst the team, at the will of the network team, often capacity or performance constrained, occasionally faulty for unexplained reasons.

What’s most realistic about this side-task is that the goal of my job is not managing this cluster. My work uses the clusters as a resource for my actual work. Maybe I need to test new code. Maybe I need to document a user experience. No matter the end, running the infrastructure is not what my work is about. It’s simply a means.

This last point resonates with me most of all. As I look to experience what my customers experience, I find this setup to be a perfect way to do so. Running a small cluster in order to demo our product makes it a means to a much more business-centric end. If something is broken for a while, I mostly don’t care. If it doesn’t work  when I need to demo, I care a great deal.

That’s real world infrastructure for you, and probably is more true than we like to admit. Those who prioritize their tasks do not always fix what’s broken. They fix what’s broken before it is needed and as quickly as they can while doing it well. It’s a different mindset.

This experience teaches me two lessons I wish to share with you:

  1. If you want to sell infrastructure (that includes you, Marketing), you should run one. No matter how small. Download Autolab and get one running on your laptop.
  2. As organizational silos fall between storage, network, and compute you can imagine even less time spent twiddling with infrastructure knobs. Automation – even if it’s good enough automation – will eat up these menial tasks in even the smallest organizations. The only people left needing to know the details will be Technical Support.
heartbleed

Heartbleed-ing Your Way into Better Password Management

The Internet responded to Heartbleed. They even have stickers.*

But if you’re reading this, you’re still mulling it all over. I know I am

So let’s cut the subtly and even the low-level tech conversation. Let’s talk about why you need to act and what you need to do to act right now. 

Step 1: What Heartbleed Means To You

Here’s how I understand it:

Assume every single website you’ve ever logged into can be logged into, as you, by someone else.

If that does not scare you a little, read it again.

Ever single place you have an account on the Internet could be logged into by a total stranger without your password.

The discovery of Heartbleed shows that a fundamental building block of Internet security has not been secure for a while. It’s identify theft to the max.

It doesn’t mean your assets are in danger. Every respectable website that manages your money pays attention to your IP address and access patterns.

Step 2: Manage Your Passwords with LastPass

Let’s look at the bright side. Your password strategy sucked already. You use the same password everywhere or you forget it once a week and have to reset.

Maybe Heartbleed is a fresh start for you.

Do yourself a huge favor as you start fresh in the right direction: use LastPass to manage your password.

The software is simple and secure. No further thought be needed here. It installs per web browser you use (like Chrome, Firefox, Safari) all from the same location:

From this point forward, you have just one password you HAVE to remember. Remember your password to LastPass.

Since we’re doing this right, make it a passphrase, like XKCD explained so well. This guy made a generator for you too.

In all honesty, you could still keep your passwords memorable through a password theme. It makes it easy to remember each one you write by having a certain website-dependent structure. For example, “[website]77Wahoo!!” could be your format. You would use:

  • Facebook77Wahoo!! on Facebook.com
  • Twitter77Wahoo!! on Twitter.com
  • Google77Wahoo!! on Google.com
  • etc

Side note for those interested: I ran LastPass side-by-side with Password1 and found Password1 didn’t keep up. There are two cases that killed it:

  1. Password updates: Password1 could create duplicate entries on update and required manual intervention to fix it. LastPass has a beautiful auto-update feature.
  2. Form Filling: LastPass is a ninja on filling out forms. I haven’t written out my home address on a website since using it. Password1 supposedly has a form filling functionality, but it didn’t fill out all forms smoothly, nor handled drop downs as seamlessly as LastPass.
  3. Bonus: LastPass is free. And it’s better. And also free.

 Step 3: Change Your Passwords After You Get The Email

First you need “the email” from your vendor, like this one:

IFTTT-heartbleed-notification

IFTTT is telling you they are no longer vulnerable. That’s a green light to update your password.

If you update your password before the vulnerability is fixed, you just have a new password that’s easily bypassed through Heartbleed.

So wait for the email.

So you’ve waited for the email. You have LastPass on your favorite browser(s).

Good. Now you have to deal with Heartbleed. Now go change all your passwords.  Use LastPass to save them securely. You can even choose to auto-login on sites like this:

Auto-Login is awesome

What sites do you really have to change?

Mashable put together a list.  If you own a site, give this a read. My rule is if it would ruin your day for someone else to log into the website, change the password.

Conclusion: Is The Internet Still Safe?  

I think of the honest people on the Internet as a pack of gazelle.

Gazelle-Pack

The more noticeable you are – public figure or due to personal assets – the further you are from the center of the pack. The less noticeable you are on the internet, the closer you are to the center.

Now the other factor.

The more up-to-date you are on software updates, the more often you change your password, the bigger, faster and stronger you are. The less secure your practices, the smaller and weaker you are. 

Long metaphor short, don’t be this guy.

You on the Internet.

Will I still continue to bank, buy and build everything online?

Of course.

 

* Feel free to send me a sticker if this post is helpful!

 

multiple_paths

Technical Short: What’s with iSCSI Port Binding?

I’m learning much more about virtual networking in VMware as I work with customers as a Sales Engineer.

One checkbox I have to pay close attention to right now is called iSCSI port binding.

I love this image, compliments of Chad back in 2009.
I love this image, compliments of Chad back in 2009.

 

First – let’s define it from VMware’s very own language in KB 2038869:

Port binding is used in iSCSI when multiple VMkernel ports for iSCSI reside in the same broadcast domain and IP subnet to allow multiple paths to an iSCSI array that broadcasts a single IP address.

If you’re anything like me, you’ve noticed the checkbox for iSCSI port binding and simply ignored it.

As an SE for Infinio, I now need to verify that customers do not have iSCSI port binding enabled on the vmkernel interface they’re using for NFS traffic.

Why does a NFS-only server-side caching solution care about iSCSI port binding?

I had to find out.

Here’s what I understand so far: enabling port binding bypasses some significant vSwitch functionality. With it enabled, the vmkernel interface takes the pNIC associated with it. No vSwitch logic, which would cut Infinio out of the data path.

No data path, no acceleration.

To be honest, I still don’t understand exactly how  port binding jumps in the way. I think of it like a raw device mapping for pNICs.

The team at Infinio has tested and found – even with Promiscuous Mode enabled on the vSwitch – you cannot sniff traffic going over the pNIC taken by port binding.

Technical details admittedly unknown, VMware is very clear about what’s import to keep in mind in the case where iSCSI port binding should be used (from the same KB as above):

When using port binding, you must remember that:

  • Array Target iSCSI ports must reside in the same broadcast domain and IP subnet as the VMkernel port.

  • All VMkernel ports used for iSCSI connectivity must reside in the same broadcast domain and IP subnet.

  • All VMkernel ports used for iSCSI connectivity must reside in the same vSwitch.

one-does-not-simply-iscsi-port-binding

My simple understanding of the matter comes down to this:

  • If you’re not using multiple physical NICs for iSCSI multipathing, there’s no reason to enabled iSCSI port binding
  • If you are using multiple pNICs for iSCSI traffic, have your vmkernel interface for NFS traffic on a separate pNIC

For those more curious on how to configure port binding, Brian Tobia goes over how to setup iSCSI Port Binding on vPrimerBuild Virtual also has a tutorial, which including the CLI commands.

wire-hell_2

Technical Short: The Complication that is VMkernel Multi-homing

I ran into a strange NFS permissions error in my work lab that kept me busy for a while. Here’s what I learned from it:

There is a configuration that VMware ESXi allows, but is reasonably well documented as a no no. The technical term for it is multi-homing (KB 2010877).

It begins when someone don’t follow this statement:

Storage networking should always be in a dedicated subnet associated with a non-routable VLAN or a dedicated physical switch.

As someone who is a little rough around the edges on networking, this one got me at first. Here’s the crux:

For example, if you have VMkernel ports configured like this:

  • One VMkernel port for vMotion, named vmk0
  • Another VMkernel port for iSCSI, named vmk1

If both of these vmknics are configured to be on the same IP subnet, the VMkernel TCP/IP stack chooses one of the two interfaces for all VMkernel traffic (vMotion and NFS) going out on that subnet.

Thankfully I came across Mike Da Costa’s great write up on the topic that walks you through exactly what you can expect when you create a multi-homing configuration. These best practices, laid out by Mike, are stuck in my mind:

  1. Have only one VMkernel port per IP subnet (the only exception here is for iSCSI multi-pathing, or multi-NIC vMotion in vSphere 5.x).

  2. A dedicated non-routable VLAN or dedicated physical switch for vMotion purposes.

  3. A dedicated non-routable VLAN or dedicated physical switch for IP Storage purposes.

  4. A dedicated non-routable VLAN or dedicated physical switch for Fault Tolerance purposes.

What I take away from this experience is that VMkernel interface subnet isolation is a good first assumption to make when whipping up a design, even in the lab.

May this post save you a few minutes of your time and also inspire you to pick up Chris Wahl and Steve Pantol’s new book like I am.

[Update: 9:50am on April 8th]

I had a great response from Scott (S.) Lowe on Twitter, expanding upon the conversation started here:

vmware-twitter-scott_lowe

 

These updates are well documented in the new vSphere Networking Guide from VMware.

[//End update]