Four Roadblocks To ‘Technical’ Challenges in Agile/DevOps

I do a lot of work in automation, and oftentimes we run into roadblocks to getting all of the tasks into an automation pipeline.  A lot of the time, when going over automation steps for a project, I’ll hear people say things like, “Well you can’t do that,” or, “That’s just not possible”, and unfortunately, a lot of people take those reactions as gospel and stop going down the path without asking the most important question: “Why?”

I could recount numerous stories where I ran into blockers that were absolutely able to be overcome, but because an organization has simply “always done it this way” they were not inclined to go any further down the road to make a change.  Usually, these blockers can be throw into one of four buckets – Technology, Process, Legal, or People.  Once you know what bucket it goes in, it becomes easier to decide what’s needed to approach the problem, make a recommendation, and get it resolved.

Technology

Technology problems are, in reality, oftentimes not the root problem.  When it is, it’s usually because the application doesn’t support the methods and goals that you’re trying to achieve.

Say for instance, you’re attempting to automate the process to create a virtual machine.  And the network team issues static IP addresses and logs what is being used in a spreadsheet.  Or, you’re attempting to automate the process for onboarding new employees, but the HR department is using an antiquated system for how they track new employees and who they work for and isn’t designed to be integrated with any other application.

These are technology problems which are relatively easy to fix.  You can use existing technologies like a database to track these things, or you can get dedicated modern applications that integrate with other platforms to be able to accomplish tasks.

Really, the biggest challenge to a technology problem is whether or not your organization is willing to make an investment towards the change.  This investment may be monetary, and if it is, that can be solved with buy-in from management and budgeting.  The other investment is putting in the effort to get the work done – in either case this stops becoming a technology problem and it becomes a people problem.  More on that later.

Process

Just about every company runs into this one.  I once worked with a company that had a 27 day SLA to deploy a single virtual machine.  27 days!  When my partner and I got a hold of the process flow, a lot of it was manual process and approvals of all of the different organizations involved.  When we laid out what could be automated, we were able to boil that process down with automated checks and balances that could get the same amount of work done within a few hours.

The problem, however, was that the documented processes and procedures that the company had been beholden to required that the manager of each organization manually sign off on the approval.  When those processes were challenged, none of the stakeholders were willing to champion an effort to drive the changes necessary.

Bart Schlatmann, former COO of ING Nederland I think put it best when he said, “Agility is about flexibility and the ability of an organization to rapidly adapt and steer itself in a new direction. It’s about minimizing handovers and bureaucracy, and empowering people.

You are as agile as your most rigid process.

Legal

Legal challenges don’t often come up, but when they do, they often can be solvable.  The most frequent I’ve seen is with open-source licensing.  Oftentimes, organizations will just issue a blanket “no” across the board for open-source software.  This is a mistake.

Yes.  Understand the licenses involved for using open-source.  Yes.  Ensure your company is reasonably covered when using it.  But if you are taking open-source entirely off the table for use, you’re hamstringing yourself and your initiatives.  No one moves to innovate and improve product faster than the open community.

For anything outside of that, the important thing to remember is to ask questions.  Understand the legal issues at hand.  Most importantly, understand what would be needed to stay within the legal lanes and still get the job done.

People

The most common, and the hardest to solve.  For the most part, people can be innovative, creative, and passionate about their work.  But they can also be territorial, stubborn, and unyielding to change.

Everyone should recognize this; not only of their peers, but of themselves as well.  Change is scary.  Habits are hard to break.  And when someone requests a change to something that you’ve worked so hard to build, it can be easy to say no.  Recognizing this in yourself is equally important as recognizing it in your peers.  Instead of a knee-jerk reaction, take a moment.  Think about the reasoning for the change and ask questions.  Understand the problem that the requester is trying to solve.

Perspective is valuable.  It’s based on our experiences, and thus everyone has a different take on things.  All perspectives are equally important.

People that are willing to embrace innovation and change are the ones who will enable an organization to move forward.  Those who fear change, and refuse to challenge their own ideals become anchors.  They will slow a company down, if not bring it to a grinding halt in terms of progress.

In this day and age of disruptors who surpass industry juggernauts in market share and value, it’s important to understand and evaluate the challenges you have to move your organization forward.  Keep your company flexible and adaptable to change.  Understand processes that are holding you back and adopt new ones to address the same risks but allow you to move forward.  Evaluate your people.  Coach them to ask questions of themselves as much as their peers, and reallocate those who are unwilling to adapt.

Finally, ask questions.  Don’t take ‘no’, or ‘we can’t do that’ as the final word.  Challenge the response with a question.  Understand the reasoning behind the response.  ‘No’ isn’t a solution.  It’s a roadblock.  Anything is solvable.  But you need to understand the why before you can get to how.

Three Things That Make a Good Technical Candidate

I have this conversation a lot with managers and colleagues in the industry.  It usually starts with, “what do you look for in a candidate when you’re doing an interview?”

Oftentimes, I’ll hear people talk about how their company only hires people with college degrees, or that they look for the number of certifications that the candidate has.  Usually when I hear these two things, the first thing I do is sigh.  While I will never beget someone for taking the time, effort, and financial expense to acquire a college or university degree, it also doesn’t necessarily give them the technical know-how to perform the kind of work we do.  Nor do certifications necessarily reflect technical knowledge.  Most certification exams that I’ve seen don’t actually test your practical knowledge of how something works, but rather your ability to memorize a book.

When I was going for my aircraft mechanic certifications, I was tested on three areas – General, Airframe, and PowerPlant.  Each area required three exams; an oral (spoken), practical, and written test.  Nine exams altogether.  The first thing we learned in mechanics’ school was that when we received these certifications, the only thing that we were really certified to do was learn how to do our job.  Until I gained enough experience performing tasks with someone inspecting my work each step of the way, there was no way I was certified to return an aircraft to service.

The same really goes in our industry.  I am sure as hell not going to turn some person fresh out of school loose on my production environment and simply trust them to do work.  So what really matters is their experience in the field, not the schooling.  But what really makes an exceptional candidate can be boiled down to three things – Critical Thinking Skills, Initiative, and Enthusiasm.

Critical Thinking Skills

When I’m interviewing a candidate for a position, I’m not barraging them with questions to troubleshoot some obscure issue.  That’s just uncomfortable for both you and the candidate.  What I do is give them a scenario and ask them to think the process through to the logical conclusion.

What’s the difference you may ask?

Troubleshooting scenarios usually consist of a lot of back and forth with the interviewer saying, “Okay, that didn’t work.  What next?” and then the candidate following up with next step questions.  It’s annoying and uncomfortable.  Often, this line of questioning devolves into a power struggle between the interviewer and candidate.  Stop it.

Instead, give them a problem scenario and ask them what they would do to investigate it.  Question them on why they chose the particular path they did.  Find out the reasoning behind it.  I want someone that can think about the task logically and follow the thread to it’s conclusion.  Ultimately, we are all going to run into issues that are going to be beyond our experience, and following a checklist isn’t going to help.  I need someone that will work methodically and logically and outside of the box.

Initiative

This is where the troubleshooting scenario also tends to fall short.  Oftentimes it follows a scripted, linear scenario.  This only proves that the person I’m interviewing is going to follow a scripted, linear process.  On top of being boring, it doesn’t help in those real-world scenarios that aren’t documented.

Don’t ask, about some obscure error code with the expectation that they’re going to blurt out what it means.  Ask them about situations that will show how they will go about finding the information.  A very basic question that usually weeds out a lot of candidates for me is, “If you’re looking through a co-workers code and you see a PowerShell command that you’re not familiar with, how would you find out what it does and how it works?”

I actually had a guy tell me that he would just run the code.  Awesome.  I want you looking at stuff in my production environment.

On the other side of the coin, I was once given a scenario question where I received an error in SCCM, and they asked me what log file I would look in to investigate.

Dude, really?  Do you even know how many log files are in an SCCM server depending on the role?

When I told him that I’d look up the error online first to nail down which log to look at, he wasn’t impressed with my answer, and I wasn’t really impressed with his response.

Enthusiasm (Willingness to Learn and Share)

Finally, enthusiasm.  You don’t want someone who’s going to come and and just do a job.  Our industry is changing every day and it affects every aspect of what we do.  There are going to be new technologies and challenges.  Our way of approaching problems is changing as we implement new things.  You want someone who’s always looking to the next thing, learns something new, and shares it with their team.

Ultimately, if they aren’t willing to learn something new and share it with others, they are going to slowly become a liability that holds the organization back from growing.  If they can’t go beyond a script to troubleshoot a problem, I can’t rely on that candidate to handle an issue that isn’t documented.  And finally, if the candidate can’t think logically, I can’t trust them to make the right decisions based on the available data to get the job done.

NASA, SpaceX, and Showmanship

Tuesday marked the maiden voyage of SpaceX’s Falcon Heavy.  Their heavy lift rocket system to compete with the United Launch Alliance’s Delta IV Heavy and Vulcan, Blue Origin’s New Glenn, and NASA’s upcoming Orion replacement – the Space Launch System.  The launch went off with tons of fanfare as millions of viewers hopped online to catch the live stream as Elon Musk’s company fired 27 engines across three boosters to launch his little red Tesla Roadster on an orbit that will cross the orbit of Mars.

Elon Musk just shot a car into space out towards Mars.  With a mannequin in the driver’s seat wearing a production model of SpaceX’s space suit, and the radio blaring David Bowie’s Space Oddity on repeat in the vacuum of space.  Oh, and two words displayed on the radio: “Don’t Panic!”

That’s an attention grabber, and NASA could learn a lot from what SpaceX does to keep the attention of the public.  I’m not saying that NASA needs to partner with a car company and start filling the solar system with an automotive museum.  But aside from the occasional two minute blip on the news, what do we really hear about them?

NASA does some amazing things, and they’ve managed to swipe a few headlines here and there.  NASA has been delivering some gorgeous photos of Jupiter a la Juno.  But it hasn’t really seemed to resonate with the public like a SpaceX launch.

Just about every single flight of the Falcon series of rockets draws a massive audience to their online webcasts.  Whether they’re deploying a Korean Communications Satellite or a Top Secret government satellite that may or may not have failed in flight, SpaceX gets hundreds of thousands (if not millions) of views on every launch.  Their lowest viewed video of the last year appears to be the EchoStar XXIII Technical Webcast at 111,000 views, while NASA has only had 21 of their 300 videos in the last year exceed that number.

So what is NASA’s problem?  Well…it really boils down to showmanship.  If you can put on a show – make a spectacle of your accomplishment – you can capture an audience.  But to do that, you need a critical component that NASA sorely lacks; people.

Elon Musk has become almost a household name alongside that of Bill Gates and Steve Jobs.  He’s in the forefront of every major piece of news that comes out about SpaceX and Tesla.  He gets peoples’ attention with crazy stunts like shooting his car into space, or taking pre-orders for personal flame throwers, and people are loving it!  There is even a weekly YouTube show dedicated to all things Elon Musk.

Seriously. This.

Astronauts Scott Kelly and Chris Hadfield did amazing work for NASA from this perspective.  Their regular interactions on social media while they were on the International Space Station kept the attention of the masses by interacting with people on Earth.  Hadfield’s Space Oddity video alone has grabbed over 38 million views on YouTube.  And Kelly’s comedic skit on the ISS with an ape suit grabbed quite a few headlines.  But stunts like these don’t seem to happen very often with NASA, and I think it really hurts them in the public eye.

NASA has always struggled with keeping the attention of the public when things become ‘routine’.  It’s well documented that the live broadcasts from Apollo 13 (prior to the emergency) weren’t being aired because spaceflight had become boring to the public.  The same thing happened with the Space Shuttle.  Granted, their publicity stunt in 1986 ended in tragedy, but had they figured out how to keep the attention of the public in the first place, I wonder if Challenger would have actually been flown that day.  NASA had been under intense pressure to get a highly publicized flight off the ground after a number of delays.  If they already had a level of attention and interest from the public that they were seeking with this flight, you have to wonder if they might have leaned more towards safety and prudence.

SpaceX’s showmanship isn’t limited to Elon Musk alone.  Watch their webcasts.  They have an opening montage with cool music.  Announcers (plural) keeping you informed of what’s going on with the rocket, and how it works (in layman’s terms).  At launch, you can hear the crowd at SpaceX’s mission control cheering at every stage of flight.  Hey, the employee cheering may or may not be staged, but it keeps you engaged and on the edge of your seat!

With NASA, you got a wide shot of the rocket and absolute silence with the exception of the occasional communications callout.  And maybe a monotone voice explaining the dry details of what was going on or what would happen next.

NASA needs to get people engaged consistently.  They need to establish familiar personalities that interact with the public on a regular basis.  I would go so far as to say that Destin Sandlin of Smarter Every Day would be perfect for the job.  He’s personable, understands the underlying science to a lot of things related to spaceflight (because he works in the industry), and knows how to keep people engaged.  If you stuck him in orbit on station for a year, you would have an audience.  Keep him at mission control after that, or maybe vlogging about some of the cool experiments and projects that NASA is working on, and you’d have engagement well beyond that.

SpaceX is making huge strides in innovating the aerospace industry and they’re taking the public along for the ride – quite literally.  NASA needs to figure out what it wants to do.  They’re going to have to eventually choose to either leave the innovation and spaceflight to organizations like SpaceX or Blue Origin and become a regulating agency; or they’re going to have to really start working on their audience problem and find some people to bring some personality to their mission.  If they don’t, I fear that Congress will eventually make the decision by budget.

If you didn’t catch the maiden voyage of SpaceX’s Falcon Heavy rocket, you missed out on quite a show.  Fortunately, you can catch the recording here:

Using Azure to Keep Moving Your Career Forward

As admins and engineers, it’s often left on us to gain the knowledge and experience needed to further our careers.  Even if we’re lucky enough to work for a company that will pay for some training, it’s often directly related to the position that you’re currently working.  For large environments with silo’d IT groups, this means that you’ll likely get trained on one or two products that you already have experience working in.

Sure, getting that training is cool, and hopefully you’ll learn some things that you didn’t know before (and hopefully make you more efficient), but what if you want to expand your horizons to move to a different position or just gain a broader understanding of how everything works?

WhyAzure

A lot of the talk these days is about cloud, and if you’re in a Microsoft environment, that means Azure.  Furthermore, PowerShell has moved beyond a simple system management tool, to a tool for handling configuration management and deployment of applications among other things.  Desired State Configuration, released with PowerShell v4.0 just 16 months ago, has already celebrated it’s 10th resource kit wave.  PowerShell 5.0, slated to launch with Windows 10, will provide application deployments using OneGet.  Even though I’ve been pretty hot and heavy on the PowerShell track for a while now, I’m still feeling pretty far behind.

PowerShell is a management platform that has absolutely taken off in the last couple of iterations, and there’s no indication from Redmond that it’s going to slow down anytime soon.  New PowerShell cmdlets are made available in wave updates to Windows, and other applications are following suit by releasing new or enhanced product-specific cmdlets in cumulative update releases.  So for those that haven’t started learning PowerShell, you might want to consider taking your IT education into your own hands.

But let’s step away from the PowerShell discussion for the moment and talk about those other applications and operating systems themselves.  Companies are increasingly relying on us to be knowledgeable about many new apps and server platforms the moment they hit RTM.  But getting VMs spun up in a lab or non-production environment, and scheduling time during work hours is pretty close to impossible.  But of course, how do you begin to overcome those challenges?

For a long time, I used a home PC as a lab environment, leveraging Microsoft Virtual PC, and later, Hyper-V.  But I find that as I do more presentations, I need more flexibility than carrying around a massive PC with me, and my Surface just doesn’t have enough power to support five or six VM instances.  Even if you take presentation out of the equation, there’s still the question of managing legal server licenses and software, or tearing down and rebuilding an environment every 90 days if you’re using evals.  So I decided to try out Microsoft’s Azure service to see what it could offer me from a learning perspective, as well as a presentation point.

The Pros

Well first off, you’re going to be directly learning a technology that you’re going to have to eventually learn to deal with.  Whether it’s in Microsoft’s cloud, or your own internal one, my gut tells me that Azure is going to be the management platform for Windows Server for many moons to come.  On top of that, you’ll have access to the latest versions of server, and many applications depending on what subscription level you’re running, all without having to manage as many licenses as you were previously dealing with.

Want to try something new?  Spin up a new VM in minutes.  If it explodes the machine, you can delete it and start all over again without having to build a machine from scratch.  Getting underway is super fast and easy.

You’re also dealing with products that those exam prep books are talking about!  You can build up your environment along with the study guide and get underway to your next certification in hopefully minimal time.

WhyAzure2

Finally, you can access it pretty much anywhere you have an internet connection.  So if you’ve got a presentation to head out to, or you’re on the road and want to test out a theory or new configuration, you can do so through RDP or PowerShell Remoting.

The Cons

It costs money.  Not a lot mind you; especially if you’re careful.  Microsoft basically charges you for what you’re using, so if you shut down your VMs when they’re not in use, it won’t cost you as much.  Though a couple of times, I have managed to leave a large number of VMs on overnight, and that wound up costing me about $6 for the overnight mistake.  But if you’re careful, you can keep the bill under $50 a month USD.

It’s internet-based.  So if you’re unable to access the internet from your location (or they have a slow connection), you can’t get to your environment.  From a presentation perspective, this is becoming less and less of an issue, but still something you’ll want to check in on when presenting at a new locale.

In the End

It’s cool to say that you’ve built out your own infrastructure from scratch and blah blah blah…  Actually, who are we kidding?  Nobody thinks it’s cool or fun; even other people that do it themselves.  It’s a lot of work to maintain, and a total pain in the ass to lug around!  The cost of an hour or two of your monthly salary can save you tons of headaches and give you a foundation of new technology that everyone’s talking about.  It might not be free, but it’s been my experience that if you’re not willing to make a financial commitment to your own career to get further ahead, then maybe it’s time to consider investing in a new direction.

A PowerShell Life: Moving To Server Core

powershell3We’ve all had a good, long time to settle into the daily routine the goes with being a Windows Server administrator or engineer.  We’ve had time to learn all of the nuances behind the graphical user interface.  We know where to go to get our management consoles, what commands to run to make the tweaks we need and get the things we need to get done completed.  And I, like many of my counterparts, looked at Server Core when it was first introduced with Server 2008 with apprehension.  Why would I leave the confines of my comfortable UI for something so stripped down?  What was this BS that Microsoft was feeding us?  I’m not a UNIX guy!

Sure, you have a smaller foot print for your OS when it comes to storage; and yeah, it uses less compute and memory too.  Sure, it requires less patching, and fewer reboots.  But what about my tools!?  What about my Start button!?

Truth be told, ever since I started working in PowerShell, I’ve found that I’ve used the administration consoles less and less.  Oftentimes, working in the console requires a one-at-a-time mentality, whereas with PowerShell I can manage multiple machines at once.  Even remoting to a server’s desktop was far too time consuming, especially if I could just get the information or execute the command I wanted through PowerShell Remoting.  As a ConfigMan engineer, I’ve even challenged myself to move outside of the box and create my own modules to service SCCM Clients through PowerShell so I don’t even have to open my console to do any troubleshooting, and can share these tools with my colleagues so they too can troubleshoot without directly interacting with a console or the client.

So when it came to be time to start taking a good hard look at my environment, and the footprint it was leaving in virtualization resources, it was time to take a look at myself as an admin and engineer.  To get a little philosophical – we, as architects and stewards of technology, can view the design of our environment as a mirror reflection of our own selves.  Our knowledge and experience makes up just as much of the environment as the best practices and company directives that guide our hand in creating the infrastructure that will support our users for the next life cycle.  Once I realized that the graphical interface served no purpose other than being that comfortable space that I’ve known for the last 18 years, it no longer made sense to incur additional costs in resources and a larger attack surface in my environment.

Of course, we have our ways of easing into the water.  I’ve been testing the applications I’m responsible for in a core instance for a few months now.  Some applications made the cut and were able to be used effectively in Core, and some just weren’t.  When I was finally ready to take the leap with those that would, I was able to use a simple PowerShell script that removed the GUI and rebooted the server.

Piece of cake.

It’s since been a couple of weeks since that implementation, and so far so good.  If anything, the most notable change that I’ve encountered is with myself.  I wrote a lot of scripts when I began learning PowerShell, and now that I’m working with core, I’m writing new scripts more frequently and refining old ones as my knowledge grows.  My environment is using fewer resources, and I’m becoming a smarter PowerShell Administrator.  I’d say that’s a win-win scenario.