Month: July 2008

Dublin is a Small Town

1 month ago, we sent out a job offer to an Irish guy who was raised in mid-west America.  He accepted the job and was excited about it.  It was a permanent role (of value in a slowing economy) and would offer him the chance to get into server administration in an advanced infrastructure (it’s pretty leading edge).  We would expose him to lots of stuff and he was bringing some valued skills, i.e. Linux.

In the meantime we made plans for him.  He was going to have lots of interesting work, not just "pressing buttons".  I spent time preparing his laptop, getting his accounts ready in our data centre, and planning his induction.

"Heavy-D" was due in the office yesterday morning.  Nothing.   No sign of him.  I tried to ring but his phone rang out.  I mailed him to let him know that we were assuming he didn’t want to join the company.  Then he decided to call our MD.  He claimed that he got his dates mixed up.  Huh!?!?  That seems like a lack of attention to detail because it was clearly printed in his employment contract.  He was told to be in today at 9 sharp.

10:00am comes and goes and "Heavy-D" still didn’t turn up.  OK, I was done with this chump.  I wasn’t having some one start their job reporting to me like this.  I reported it to the MD.  My opinion of Heavy-D now is that he has a lack of attention to detail, is unprofessional and unreliable, i.e. I deem him to be unemployable.  The only excuse is if he fell under the #72 bus.

If you’re not from Dublin or not experienced it then here’s the crux of the story.  Dublin may have 1+ million residents but it is a small town.  No one in business is more than 1 or 2 degrees away from anyone else (like the Kevin Bacon game), e.g. if you want to know about someone then you ask around a little and you find someone who has worked with them, sold to them or bought from them.  Ruining your reputation with one person is not a good career move because people in Dublin like to talk.  Example, I had looked into "Heavy-D" through a friend.  I’m now telling that friend about "Heavy-D" who’ll probably tell "Heavy-D’s" former workmates about the story.

So, "Heavy-D", you’ve lost out on the chance to work on a super infrastructure.  You’ve also gone and shot your career in Dublin.  Maybe you should go click your heels like Dorothy and vamoose back from whence you came.

A Particularly Odd OpsMgr 2007 Problem (And Solution)

The Operations Manager 2007 agent and management server communicate with each other and perform mutual authentication using Kerberos.  They’re in the same forest and hence in the same Kerberos domain.  But what happens if you have agents outside the forest?  If you read anything from Microsoft (or the OpsMgr book I just bought) you’d be left under the impression that you must install the OpsMgr gateway.  You’d then install a custom X.509 cert (requiring a cert server running on Windows Enterprise Edition) on that machine and on the OpsMgr server.  There’s two problems with this:

  • What if the un-trusted network is a workgroup, e.g. a DMZ?  There’s no Kerberos domain for the agents on the network to authenticate with the Gateway.
  • What if you are monitoring many networks with only one or two agents on each network?  Are you going to install lots and lots of Gateways?

If you are persistent with your searches you will find that:

  • There is one mention by Microsoft in a downloadable Word document that you can install agents with the X.509 cert so that the agents can communicate directly with the management server.
  • There is an almost complete guide by Duncan McAlynn on how to install the certs using MOMCERTIMPORT /SUBJECTNAME (the subject name is the name of the cert in the certificate store).

Duncan appears to be the only person to have attempted to document this process so he deserves credit for it.  The MS documentation folks have done a poor job with OpsMgr, e.g. failing to cover this subject and failing to document complete management pack authoring.  The instructions for setting up the CA are in the OpsMgr 2007 Security Guide and Duncan walks you through installing the agent.  The only missing step is you need to install and import CA and agent certs on the OpsMgr management server(s) so that they have a means for mutual authentication with the agents.

I’d been doing this successfully on servers and then I hit one server where the agent could not use the cert.  I saw the following in the Operations Manager Event Log:

Source: OpsMgr Connector

Type: Error

Event ID: 21036

The certificate specified in the registry at HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft Operations Manager3.0Machine Settings cannot be used for authentication.  The error is The credentials supplied to the package were not recognized
(0x8009030D).

I reissued that cert, re-imported it, re-installed the agent half a dozen times.  I’d opened a call with MS (thanks to IT Pro Momentum) but the first PSS agent was not the Mae West to deal with.  He kept claiming the my CA was at fault but I knew it wasn’t – other agents were fine.  Finally the ticket got reassigned to Brian who was a pleasure to work with.

He started coming up with some new ideas straight away.  The first was maybe the cert store was corrupt.  I tried a fix for that (CERTUTIL -F -REPAIRSTORE MY “<thumbprint of agent cert>”) but that didn’t fix the problem.  Brian asked if we could look at the server together using "EasyAssist" … it’s MS’s answer to WebEx or LogMeIn so they can get Remote Assistance over web friendly protocols.  We poked around and saw something interesting.

  • The CA cert in ComputerTrusted Root Authorities was fine.
  • The agent cert in the ComputerPersonal store was fine.  The certification path was fine.
  • When you run MOMCERTIMPORT it copies the cert into ComputerOperations Manager in the certificate store.  I had overlooked this.  Here, the certification path was invalid.  Weird, because it was fine in the ComputerPersonal store.

We manually imported the cert into there and the certification path was still screwed.  We re-imported the CA cert but it was still screwed.  We re-imported the CA cert and the operations manager copy of the cert.  The certification path was fine but the agent didn’t appear to be using it.  We re-ran MOMCERTIMPORT and the certification path was invalid again.  OK … I thought we’d try this:

  • Delete all copies of the agent and CA certs from the certificate store.
  • Brian suggested restarting the cryptography and the OpsMgr Health service.
  • I went through the process of re-importing: Import the CA cert into ComputerTrusted Root Authorities, import the agent PFX into ComputerPersonal, re-run MOMCERTIMPORT /SUBJECTNAME and restarted the OpsMgr Health service.

Lo and behold … it worked!  In fact, it worked so well that we detected a hardware fault on the server that we hadn’t known about.  Sweet; OpsMgr rules!

A big "Thank You" to Brian for helping out on that one.  For the most part, I’ve always had good dealings with MS PSS agents going back to 2003.  It was good to see this one being rescued so professionally.

Official: Support for Operations Manager 2007 on Windows Server 2008

Microsoft has just given us the green light to install OpsMgr 2007 on W2008.  We’ve been waiting since February but we finally have support and as I mentioned earlier today, we saw the first few management packs hit the streets. 

It’s a complicated process to be compliant before installing SCOM 2007 on Windows 2008.  You have to first install 3 updates:

Then you need to install a hotfix rollup.

The “Mojave Experiment”

Microsoft is going to launch their fresh attempt at marketing Windows Vista tomorrow, entitled the "Mojave Experiment" (pronounced mo-have-ee – after the desert). 

Microsoft has faced a lot of negative press about Vista, right from the early days, e.g. 2003.  The hardware requirements were pretty steep when it was launched compared to what people had bought in the previous few years.  Heck, I remember reading the requirements in 2003 when we’d ordered hundreds of PC’s and thinking that we might never run Vista – it required hardware that wasn’t publicly available back then.  When it hit the market in late 2006, there was plenty of hardware on the market that wasn’t really suitable but people bought it with/for Vista and had a bad experience.

Then there’s the OS itself.  A lot has changed.  I’m not a big fan of the network management in it (I am a fan of the new network stack!).  I’m also not a fan of renaming and moving things about for the sake of it.  Some things just seem hardware for the sake of it.  The security is locked down some.  A lot of legacy applications just won’t work on Vista so that’s messed up organisations with large application catalogues.  Comments like "give out to your suppliers" or "Use compatibility toolkits" don’t go down well with those organisations because they see that as unnecessary work – XP runs just fine as is so why upgrade for what they see as an upgrade for the sake of upgrading? 

I think MS might have gotten things all messed up.  I remember hearing the story of how MS were trying to market how "pretty" Vista is.  What?  Why does a corporate want to hear about pretty?  When Vista was launched all we saw was the new <ALT-TAB> and stories about some granny in the USA who wanted to burn photos on her DVD drive.  Why would a university or bank care about that?  The home user was alienated too.  The OS changed so much that old hardware was insufficient and trusted home applications or peripherals no longer worked.  How’s a home user expected to resolve those issues?  They barely know how to use Office and print.

What ended up happening is that most business consumers shrugged their shoulders and kept deploying XP.  Home users complained about poor performance and old purchases not working anymore.  CIO’s and CEO’s happen to be home users.  These decision makers saw trouble at home and didn’t want that experience on their networks.  The jungle grapevine is powerful too.  I see it all the time at social occasions when I’m asked about a prospective new PC purchase and someone pipes in about Vista being awful.

Vista isn’t awful, but I think it’s gotten mixed up.  There are some vast improvements and some things that aren’t great at all.

So MS is going to tackle the perception that Vista is awful.  They rounded up loads of people in San Francisco who disliked Vista.  They sat them down in front of a PC, asked them to try an operating system and video recorded their experience.  Surprise!  It was Vista all along.  The videos will be played online starting from tomorrow (probably night Irish time).

Operations Manager Management Packs for Windows Server 2008

Finally!  Microsoft has released a set of management packs that include monitoring support for Windows Server 2008.  These include:

 

I haven’t seen anything on agent support for 2008 yet.  I was under the impression that a patch would be required.  Hold off on deploying agents to 2008 until you read something official from MS.

Windows Essentials Documentation

Microsoft has released a bunch of document for Windows Essential Business Server:

A Forgotten Skill: Listening

In some ways in my education and career, I’ve been lucky.  In college we did "Communications" for two years where we were forced to do public presentations and learn about how to interact with customers, etc.  I’m not saying I perfected this (because I didn’t!) but I picked up a few handy tips.  One of them is listening.  I am a geek that does get excited about my work and I love to get involved in discussing a problem.  I’ve found that there’s times where I need to force myself to sit back and say nothing.  The benefits of doing this cannot be measured.   In my first job after college, I was lucky to work with some great consultants and I got to see masters in action.  The best of these was one of the quietest people you’d ever meet; not exactly something you expect for a consultant that cost customers £1,000/day back in the mid 90’s.

I was involved in a politically sensitive project a few years ago.  I was working as a consultant on a site where an implementation project had been slow to get off the ground.  The project manager and the staff felt uncomfortable with the projects architecture and direction.  With no knowledge of the customer I was sent in to see what I could do to help.  I spent two days in a meeting with 20 or so staff members.  For the first 4 or 5 hours, I did nothing but ask short quick questions, sit back and take notes.  My notebook (which I take everywhere) was filling up fast.  This customer was complex both in terms of infrastructure and organisation.  I wrote up a summary and a general plan for how to move forward.  The feedback was positive.  In fact, they were genuinely interested.  We ended up have a series of these meetings where we would focus on different goals.  I’d kick things off and let the staff explore the issues.  My input was to either steer things back on course or to steer the exploration towards new sub-issues.  I was purely exploring the problems and the possibilities of potential solutions.  In fact, in the meetings I talked very little at all.  Most of my talking was before/after the meetings or at lunch.  I’d submit a document with my findings and proposal.  This would then be followed up by the staff (who were capable but relatively inexperienced with the technology in question) or some of our other consultants.

The key to success was forcing myself to listen.  It’s amazing what the difference is between hearing and listening.

I’m on both sides of that fence now.  I’m a service provider and a consumer of services/goods.  As a service provider I still have to listen to the market and to the individual client.  I tend to work with clients who might not be experienced in what we do so I have to get quite involved in teasing out their requirements and proposing alternate/better directions for them.  The key is in hearing their business and technology requirements and translating that into a platform that they can build on.

My experience as a consumer (for the first time since 2005) has been interesting to say the least.  For most things, I tend to be self sufficient.  Firms I work for (that let me do things my way) don’t need consulting skills the way that some others do – they save money and develop internal expertise.  But there are times where I need specialist skills.  In 2003-2005 I was lucky to work with a hardware supplier who I treated as a partner.  Our sales contact was educated about their products and I got great service from them.  Today, I work with a great network service provider who I can trust the same way.

But not everything is smelling of roses.  We’re about to make a significant hardware purchase.  Unlike most companies, this isn’t something finite with X CPU’s and Y GB’s of disk; this is just a foundation which will be followed by continual purchasing.  I’ve been leading the interaction with several hardware vendors of different types.  I couldn’t have been clearer about telling them each to listen and to work well with me on this.  I am evaluating them to see if they are firms I can work with over the coming 3 years.  It’s funny because the number of competitors whittled themselves down very, very quickly.  The losing competitors are ruling themselves out because they haven’t read emails or listened to me in meetings/on the phone.  Most salesmen seem to think that people only think in numbers.  Me?  That’s still very important but enjoying my day at work is important too.  I don’t need some person wrecking my head all day long and ruining our relationship with our clients.

A simple skill that requires no €2,000 training courses such as listening can be a major tool in your arsenal.  I struggle myself at times with it but when I force myself, things work out much better.  I’d highly recommend it to anyone that’s a service provider.  As a consumer, I’d recommend that you evaluate your service providers ability to listen too.