In case you don’t know, Windows Weekly on the TWiT online channel is probably the biggest Windows “podcast” (it’s also a live show) on the net. It is hosted by Leo Laporte with top tech journalists Paul Thurrott and Mary Jo Foley. Last night, they discussed the recent patching issues and Mary Jo brought up my advice to delay deploying updates for 1 month – I normally try to watch live but I listened in the car this morning.
Go to around the 34 minute mark to hear for yourself
Leo didn’t like my advice – Leo also hosts Security Now and hears on a weekly basis about the various ways that computers can be attacked from Steve Gibson. Leo was worried about “zero day” attacks. Paul appeared to have a very pragmatic view on things, wishing that we didn’t have this problem in the first place.
So here’s my views on the discussion. I understand why Leo doesn’t like my recommendation. I don’t like my recommendation to delay release of updates for 1 month. But I’ve been seeing for the last 2 years how bad updates for Windows Server (and thus Windows client) and System Center have been. We’re seeing failures and release withdrawals almost on a quarterly basis. And these aren’t just niche scenarios like a shortcut to a font file in the wrong place on Windows 7 Home Premium. This is widely used designs, basic installs, and so on. To be honest, I see the approval of new updates from Microsoft as a bigger risk than malware at this point; releasing an untested update (if I was still an admin) to 100 VMs and 1000 desktops is sure to get me fired within 3-6 months when the business goes in the dark a couple of times because of bad updates. On the other hand, I’ve never had a malware breakout on a network I owned in my career – I’ve only seen malware get trapped by well-managed AV.
I wish I could recommend approving MSFT updates for near-instant deployment, as Leo has suggested. But I cannot – I’ve heard of and reported on too many failures. And any business that needs to rely on their IT cannot take risks.
Paul has it right; Microsoft management is pushing releases (patches, rollups, full product milestones) faster than they should be – and testing is taking second place. I know that technical people that I have great respect for in Redmond are embarrassed by what is going on. Unfortunately, it’s going to take something really bad for Satya Nadella to undo the damage that is happening under his watch, that I guess is probably his doing.
Leo (not that you’ll ever read this), I completely understand your point of view. I used to be a person who said “get the updates out within a week”. But because of the events of the last 2 years, I respectfully have to disagree with you.
BTW, you can take the approach I recommended using SCCM ADRs and tweak it so you create ADRs to approve “critical” updates more rapidly. That will give you a middle ground for security updates, but the risk is yours to measure and take. This is a management decision!
There is a happy compromise I have seen in well-run security organizations in some companies I have dealt with in the past. Their security team will look at all the patches as soon as they can. They rank their importance based on their knowledge of their environment and whether or not the particular patch might have a chance of giving them problems. (Remember, that some patches Microsoft ranks as critical are not critical if the shop is not using the feature that is being patched). If a patch is deemed necessary, they give it a quick test in a lab environment and then deploy fairly quickly thereafter if it does not break anything in their test environment which is a microcosm of their whole system. Other patches are relegated to a batch that is run every three months. That patch batch is tested in the same test environment to ensure everything deploys as it should. Only after that is the package approved for corporate deployment.
Granted, this might seem like a lot of work for a small shop, but it is performed under non-stressful conditions – unlike trying to recover from a bad patch. Critical patches are deployed quickly, and low concern patches are still deployed in a timely manner. If necessary, an exception can be made.
Aidan,
Quite frankly I fully agree with this stance. We test all patches internally on behalf of our clients, at least in the SBS field, prior to releasing them to client networks.
Our business model has changed over the last few years. We are now focused a lot more on Hyper-V and Scale-Out File Server Clusters. From RTM related issues, that is bugs out of the box, to updates that can bring down a cluster we’ve been very hesitant with deploying updates until they are fully tested on our lab systems.
It’s an unfortunate situation to be in. But, at least in this case, the risks far outweigh the benefits for SMB/SME clients that do not have a dedicated IT Team and/or Lab and rely on us.