2008
05.29

I installed a new W2008 x64 DC at work in our W2003 native (single domain) forest.  I’m happy to report that:

  1. It was easy.
  2. It went flawlessly.

I’m planning on wiping out the W2003 presence on our DC’s to have a native W2008 domain.  Right now, there’s no support for monitoring it using SCOM 2007 so I’ll have to wait for a wee while for the management pack and agent support.  I want to be able to monitor our AD so I’ll wait before completing this project.

Here’s one way to introduce a W2008 DC to your existing W2003 AD.

The first question is: to upgrade or do a lean install?  MS are strongly recommending clean installs.  In fact, they almost go as far as saying don’t upgrade.  They do clearly say that a machine with only W2003 components can be upgraded fairly dependably but you’ll want to verify that the machine spec and configuration are good.  Watch out for the desired 40GB C drive – you’ll need to buy 72GB drives if using HP like me.  Things like dodgy AV (I mean you Muckafee and Sinmantec), well …. you’ll want to do a clean install there because, in my opinion, Sinmantec trash the TCP stack when they get their hands on it and the W2008 stack is a complete re-write.

Next question: do you need a rollback plan for the required schema updates?  Best practice is "yes".  The best plan here is to power down selected DC’s before the upgrade and leave them off until you’re sure everything is OK.  Keep the holder of the Schema Master FSMO role turned on – we need it.  If so, then just power on those DC’s and continue as normal.

If something does go wrong with the schema updates then you power off the powered on DC’s and only then would you power on the standby DC’s.  Seize the FSMO roles to one of the now powered on standby DC’s.  Do a metadata cleanup to wipe away all traces of the powered off DC’s.  The powered off DC’s would be disconnected from the network (to prevent AD replication), rebuilt, reattached to the network and DCPROMO’ed.

We’re assuming everything is good.  I’ve not heard of anyone having a schema corruption via a MS update but I’d always recommend being safe.

Now you can follow the process that MS describes.  It’s pretty simple:

  • Copy the "sourcesadprep" folder from the W2008 media to a W2003 DC where you will run the schema updates.  The best DC for this is the holder of the Schema Master FSMO role.  The tools you’ll use are in this folder.
  • Run adprep /forestprep to prepare the forest..
  • Run adprep /domainprep /gpprep to prepare the domain.
  • MS says to only run adprep /rodcprep if you want to run Read-Only DC’s.  As discussed on the Minasi forum, the W2008 version of Dcdiag.exe returns an error when it runs the NCSecDesc test if you don’t do this step.  I did it anyway just to get clean results from DCDIAG.
  • Now you should build your W2008 DC’s operating system and configure it as required.  Install the AD services role.  This will probably install DNS as well.
  • You’re all ready to do your DCPROMO.  It’s pretty much the same as before apart from an annoying DNS warning.  At the end, I’d recommend saving the unattended answer file settings.  You can use this when you plan to DCPROMO your next W2008 DC.

Now, keep an eye on your network, e.g. DFS, FRS, Directory Services, System and Application logs.  I finished off by moving the FSMO roles to my new W2008 DC.

2008
05.27

Microsoft released Service Pack 1 for System Center Comfiguration Manager 2007.  This adds support for Windows Server 2008 and Windows Vista.  The details of the release are:

Configuration Manager 2007 SP1 now offers full support for management with Windows Vista SP1 and Windows Server 2008, integrates customer feedback, feature Integration with Intel vPro Technology and enhances Asset Intelligent features.

  • Full Windows Vista SP1 and Windows Server 2008 Support: Deploy and manage Windows Vista SP1 and Windows Server 2008—with full support for the latest Windows platforms, from planning through inventory, to deployment, and into operational scenarios such as software distribution, software update management, desired configuration management, and more.
  • AMT Integration: Configuration Manager 2007 SP1 integration with Intel Active Management Technology (AMT) enables hardware-based power control (on/off/restart) and delivers many new remote diagnostic and troubleshooting capabilities. Configuration Manager can now perform scheduled or on-demand power control operations on Intel vPro enabled systems in the enterprise, enabling higher levels of software update compliance as well as increasing application installation and operating system deployment success rates. The new out of band management console provides direct hardware interaction using Windows Remote Management (the Microsoft implementation of WS-MAN). This enables remote boot control, allows forced PXE boot for operating system deployments, remote network boot for customized remote tasks and diagnostics, and direct inspection of hardware inventory and power state—even if the system is powered off.
  • Asset Intelligence: Building on the original release within Configuration Manager, this enhancement to the inventory capabilities of Configuration Manager 2007 provides improvements for stronger inventory of hardware, software, and software licenses in use throughout the enterprise. The enhancements made enable administrators to more easily, and more accurately, inventory and manage hardware and software assets as well as view and manage purchased software license information. By providing this essential information, Asset Intelligence makes it easier for administrators and asset managers to more effectively plan for upgrades, migrations, and software license compliance reporting.
    Asset Intelligence in Configuration Manager 2007 SP1 adds the following additional functionality over that provided by the Asset Intelligence feature in Configuration Manager 2007:
    • The Asset Intelligence feature node has been added to the Configuration Manager console to allow easier Asset Intelligence–related administration tasks and rich reporting capabilities.
    • The Asset Intelligence Configuration Manager Console home page has been added to provide at-a-glance feature state status and information.
    • The Asset Intelligence catalog has been expanded to contain categorization and identification information of a large catalog of software titles—both Microsoft and 3rd party—as well as the hardware requirement information for many software titles found in today’s IT environments.
    • The ability to customize the Asset Intelligence catalog with additional software categorization information and hardware requirements information has been added.
    • New reports have been added that enable administrators to generate a total of 70 reports, based on inventoried information, that present data about hardware, software, and license usage.
      • General reports are linked to more specific reports and allow IT administrators to query general information or drill down to more detailed levels if required.
    • Hardware inventory enhancements have been added to gather information such as processor age, speed, and USB devices in use or when hardware has changed since the last inventory or during a specified period of time.
    • Installed software inventory enhancements have been added that gather information about installed software in use in the enterprise.
      • These enhancements allow IT organizations to identify and better categorize their software assets.
      • Robust reports provide information about types of software in use to help identify redundant software and optimize software support and purchasing.
    • Software license management capabilities have been added that allow purchased software license data (both Microsoft and non-Microsoft) to be imported into the Asset Intelligence catalog to enable better license management and reporting.
    • Improvements have been made to provide data about utilized Client Access Licenses (Windows Server, and Exchange Server) and computers acting as Key Management Servers for Windows Vista activation.
      • The report output format is congruent with Microsoft License Statements facilitating system-wide license tracking and compliance.
2008
05.26

As you’ve probably guessed based on my recent posts, I’m doing a SCOM 2007 deployment.  The servers are a mix of IBM (managed by IBM Director) and HP (managed by HP Insight Manager Agents supplied by the PSP V8.0).

On deploying agents to the HP servers I started getting inundated by faults:

  • The Foundation Agent kept failing.  Some research found I wasn’t alone.  Someone on the HP support forum claimed that HP said they fixed this in V8.1 of the PSP or the agents.  I downloaded the agents and tried it.  Things quieted down … a little.
  • Performance monitoring errors.  Lots of them.  Go into Control Panel – HP Insight Manager (or something like that) and remove the Performance agent in there.
  • Management Processor critical errors: HP enabled the ILO and ILO monitoring agent by default.  This is a pain if you don’t actually use ILO and it alerts if a cable isn’t connected to the ILO port.  You can do the Control Panel fix to remove the remote monitoring agent.  You still get an error in the HP Insight Manager web page but the agents don’t report it to SCOM anymore.
2008
05.25

I’m going to do this all using HP Proliants …

Here’s the scenario.  There’s going to be a number of web servers running Windows Server 2003.  They’ll work cooperatively and share files somehow.  They must be load balanced using Windows NLB.  This means using the Unicast method with 2 NIC’s – Unicast allows the servers to talk to each other within the cluster.  HP Proliant servers come a pair of built-in NIC’s so you’d think you’re sorted.  Nope!  You must allow for NIC failure so that means putting in 4 NIC’s and creating two NIC teams, each consisting of a pair of physical NIC’s.

A NIC team is created using at least 2 NIC’s in the HP Network Configuration Utility (NCU).  The newly created virtual NIC has a virtual MAC address or Locally Administered Address (LAA).

Here’s the problem.  When you associate a NIC with a NLB cluster, you are applying a virtual MAC to it.  This MAC is applied identically to all of the NLB NIC’s on every server in the cluster.  Now think … your NLB NIC is actually a virtual NIC made from two physical NIC’s and already has a virtual MAC or LAA.  So which LAA should be applied?  The correct answer is the LAA of the NLB cluster.  This is because the IP address of the NLB cluster is associated with the LAA that should be assigned to the NLB NIC (the NIC team).  Without it having the right LAA, the Ethernet cannot direct traffic to it.

Normally you’d go into the properties of the NIC and configure the driver to set the LAA.  You can’t do this with a HP NIC team.  Instead, once you’ve associated a server’s NIC team with the NLB cluster, just open the HP NCU.  You’re warned that it knows there should be a different LAA for the team in question.  That’s cool.  Just click on OK to save the new configuration and you’re sorted.  Do not click on cancel to exit the NCU because it won’t save the NLB LAA for you.

Just repeat this process on each of the nodes in the NLB cluster and you’re sorted.

EDIT:

In practice, I found that the HP NCU in the HP PSP V8.0 is buggy.  I tested this thing endlessly yesterday and it was fine.  Then all of a sudden, without change, it broke overnight.  Node1 could not see the network (or Node2) but the network could see it.  Removing Node1 from the cluster repaired the network.  Adding it back in broke things again.  Doing the LAA dance in NCU fixed it for about 1 second (showing on a continuous ping).  The logic of it didn’t make sense … LAA issues would affect inbound connectivity to the NLB cluster IP but not outbound connectivity.  In the end I disabled teaming of the NLB NIC’s on both of the nodes.

2008
05.25

SCOM 2007 OOMADS.MSI

I’m doing some manual installation of agents in un-trusted domains and this includes domain controllers.  I just found that once I did this, I’d get inundated with errors from agents on domain controllers (first time I’ve installed on un-trusted DC’s):

  • AD Replication Monitoring : encountered a runtime error. Failed to create the ‘McActiveDir.ActiveDirectory’ object. The error returned was: ‘ActiveX component can’t create object’ (0x1AD)
  • AD Lost And Found Object Count : The script ‘AD Lost And Found Object Count’ failed to create object ‘McActiveDir.ActiveDirectory’. This is an unexpected error. The error returned was ‘ActiveX component can’t create object’ (0x1AD)
  • AD Database and Log : The script ‘AD Database and Log’ failed to create object ‘McActiveDir.ActiveDirectory’. The error returned was: ‘ActiveX component can’t create object’ (0x1AD)

Very annoying, you’ll agree.  I was beginning to wonder if I needed to run the agent with some sort of elevated rights.  In this scenario, you should first run OOMADS.MSI from the support tools folder on the SCOM 2007 installation media.  Only then should you install the agent.  If you made the mistake I did, you can install the tool and then restart the OpsMgr Health Service.  That’ll sort you out.

None of this is required for an agent deployed to a DC from the OpsMgr Console it’s done automatically for you.  And it’s not required at all for non-DC’s.

OOMADS.MSI creates the objects that are required to monitor AD for you.  This will save you a lot of mucking around with ADSI editing which I never recommend to anyone not familiar with AD.

EDIT:

I’ve recently found another issue where there is an event in the Operations Manager log saying that the helper object cannot be found.  This happens soon after the agent starts.  The operations manager agent is search for OOMADS.MSI in %ProgramFiles%System Center Operations Manager 2007HelperObjects and it’s not there.  The solution is to copy the file into there.  I noticed that you need to do this again if you upgrade the agent to SP1.  Restart the agent and the error doesn’t reoccur.

2008
05.24

I’ve just started using MS Live Skydrive.  It’s integrated with the other Live services so you use your passport to sign in.  Irish users just got access to it.  With it you get 5GB of free storage.  You can store documents privately, you can share just with friends or you can share with the Internet.  And you can create nested folders.

2008
05.24

I was recently having some trouble with a software package installation and had to get MS PSS involved.  The engineer asked me to run the setup.msi as:

setup.msi /Lvoicewormup C:log.txt

What the heck?  That first parameter instructs the MSI to create a log of the installation.  The second tells it where to create the log file and what to call it.  BTW, that is "worm" and not "warm".  Yeah, I know.

In my years of software deployment, I’ve never seen this before.  It created a log file of the installation that helped us diagnose the issue.

It struck me that this would make for a great package program in SMS 2003 or SCCM 2007.  You could copy one of the unattended programs and modify it to use this logging parameter.  This would help you diagnosing any issues you have during testing or trouble shooting of package deployment.

2008
05.23

WSUS Failure

Part of the fun of inheriting a network is discovering what your predecessors have done.  It’s even worse when some "expert" consultants (IT Terrorists) have had their way.

I installed a new WSUS server today.  All was well for hours.  Managed servers were discovered and downloading patches.  But suddenly, those servers stopped updating their status.  SCOM 2007 was alerting.  Uh-oh!  I was at home so I fired up the VPN and had a look.

The following events were appearing in Event Viewer on the WSUS server:

13042 Self-update is not working
12002 The Reporting Web Service is not working.
12012 The API Remoting Web Service is not working.
12032 The Server Synchronization Web Service is not working.
12022 The Client Web Service is not working.
12042 The SimpleAuth Web Service is not working.
12052 The DSS Authentication Web Service is not working.

I saw loads of blog and forum entries.  It just came back to one thing … IIS.  Opening the web sites and virtual directories gave me the dreaded "You’re not authorised to view this page" warning.  I’ve worked at a web hosting company so I’ve seen how a corrupted metabase could lead to IUSR hell.  But this was different.  IUSR was OK.

Then it struck me.  Some consultants had played "security expert" in this AD.  I’d already found tonnes of issues in this AD deployment from not understanding DNS to installing Windows 2003 in 10GB partitions (amateurs!).  I checked the policies in GPMC and sure enough, IUSR was not being granted "Allow log on locally".  This was overwriting the local security policy of the WSUS server.  My WSUS box was fine until policy applied and IUSR lost it’s right to log on locally on the WSUS server.

Some AD re-engineering and everything was sorted and my WSUS box was back to normal.

2008
05.23

I’ve been setting up our first production Windows Server 2008 (x64 naturally!) domain controller and I had some issues with the activation.  The first problem I had was the the OS wanted activation almost immediately.  I had no GUI to do anything other than activate or log out.  Bummer!  This caused me two issues: one was that I wasn’t sure of my network settings so I needed access to some controls (GUI or CMD would do) to change those and then .. well, you’ll see in a minute.

There’s a workaround :-)  Click the button to purchase a new license key online.  That opens IE.  Cancel it from going anywhere.  Type "C:" in the address bar.  That opens a new window for Windows Explorer.  From here we can click on Control Panel to make changes or fire up command prompt from the C:WindowsSystem32 folder.  Sneaky!

OK, now I was sure I had a valid network configuration.  I could even navigate the net using IE (not recommended on a server) but it’s a test of Internet connectivity.

Now when I tried to activate I got this:

ERROR: A problem occurred when Windows tried to activate. Error Code 0×80072328.

The problem here was that fat-fingers here had somehow screwed up the product key.  Use the above workaround to get into Control Panel.  Now open up System.  At the bottom you’ll see a link to change your product key.  Enter the correct and valid product key for this server.  You can now activate if everything is correct.

This should work for Vista as well.

2008
05.10

I just read on ENN that the Bank of Ireland has been rather slow at reporting the theft of 4 laptops during the course of last year that contained personal information of 30,000 customers across numerous branches.

HELLO!

Is there anyone there?

Hasn’t the subject of full disk encryption not been covered enough?  How does the nations largest bank not have enough cop-on to do the most basic of physical security operations.  This is the sort of company that hires expensive security consultants, outsources to huge international IT firms and doesn’t appear to value IT.  It’s time for the Irish Data Protection Commissioners to smack these organisations about until they rectify their ways.  And it’d be good to see those people whose personal data have been compromised to be compensated for this banks utter failure to grasp basic security.

EDIT:

It was in the newspaper that the same bank, Bank of Ireland, just disclosed that another laptop was stolen in 2001 that it had never previously reported.  Management knew it was lost but thought nothing more of it.  Personal data of customers was on the laptop – unencrypted.  As a BOI customer and an IT pro who’s been preaching about laptop security for years, I am 100% disgusted.

2008
05.06

Have you heard of MOF?  Have you heard of ITIL?  ITIL is a system of operations, procedures, service delivery, etc that was created by the UK government.  It’s designed to enable an IT service provider to provide a quality service.  The key word is service.  Everything is a service, e.g. your web application, your Risk calculations system, etc.  Services are made up of components, e.g. a network, servers, Windows, IIS, SQL, etc.  Improving your service and guaranteeing quality is based on knowing what your configuration is, managing changes to that configuration and wrapping everything up in procedures.

Microsoft tailored ITIL to a Microsoft network and included a bunch of best practice and guidance.  They called it Microsoft Operations Framework.  It’s now on it’s 4th major release, MOF 4.0.

Check it out.  Lots of company’s are looking at ITIL and MOF.  Getting yourself to be an expert could make you a lot of money in the future!

2008
05.06

TechNet Magazine has posted an article written by Mark Russinovich (MS Windows internals genius) on the changes introduced in the Windows Server 2008 Kernel.

2008
05.01

I’ve been distracted a little lately by work and by Windows Server 2008.  There’s some W2008 stuff I’m planning on doing or have already done:

  • Setting up the the typical W2008 (2 domain controller) domain (whitepaper): content done, just need to edit.  Will share it here ASAP.  Probably be before the end of the month – my weekends and weeknights are already chock-a-block.
  • Windows 2008 and Branch Office Infrastructure (75 minute presentation): Done.  Any user groups interested should drop me a mail and we can discuss travel and accommodation expenses.
  • Windows 2008 GPO and Preferences: I’m really interested in this and want to do some sort of paper and put together a presentation/demo for the Windows Server 2008 User Group.
  • Windows 2008 Domain Controller backup and recover: Again, I’m really interested in this and want to do some sort of paper and put together a presentation/demo for the Windows Server 2008 User Group.

Realistically, the last two will coincide.  The presentations for the user group would be sometime after the summer and the whitepapers would be made available days afterwards.  Watch this space.

Get Adobe Flash player