1607

Task Sequence Fails to Start 0x80041032

Recommended reading: https://blogs.msdn.microsoft.com/steverac/2014/08/25/policy-flow-the-details/

After preparing a 1730 upgrade Task Sequence for our 1607 machines, I kicked it off on a box via Software Center and walked away once the status changed to ‘Installing thinking I’d come back to an upgraded machine an hour later.  To my surprise, I was still logged on and the TS was still ‘running’.  A few hours later the button was back to ‘Install’ with a status of ‘Available’.  Thinking I goofed I tried again and the same thing happened.

I assumed it was unique to this one machine so I tried it on another and it behaved the same way.  I then ran the upgrade sequence on 5 other machines and they all exhibited the same behavior.  I knew the Task Sequence was sound because others were using it, so it was definitely something unique to my machines but what?

Since I was doing this on 1607 machines I tried upgrading form 1511 to 1703 and 1511 to 1607 but they too failed the same way confirming it was not Task Sequence specific but again unique to my machines.  After spending a quite a few cycles on this, my original machine started failing differently: I was now seeing a ‘Retry’ button with a status of ‘Failed’.  I checked the smsts.log but it didn’t have a recent date/time stamp so it never got that far.  Hmm…

Check the TSAgent.log

Opening the TSAgent.log file I could see some 80070002 errors about not being able to delete HKLM\Software\Microsoft\SMS\Task Sequence\Active Request Handle but the real cause was a bit further up.

TSAgentLog

The lines of interest:


Getting assignments from local WMI. TSAgent 9/1/2017 12:58:27 PM 1748 (0x06D4)
pIWBEMServices->;;ExecQuery (BString(L"WQL"), BString (L"select * from XXX_Policy_Policy4"), WBEM_FLAG_FORWARD_ONLY, NULL, &pWBEMInstanceEnum), HRESULT=80041032 (e:\nts_sccm_release\sms\framework\osdmessaging\libsmsmessaging.cpp,3205) TSAgent 9/1/2017 1:03:55 PM 1748 (0x06D4)
Query for assigned policies failed. 80041032 TSAgent 9/1/2017 1:03:55 PM 1748 (0x06D4)oPolicyAssignments.RequestAssignmentsLocally(), HRESULT=80041032 (e:\cm1702_rtm\sms\framework\tscore\tspolicy.cpp,990) TSAgent 9/1/2017 1:03:55 PM 1748 (0x06D4)
Failed to get assignments from local WMI (Code 0x80041032) TSAgent 9/1/2017 1:03:55 PM 1748 (0x06D4)

The source of error code 80041032 is Windows Management (WMI) and translates to ‘Call cancelled’ which presumably happened while running the query select * from XXX_Policy_Policy4, where XXX is the site code.

I ran a similar query on my machine to get a feel for the number of items in there:


(gwmi -Class xxx_policy_policy4 -Namespace root\xxx\Policy\machine\RequestedConfig).Count

Which ended up failing with a Quota violation error suggesting I’ve reached the WMI memory quota.

Increase WMI MemoryPerHost & MemoryAllHosts

Fortunately, there’s a super helpful TechNet Blog post about this.  Since all of my test machines were running into this, I decided to make life easier for myself and use PowerShell to accomplish the task on a few of them thinking I’d have to raise the limit once.


$PHQC = gwmi -Class __providerhostquotaconfiguration -Namespace root
$PHQC.MemoryPerHost = 805306368
# Below is optional but mentioned in the article
#$PHQC.MemoryAllHosts = 2147483648
$PHQC.Put()
Restart-Computer

After the machine came up I ran the same query again, and after 2 minutes and 38 seconds it returned over 1800 items.  Great!  I ran it again and after 5 minutes it failed with the same quota violation error.  Boo urns.  I kept raising MemoryPerHost and MemoryAllHosts to insane levels to get the query to run back to back successfully.

The good news is that I made progress suggesting I’m definitely hitting some sort of memory ceiling that has now been raised.

The bad news is why me and not others?  Hmm…

Root Cause Analysis

I checked the deployments available to that machine and wowzers it was super long.  I won’t embarrass myself by posting an image of that but it was very long.  This helped to identify areas of improvement in the way we’re managing collections & deploying things, be it Applications, Packages, Software Updates and so on.

On my patient zero machine I ran the following to clear out the policies stored in WMI:


gwmi -namespace root\xxx\softmgmtagent -query "select * from ccm_tsexecutionrequest" | remove-wmiobject

gwmi -namespace root\xxx -query "select * from sms_maintenancetaskrequests" | remove-wmiobject

restart-service -name ccmexec

I then downloaded the policies and tried to image – it worked!  I decided to let that machine go and focus on some administrative cleanup in SCCM.  After tidying things up a bit, the rest of my 1607 machines started the 1703 upgrade Task Sequence without issue and the 1511 machines ran the 1607 upgrade Task Sequence as well.

As we continue to phase out Windows 7, we’ll hopefully update our methods to help avoid problems like this and perform that routine maintenance a little more frequently.

 

Advertisements

Upgrading Windows 10 1511 to 1607

In 2016 we began the process of moving from Windows 7 to Windows 10 1511 learning a ton along the way.  After 1607 went Current Branch for Business (CBB) we began planning for that upgrade phase, and what lies below is a overview of that process.

Initial Smoke Test

Once 1607 went CBB we very quickly threw an upgrade Task Sequence together to see what the upgrade process looked like and what, if anything, broke.  The upgrade process went smoothly, the vast majority of applications worked but there were a handful of things that needed to be worked out

  • Remote Server Administration Tools (RSAT) was no longer present
  • Citrix Single Sign-On was broken
  • A bunch of Universal Windows Platform (UWP) or Modern Applications we didn’t want were back
  • Default File Associations had reverted
  • The Windows default wall paper had returned
  • User Experience Virtualization (UE-V/UX-V) wasn’t working.
  • Taskbar pinnings were incorrect; Specifically Edge and the Windows Store had returned

Still, everything seemed very doable so we had high hopes for getting it out there quickly.

Approach: Servicing vs Task Sequence

Wanting to get our feet wet with Windows as as Service (WaaS), we explored leveraging Servicing Plans to get everyone on to 1607 but quickly ran into a show stopper: How were we going to fix all of the 1607 upgrade ‘issues’ if we went down this path?

We couldn’t find an appealing solution for this, so we went ahead with the Task Sequence based upgrade approach.  This gave us greater flexibility because not only could we fix all the upgrade issues but also do a little housekeeping, like install new applications, upgrade existing applications, remove retired applications and more.  This was far simpler and more elegant than setting up a bunch of deployments for the various tasks we wanted to accomplish either before the upgrade or after.

Avoiding Resume Updating/Generating Events

One concern with Servicing was ensuring the upgrade wasn’t heavy handed, forcing a machine to either upgrade mid-day because they were beyond the deadline or during the night because they left their machine on.  This was because the upgrade would bounce their machine which could potentially result in lost work, something most people find undesirable.  With Servicing, we couldn’t come up with a sure-fire way to check for and block the upgrade if, say instances of work applications were detected, such as the Office suite, Acrobat and so on.

Sure, we could increase the auto-save frequency – perhaps setting it to 1 minute – and craft a technical solution to programmatically save files in the Office Suite, safe Drafts and try to do some magic to save PDF’s and so on.  But at the end of the day, we couldn’t account for every situation: we would never know if the person wanted to create a new file vs a new version or simply overwrite the existing one.  And most importantly, we didn’t want to have to answer why a bunch of Partners lost work product as a result of the upgrade.

So, we decided to go the Task Sequence route.

Task Sequence Based Upgrade

Now that we knew which way we need to go, it was just a matter of building the fixes to remediate the upgrade issues then setup the Task Sequence.

Upgrade Remediation

  • Remote Server Administration Tools (RSAT) – Prior to performing the OS upgrade, a script is executed to detect RSAT, and if present, a Boolean variable which is referenced after the upgrade is complete to triggers re-installation of RSAT.
    .
  • Citrix Single Sign-On – This is a known issue – see CTX216312 for more details.
    .
  • Universal Windows Platform (UWP) applications – Re-run our in-house script to remove the applications.
    .
  • Default File Associations
    • Option 1: Prior to performing the OS upgrade, export HKCR and HKCU\Software\Classes then import post upgrade.
    • Option 2: Re-apply the defaults via dism from our ‘master’ file.
      .
  • Wallpaper – Re-apply in the Task Sequence by taking advantage of the img0 trick.
    .
  • UE-V/UX-V – The upgrade process broke the individual components of the UE-V Scheduled Tasks requiring a rebuild.  Once fixed on a machine we copied the good/fixed files from C:\Windows\System32\Tasks\Microsoft\UE-V and setup the Task Sequence:
    1. Enable UE-V during the upgrade via PowerShell
    2. Copied the fixed files into C:\Windows\System32\Tasks\Microsoft\UE-V
    3. Updated the command line the Scheduled Task ran
    4. Disabled the ‘Synchronize Settings at Logoff‘ Scheduled Task since that was still buggy, causing clients to hang on log off.
      .
  • Taskbar Pinnings – Prior to performing the OS upgrade, export HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Taskband then import post upgrade.
    .
  • Critical Process Detection – CustomSettings.ini calls a user exit script that loops through a series of key executables (outlook.exe, winword.exe etc.) running tasklist to see if a process is detected and if so sets a Task Sequence variable that’s later evaluated.

Since we were going the Task Sequence route, and it would be generally available in Software Center, it was decided a password prompt might help prevent accidental foot shooting.  So shortly after the Task Sequence launches an HTA driven password prompt is displayed that only IT should be able to successfully navigate.  This added yet another line of defense for anyone who ‘accidentally’ launched the Task Sequence;
Even though one has to click through two prompts to confirm the installation but whatever. 🙂