Recovering from the Defender ASR Bug MO497128 #ASRmageddon

For some Friday the 13th is just another day, but for others…

TL;DR: Just want the script?

What Happened?

On Friday, January 13th, 2023, Microsoft updated the Windows Security and Microsoft Defender for Endpoint service that resulted in a series of false positive detections for the Attack Surface Reduction (ASR) rule “Block Win32 API calls from Office macro”. This updated caused many Windows shortcut (.lnk) files to be match a detection pattern which caused them to be deleted. Not tossed in the Recycle Bin but permanently hard deleted. It was a pretty incredible thing to have a mostly blank Start Menu and find other shortcuts missing outside of that location including but not limited to:

  • C:\Users\%username%\\AppData\Roaming\Microsoft\Windows\Recent
  • C:\Users\%username%\\AppData\Roaming\Microsoft\Office\Recent
  • C:\Users\%username%\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\TaskBar
  • C:\Users\%username%\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\ImplicitAppShortcuts

You can read all about it #ASRmageddon here Service health – Microsoft 365 admin center (https://admin.microsoft.com/Adminportal/Home?#/servicehealth/:/alerts/MO497128) and follow the saga on Twitter

Microsoft’s Fix

Microsoft posted a script to recreate shortcuts on GitHub here:

MDE-PowerBI-Templates/AddShortcutsV1.ps1 at master · microsoft/MDE-PowerBI-Templates · GitHub (https://github.com/microsoft/MDE-PowerBI-Templates/blob/master/ASR_scripts/AddShortcutsV1.ps1)

But it only recreates select shortcuts and it’s not 100% complete. In fact, I think every solution I’ve seen online follows the same basic process:

While I like the above strategies in general – lets be fair, their net results are WAY better than nothing! – none of them were complete, and in some cases shortcuts we re-created in the root of the Programs directory (C:\ProgramData\Microsoft\Windows\Start Menu\Programs) versus where they belong (e.g.: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\7-Zip). Not a big deal but I had a handful of folders under \Start Menu\Programs that were empty even after running these solutions which meant me and my team to fill in the gaps.

The Obvious Fix: Restore from Backup

If you have some sort of backup solution in place, then you could probably whip something up from that with ease. In my new environment, we don’t have that since virtually everything is in the cloud with very little stored locally. That said, we needed a solution in-lieu of something like Druva.

Volume Shadow Copy Service to the Rescue!

In Microsoft’s Tech Community site, they posted a page titled “Recovering from Attack Surface Reduction rule shortcut deletions” which, to be candid, was not very helpful, even with a link to their script on GitHub. However Matt Rouse posted a gem of comment about leveraging the Volume Shadow Copy Service to restore the deleted files! I was immediately reminded of the work I used to do with Citrix-based VDI’s running on VMware where manipulating the VSS was commonplace! I quickly put this to the test, on my machine then ran it past the rest of my team who confirmed it was very promising. Provided there was a VSS copy, it was hands down the absolute best solution we had seen since it allowed us to fully restore the Start Menu!

We packaged up our VSS-based restoration script into an application so that we could easily measure execution, success etc. and deployed it to our 6k+ devices. After letting it bake for a while I checked a handful random machines, including those of users who had previously called to complain of empty Start Menu’s, and I was pleased to see it was working well!

Again, ultra hat tip to Matt Rouse.

Confirm you’ve got a vss copy to potentially extract data from

Please note I am NOT an expert on the subject of the Volume Shadow Copy Service. This is ‘just-in-time’ learning & education to solve a crazy issue that took everyone globally by surprise overnight..

This should work for virtually everyone since VSS should be enabled as Microsoft does not recommend disabling it. So assuming its functional, fire up an elevated command prompt to issue the following command to see available shadow copies:

vssadmin list shadows

If you have multiple drives add /for=%SystemDrive% or /for=C: or whatever drive stores your ProgramData directory:

I prefer this method since it makes the Creation Time (in PowerShell land it’s the InstallDate) easy to read.

You can do the same from an elevated PowerShell session, by running either of these commands:

Get-CIMInstance -Class Win32_ShadowCopy
Get-WMIObject -Class Win32_ShadowCopy

Look for a DeviceObject that has an InstallDate from before the security intelligence update build (1.381.2140.0) went out, which I think is 9am UTC on January 13, 2023.

Check the InstallDate before choosing which DeviceObject to work with. In this example I would not want to work with DeviceObject \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2

In my case I’ll be using DeviceObject : \?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1 because the restore InstallDate is just before the window when the update was released.

Now we need to “mount” the Shadow Copy, and to do that we’ll be making a symbolic link (aka symlink) to it accessible via what looks like a directory in the root of C:\. So from that same elevated command prompt run this command

mklink /d c:\ShadowCopy1 \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\

Note: Make sure you replace the path to the DeviceObject with yours of course! it could be Copy2, or Copy3 or CopyN

Also, it is possible to create a symlink in PowerShell but I was struggling to sort it out, and with over 6k machines/users to sort out I went down the path of least resistance. If anyone can set me straight on that, I would very much appreciate it!

Now when I browse C:\ShadowCopy1, I’ll actually be looking at the contents of my Shadow Copy in \?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\.

From here it was as simple as copying from the “mounted” VSS copy to %ProgramData%\Microsoft\Windows\Start Menu\Programs. My gut was to initially do a mirrored robocopy:

robocopy /E /MIR "C:\ShadowCopy1\ProgramData\Microsoft\Windows\Start Menu\Programs" "C:\ProgramData\Microsoft\Windows\Start Menu\Programs"

To fix any issues that may have arisen with manual fixes. But my colleagues convinced me of the possibility that someone may have installed something between the last VSS copy and the date the recovery script executed so I erred on the side of caution and decided not to do a mirror copy.

The Script

Our script ended up being longer than what’s below to account for situations where there wasn’t a usable Shadow Copy available, but this should be a good starting point to send you down the right path.

The normal warnings & Standard Disclaimers apply:

  • Use at your own risk
  • I’m not liable if this breaks your machine or environment
  • Your Mileage May Vary (YMMV)
  • Etc.

Everyone signed? Yes? Good. On we go!

try
    {
        [System.Management.Automation.ActionPreference]$OriginalEAP = $ErrorActionPreference
        $ErrorActionPreference = [System.Management.Automation.ActionPreference]::Stop

        # The last 'good' date in UTC time 
        $DateTime = [datetime]'01/13/2023 09:00:00' 
 
        # Get the machines' current time zone for conversion from UTC to local time 
        $ToTimeZone = [System.TimeZoneInfo]::FindSystemTimeZoneById((Get-Timezone).Id) 
 
        # Convert the DateTime above from UTC to local time 
        $ConvertedTime = ([System.TimeZoneInfo]::ConvertTimeFromUtc($DateTime ,$ToTimeZone)) 
        Write-Host "$(Get-Date -Format s) ConvertedTime [$ConvertedTime]" 
 
        # Query to see what Shadow Copies are available 
        $Win32_ShadowCopy = Get-WMIObject -Class Win32_ShadowCopy 

        # Loop through each Copy to see if one has an installdate prior to $DateTime above
        :ShadowCopyCheck foreach($ShadowCopy in $Win32_ShadowCopy) 
            {                  
                if($ShadowCopy.InstallDate) 
                    { 
                        # This worked once or twice for me but then failed
                        #[datetime]$ShadowCopy_Date = $Win32_ShadowCopy.ConvertToDateTime($ShadowCopy.InstallDate) 
                        # So I'm now having to do this to ensure it's successful
                        $ShadowCopy_Date = $Win32_ShadowCopy.ConvertToDateTime($ShadowCopy.InstallDate) 
                        if($ShadowCopy_Date.Count -gt 1) { [datetime]$ShadowCopy_Date = $ShadowCopy_Date[0] } 
                        else { [datetime]$ShadowCopy_Date = $ShadowCopy_Date } 
 
                        if($ShadowCopy_Date -lt $ConvertedTime) 
                            { 
                                [bool]$ShadowCopy_Usable = $true 
                                Write-Host "$(Get-Date -Format s) Valid [$ShadowCopy_Usable] ShadowCopy from before [$ConvertedTime]: [$ShadowCopy_Date]" 
                                break :ShadowCopyCheck 
                            } 
                        else 
                            { 
                                [bool]$ShadowCopy_Usable = $false 
                                Write-Host "$(Get-Date -Format s) Invalid [$ShadowCopy_Usable] ShadowCopy is newer/after [$ConvertedTime]: [$ShadowCopy_Date]" 
                            } 
                    } 
                else 
                    { 
                        [bool]$ShadowCopy_Usable = $false 
                        Write-Host "$(Get-Date -Format s) No Shadow Copy InstallDate [$($ShadowCopy.InstallDate)]" 
                    } 
            } 
        Write-Host "$(Get-Date -Format s) ShadowCopy_Usable [$ShadowCopy_Usable]" 
 
 
        if(($ShadowCopy_Usable -eq $true) -and ($ShadowCopy)) 
            { 
                $ShadowCopy_DirName = $ShadowCopy.DeviceObject.Substring($ShadowCopy.DeviceObject.IndexOf('ShadowCopy')) 
                Write-Host "$(Get-Date -Format s) ShadowCopy_DirName [$ShadowCopy_DirName]" 
 
                $ShadowCopy_LocalPath = Join-Path $env:SystemDrive $ShadowCopy_DirName 
                Write-Host "$(Get-Date -Format s) ShadowCopy_LocalPath [$ShadowCopy_LocalPath]" 
 
                # Check to see if the symlink has already been created 
                if(!(Test-Path -Path "$ShadowCopy_LocalPath" -PathType Container)) 
                    { 
                        Write-Host "$(Get-Date -Format s) Creating symlink [$ShadowCopy_LocalPath]  $($ShadowCopy.DeviceObject)\" 
 
                        # I couldnt figure out how to create a symlink with PowerShell so relying on good-ol cmd
                        & cmd.exe /c mklink /d "$ShadowCopy_LocalPath" "$($ShadowCopy.DeviceObject)\" 
 
                        # Give it a sec to complete
                        Start-Sleep -Seconds 3 
                    } 
                else 
                    { 
                        Write-Host "$(Get-Date -Format s) WARN Path already exists [$ShadowCopy_LocalPath] suggesting a symlink has already been created" 
                    } 

                # Verify we have something to co back to
                if(Test-Path -Path "$ShadowCopy_LocalPath\ProgramData\Microsoft\Windows\Start Menu\Programs" -PathType Container) 
                    { 
                        Write-Host "$(Get-Date -Format s) Begin restore from Shadow Copy via Robocopy" 
                        & robocopy "$ShadowCopy_LocalPath\ProgramData\Microsoft\Windows\Start Menu\Programs" "$env:ProgramData\Microsoft\Windows\Start Menu\Programs" /E /R:10 /W:10 /X /V /TS /FP 
                    } 
                else 
                    { 
                        Write-Host "$(Get-Date -Format s) ERROR CAN'T FIND [$ShadowCopy_LocalPath\ProgramData\Microsoft\Windows\Start Menu\Programs]" 
                    } 
 
                if(Test-Path -Path $ShadowCopy_LocalPath -PathType Container)
                    {
                        Write-Host "$(Get-Date -Format s) Removing Shadow Copy symlnk" 
                        & cmd.exe /c RD "$ShadowCopy_LocalPath"
 
                        # Give it a sec to complete
                        Start-Sleep -Seconds 3
                    }
            } 
        else 
            { 
                Write-Host "$(Get-Date -Format s) No usable Shadow Copies available."
            }

        Write-Host "$(Get-Date -Format s) Main script completed. ^^^ See above for details ^^^"
    }
catch
    {
        throw $_
    }
finally
    {
        $ErrorActionPreference = $OriginalEAP
        Remove-Variable OriginalEAP,DateTime,ToTimeZone,ConvertedTime,Win32_ShadowCopy,shadow* -ErrorAction SilentlyContinue
    }

I’ll say that was one heck of a Friday [the 13th] and hopefully the last crazy IT-related thing for this year.

Thanks for hanging with me on this. I hope this helps and if you have any questions, let me know in the comments.

Good Providence to you!

4 comments

    1. Hey there and thanks for taking the time to comment! Thank YOU for your script too as it served as a good base for the fallback solution. Many thanks 🙂

      Liked by 1 person

      Reply

  2. Thanks for the explanation and the script!

    I’d request one fix prior to manually performing the robocopy steps: “my gut was to initially do a mirrored robocopy” you’ve added a space in “Program Data” path – it should be “ProgramData”

    robocopy /E /MIR “C:\ShadowCopy1\ProgramData\Microsoft\Windows\Start Menu\Programs” “C:\ProgramXData\Microsoft\Windows\Start Menu\Programs”

    Like

    Reply

Leave a comment