Azure Local - Troubleshooting: MOC vs. Failover Cluster VM placement

Intro
Problem
Root cause
The script
Solution
Recommendation

Intro

A customer added two more nodes to an existing Azure Local cluster, going from 2 to 4 nodes. Shortly after, some virtual machines could no longer be started or stopped from the Azure portal - the operations either failed outright or returned NotFound. The cluster itself looked healthy in Failover Cluster Manager, and the VMs were clearly still running somewhere, so this was not a simple “the VM is broken” case.

Full credit for figuring this out goes to my colleague Mads Asdal, who built the PowerShell script that got us to the bottom of it: Compare-MOCVMplacement.ps1. It compares what the Failover Cluster believes about VM ownership (which node a VM group is currently hosted on) against what MOC - the Microsoft On-premises Cloud agent that Azure Local uses to represent VMs to Azure - believes about that same VM. When those two views disagree, Azure operations against the VM start failing, because Azure talks to MOC, not directly to the cluster.

This post shows the script, what its output looked like in our case, and how we got VMs back in sync - without going into the underlying cluster-level fixes that caused the drift in the first place, since those are specific to what triggered it in this environment.

Problem

After scaling the cluster from 2 to 4 nodes:

Some VMs could not be started or stopped from the Azure portal or via az/ARM.
Some operations against VMs failed with NotFound, even though the VM was clearly running when checked directly on the cluster.
Failover Cluster Manager showed the VMs as Online and otherwise healthy.

This is the kind of symptom that is easy to misdiagnose as “the VM’s Arc/MOC registration is broken” or “re-onboard the VM”, when the actual problem is narrower: MOC’s record of which host the VM lives on had gone stale relative to the cluster’s own view, most likely as a side effect of the node-count change and the VM ownership moves that came with it.

Root cause

Azure Local VMs are represented to Azure through MOC (Microsoft On-premises Cloud), which keeps its own record of VM placement - including which cluster node currently hosts each VM. The Failover Cluster is the actual source of truth for VM ownership.

When MOC’s placement record for a VM disagrees with the cluster’s actual owner node, Azure-side operations against that VM (start, stop, and similar) go through MOC first and can fail or return NotFound, even though the VM is running fine from the cluster’s perspective.

There were three distinct ways a VM could show up as “not in sync” in this environment, and the script deliberately keeps them separate rather than reporting one generic “mismatch” count:

Not present in MOC at all - these return NotFound from ARM/the portal.
Stale placement record - MOC knows about the VM, but believes the wrong owner node.
Power state disagreement - the cluster and MOC disagree on whether the VM is running or off.

HINT

The actual cluster-level fixes that resolved why the drift happened are specific to this environment and are not covered here. This post is about identifying the drift and the general recovery step that got VMs back in sync, not the root-cause remediation itself.

The script

Compare-MOCVMplacement.ps1 is read-only - it makes no changes to MOC, the cluster, or any VM. It only reports. It must be run in a local console or RDP session directly on a cluster node, since the MOC cmdlets it depends on do not survive nested WinRM.

<#
.SYNOPSIS
    Compares Azure Local (MOC) VM host placement against the Failover Cluster's view of VM ownership.

.DESCRIPTION
    Collects all MOC-registered VMs for a given location, indexes them by name, and joins that data
    against the cluster's VM groups to flag VMs where the cluster's owner node does not match the
    host MOC believes the VM is running on.

.PARAMETER MocLocation
    The MOC location/region to query for VM groups.

.PARAMETER ClusterName
    The name of the failover cluster to compare against.

.PARAMETER ExcludeVmNamePattern
    Regex applied to VM names to drop MOC-internal infrastructure VMs (e.g. AKS Arc / CAPI
    control-plane and machine-deployment nodes named like
    '7aa9df718d1c5b5354c576d7238450ff1d8671d9bc296-control-plane-0-44134e1a') from both the
    cluster join and the orphan report. Pass an empty string to disable filtering.

.EXAMPLE
    ./Compare-MOCVMplacement.ps1
    ./Compare-MOCVMplacement.ps1 -MocLocation 'MocLocation' -ClusterName 'CLUSTERNAME' -Verbose

.NOTES
    Read-only. Makes no changes to MOC, the cluster, or any VM.

    MUST be run in a local console / RDP session on a cluster node. The MOC cmdlets do not
    survive nested WinRM.

    Gotcha this script exists to work around:
    Get-MocVirtualMachine emits ONE object that is itself an array, rather than streaming one
    object per VM. Collecting it with a bare foreach therefore yields one element per GROUP
    (typically 2), not one per VM (~79). $vm then binds to a whole array, $vm.tags.'VM-Name'
    member-enumerates into an array, and the hashtable ends up keyed by array objects - so every
    string lookup misses and the report shows InMoc=False for every VM. The
    '| ForEach-Object { $_ }' in Get-MocVmIndex forces the unroll. Do not remove it.
#>
[CmdletBinding()]
param(
    [string]$MocLocation = "MocLocation",
    [string]$ClusterName = (get-cluster).name,
    [string]$ExcludeVmNamePattern = '^[0-9a-f]{20,}-(control-plane|md)-\d+(-[0-9a-f]{6,})?$'
)

function Get-MocVmIndex {
    <#
    .SYNOPSIS
        Builds a lookup table of MOC VMs keyed by their undecorated (cluster-facing) name.
    #>
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)]
        [string]$Location
    )

    # The '| ForEach-Object { $_ }' is load-bearing - see the note in the script header.
    $mocVms = foreach ($grp in (Get-MocGroup -location $Location).name) {
        Get-MocVirtualMachine -group $grp | ForEach-Object { $_ }
    }
    Write-Verbose "MOC VMs collected: $($mocVms.Count)"

    if ($mocVms.Count -lt 10) {
        throw ("Only $($mocVms.Count) MOC VM(s) collected - this almost certainly means the " +
               "wrapped-array unroll failed, not that the cluster has that few VMs. Check that " +
               "'Get-MocVirtualMachine -group <grp> | ForEach-Object { `$_ }' is intact.")
    }

    $index = @{}
    foreach ($vm in $mocVms) {
        $key = if ($vm.tags.'VM-Name') { $vm.tags.'VM-Name' } else { $vm.name }

        if ($key -isnot [string]) {
            throw ("Index key is a $($key.GetType().Name), not a string. The MOC result was not " +
                   "unrolled correctly - see the note in the script header.")
        }

        if ($index.ContainsKey($key)) { Write-Warning "Duplicate MOC VM name '$key' - later entry wins." }
        $index[$key] = $vm
    }
    Write-Verbose "Indexed keys: $($index.Count)"

    return $index
}

function Compare-ClusterVmPlacement {
    <#
    .SYNOPSIS
        Joins cluster VM group ownership against a MOC VM index and flags placement mismatches.
    #>
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)]
        [string]$ClusterName,

        [Parameter(Mandatory)]
        [hashtable]$MocIndex,

        [string]$ExcludeVmNamePattern
    )

    Get-ClusterGroup -Cluster $ClusterName |
        Where-Object GroupType -eq 'VirtualMachine' |
        Where-Object { -not $ExcludeVmNamePattern -or $_.Name -notmatch $ExcludeVmNamePattern } |
        ForEach-Object {
            $group = $_
            $mocVm = $MocIndex[$group.Name]

            # host.id is the bare node name in lower case, e.g. 'hciprdofrc03'.
            # -eq is case-insensitive, so comparing straight to OwnerNode.Name is correct.
            $mocKnownHost = $mocVm.virtualmachineproperties.host.id

            [pscustomobject]@{
                VM               = $group.Name
                ClusterNodeOwner = $group.OwnerNode.Name
                MocKnownHost     = $mocKnownHost
                InMoc            = [bool]$mocVm
                Matches          = [bool]($mocKnownHost -and $mocKnownHost -eq $group.OwnerNode.Name)
                ClusterState     = $group.State
                MocPower         = $mocVm.virtualmachineproperties.statuses.PowerState
            }
        }
}

$mocIndex = Get-MocVmIndex -Location $MocLocation
$rows = @(Compare-ClusterVmPlacement -ClusterName $ClusterName -MocIndex $mocIndex -ExcludeVmNamePattern $ExcludeVmNamePattern)

$rows | Sort-Object Matches, VM | Format-Table -AutoSize

# Three distinct divergence classes - do not collapse them into one "mismatch" number.
$notInMoc   = @($rows | Where-Object { -not $_.InMoc })
$wrongHost  = @($rows | Where-Object { $_.InMoc -and -not $_.Matches })
$powerDrift = @($rows | Where-Object {
                    $_.InMoc -and
                    (($_.ClusterState -eq 'Online'  -and $_.MocPower -eq 'Off') -or
                     ($_.ClusterState -eq 'Offline' -and $_.MocPower -eq 'Running'))
                })

''
"Cluster VM groups        : $($rows.Count)"
"MOC VMs indexed          : $($mocIndex.Count)"
"Not present in MOC       : $($notInMoc.Count)    <- these return NotFound from ARM/Portal"
"Stale placement record   : $($wrongHost.Count)    <- MOC believes the wrong owner node"
"Power state disagreement : $($powerDrift.Count)"

if ($notInMoc.Count)   { ''; 'Not in MOC:';         $notInMoc   | Select-Object VM, ClusterNodeOwner, ClusterState | Format-Table -AutoSize }
if ($wrongHost.Count)  { ''; 'Stale placement:';    $wrongHost  | Select-Object VM, ClusterNodeOwner, MocKnownHost | Format-Table -AutoSize }
if ($powerDrift.Count) { ''; 'Power disagreement:'; $powerDrift | Select-Object VM, ClusterState, MocPower | Format-Table -AutoSize }

# Reverse direction: MOC entries with no corresponding cluster group.
$clusterNames = $rows.VM
$orphans = @($mocIndex.GetEnumerator() | Where-Object {
                  $_.Key -notin $clusterNames -and
                  (-not $ExcludeVmNamePattern -or $_.Key -notmatch $ExcludeVmNamePattern)
              })
if ($orphans.Count) {
    ''
    "MOC entries with no cluster group: $($orphans.Count)"
    $orphans | ForEach-Object {
        [pscustomobject]@{
            MocKey  = $_.Key
            MocName = $_.Value.name
            MocHost = $_.Value.virtualmachineproperties.host.id
            Power   = $_.Value.virtualmachineproperties.statuses.PowerState
        }
    } | Format-Table -AutoSize
}

Solution

Running the script against the affected cluster gave us a clear summary to work from:

The full per-VM table let us see exactly which VMs the cluster and MOC disagreed on, and in which direction:

In our case, most of the flagged VMs turned out to be VMs that had never been created from Azure in the first place (Migrated VMs, not Arc-managed workloads) - these correctly show up as NotFound in MOC and are expected, not a symptom of the actual issue:

The VMs that mattered were the ones Azure had created, where MOC’s placement record had genuinely gone stale relative to the cluster:

After applying a series of cluster-level fixes specific to this environment (not covered in this post), the recovery step that actually got the affected VMs back in sync with MOC was simple: stop and start the VM from Failover Cluster Manager (not from Azure). Doing this for each affected VM forced a fresh ownership/placement event that MOC picked up correctly, after which Azure-side start/stop operations against those VMs worked again.

Recommendation

If you scale an Azure Local cluster by adding nodes, I recommend running a placement comparison like this afterwards, before assuming everything is fine just because Failover Cluster Manager looks healthy. MOC and the cluster are two separate sources of truth, and Azure only ever talks to one of them.

HINT

Keep the three divergence categories (not in MOC / stale placement / power state disagreement) separate when triaging. They point to different problems, and collapsing them into a single “mismatch count” hides which VMs actually need action.

Final remark: when VMs suddenly cannot be started or stopped from Azure after a cluster topology change, check for a MOC-vs-cluster placement mismatch before assuming the VM itself, its Arc registration, or the extension stack is broken.

Have feedback on this post?

Send me a message and I'll get back to you.