• The error
• Intro
• The solution

The error

[UpgradeArbAndExtensions :arcappliance upgrade] Failed with Error C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd arcappliance upgrade hci --config-file "C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\MocArb\WorkingDirectory\Appliance\hci-appliance.yaml" --only-show-errors returned a non empty error stream [ERROR: (KubeAdmControlPlaneErrorOnPrem) Upgrade Operation Failed with error: "{\n \"code\": \"PostOperationsError\",\n \"message\": \"waiting for kubeadmcontrolplane to be provisioned: default/85c19b633461f986a9c45610fbc24fcf44f5a3286749b: timed out waiting for the condition\",\n \"category\": \"\"\n }" Code: KubeAdmControlPlaneErrorOnPrem Message: Upgrade Operation Failed with error: "{\n \"code\": \"PostOperationsError\",\n \"message\": \"waiting for kubeadmcontrolplane to be provisioned: default/85c19b633461f986a9c45610fbc24fcf44f5a3286749b: timed out waiting for the condition\",\n \"category\": \"\"\n }"] at [at Invoke-ArcHciAzCommandLine, C:\Program Files\WindowsPowerShell\Modules\ArcHci\1.3.15\archci.psm1: line 3471 at Invoke-ArcHciAzCommand, C:\Program Files\WindowsPowerShell\Modules\ArcHci\1.3.15\archci.psm1: line 3345 at UpgradeArbAndExtensionsInternal, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbHelper.psm1: line 2692 at UpdateArbAndExtensions, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbLifeCycleManager.psm1: line 430 at , C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 165 at Invoke-EceInterfaceInternal, C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 160 at , : line 50]
Command Arguments
---
---- ---------
UpgradeArbAndExtensionsInternal {Parameters=CloudEngine.Configurations.EceInterfaceParameters}
{}
{CloudEngine.Configurations.EceInterfaceParameters, MocArb, UpdateArbAndExtensions, C…
Invoke-EceInterfaceInternal {CloudDeploymentModulePath=C:\NugetStore\Microsoft.AzureStack.Solution.Deploy.CloudDe…
{CloudEngine.Configurations.EceInterfaceParameters, 00000000-0000-0000-0000-000000000… at Trace-Error, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\Common\Tracer.psm1: line 63
at UpgradeArbAndExtensionsInternal, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbHelper.psm1: line 2695
at UpdateArbAndExtensions, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbLifeCycleManager.psm1: line 430
at , C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 165
at Invoke-EceInterfaceInternal, C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 160
at , : line 50
Command Arguments
------- ---------
UpgradeArbAndExtensionsInternal {Parameters=CloudEngine.Configurations.EceInterfaceParameters}
{}
{CloudEngine.Configurations.EceInterfaceParameters, MocArb, UpdateArbAndExtensions, C…
Invoke-EceInterfaceInternal {CloudDeploymentModulePath=C:\NugetStore\Microsoft.AzureStack.Solution.Deploy.CloudDe…
{CloudEngine.Configurations.EceInterfaceParameters, 00000000-0000-0000-0000-000000000… at Trace-Error, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\Common\Tracer.psm1: line 63
at UpgradeArbAndExtensionsInternal, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbHelper.psm1: line 3083
at UpdateArbAndExtensions, C:\NugetStore\Microsoft.AzureStack.MocArb.LifeCycle.1.2510.0.8\content\Scripts\MocArbLifeCycleManager.psm1: line 430
at , C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 165
at Invoke-EceInterfaceInternal, C:\Agents\Microsoft.AzureStack.Solution.ECEWinService.10.2510.0.1134\content\ECEWinService\InvokeInterfaceInternal.psm1: line 160
at , : line 50
at CloudEngine.Actions.PowerShellHost.WaitAndReceiveJob(Job job, CancellationToken token, UInt32 timeoutSeconds, Stopwatch watch)
at CloudEngine.Actions.PowerShellHost.Invoke(InterfaceParameters parameters, CancellationToken token, UInt32 timeoutSeconds, ThrottlingDescription throttlingDesc)
at CloudEngine.Actions.InterfaceTask.Invoke(Configuration roleConfiguration, String startStep, String endStep, String[] skip, Nullable1 retries, Nullable1 interfaceTimeout, CancellationToken token, Dictionary`2 runtimeParameter, Boolean runInProcess, String storeName, String storeRoot)
VERBOSE: ActionPlan : ActionPlanStatus: Failed

Intro

I was updating an Azure Local Stack from 11.2509 to 11.2510. Doing the update, I hit an issue about the MOC ARB update.
The error above is not very helpful, but I found better information in these logs: “\NODE01\C$\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\MocArb\WorkingDirectory\CloudStore\log\agent-log-2692″

Showed that one of the nodes was not responding on port 45000 (not easy to read out of it but the key is in there):

health: provisioningStatus: lastError: version: downloadStatus:<> validationStatus: validationState: > placementStatus:<> uploadStatus:<> > locationName:\"MocLocation\" hardware: powerState:Running security: > tags: tags: > guestAgent:<> > ], Error []","Attributes":[{"Key":"CallerLocation","Value":{"Type":"STRING","Value":"virtualmachine.go:103 "}}],"DroppedAttributeCount":0,"Time":"2025-11-14T05:14:54.8439212-08:00"}],"Links":null,"Status":{"Code":"Unset","Description":""}}
{"verbositylevel":0,"name":"VirtualMachineImage Get name:\"85c19b633461f986a9c45610fbc24fcf44f5a3286749b-control-plan7ntf4-OSDisk-074ed046\" groupName:\"management\" ","traceid":"347d1d6e1d295f8deeffbf65ff5351b8","id":"aef6418a6f43d8a7","parentid":"0000000000000000","starttime":"2025-11-14T05:14:54.8451002-08:00","endtime":"2025-11-14T05:14:54.8451002-08:00","duration":"0.00s","entity":"","Annotations":null,"Attributes":[{"Key":"verbositylevel","Value":{"Type":"INT64","Value":0}}],"Events":null,"Links":null,"Status":{"Code":"Unset","Description":""}}
{"verbositylevel":0,"name":"VirtualMachineImage Get name:\"85c19b633461f986a9c45610fbc24fcf44f5a3286749b-control-planfmfj6-OSDisk-e5a7aa47\" groupName:\"management\" ","traceid":"45b6e4df4ab933179040fb0b16f63334","id":"fb6ad57b8537771b","parentid":"0000000000000000","starttime":"2025-11-14T05:14:54.8451002-08:00","endtime":"2025-11-14T05:14:54.8451002-08:00","duration":"0.00s","entity":"","Annotations":null,"Attributes":[{"Key":"verbositylevel","Value":{"Type":"INT64","Value":0}}],"Events":null,"Links":null,"Status":{"Code":"Unset","Description":""}}
{"verbositylevel":0,"name":"Location Get ","traceid":"245c2a49bb5aa308915fe60b0e046bfa","id":"933471a43c2d3dc2","parentid":"aacd10fff01c89d2","starttime":"2025-11-14T05:14:54.8451002-08:00","endtime":"2025-11-14T05:14:54.8451002-08:00","duration":"0.00s","entity":"","Annotations":null,"Attributes":[{"Key":"correlationId","Value":{"Type":"STRING","Value":"e9478eba-c15b-11f0-a715-1070fd9a23ba"}},{"Key":"verbositylevel","Value":{"Type":"INT64","Value":0}}],"Events":null,"Links":null,"Status":{"Code":"Unset","Description":""}}
{"verbositylevel":0,"name":"Cluster Registry Store Put","traceid":"87d50a2a9942c34c6a1e76ab2c901538","id":"4fee356130f70b71","parentid":"0000000000000000","starttime":"2025-11-14T05:14:54.8461389-08:00","endtime":"2025-11-14T05:14:54.8580181-08:00","duration":"0.01s","entity":"","Annotations":null,"Attributes":[{"Key":"verbositylevel","Value":{"Type":"INT64","Value":0}}],"Events":[{"Name":"163308140d6e535","Attributes":[{"Key":"CallerLocation","Value":{"Type":"STRING","Value":"cluster_registry_store.go:99 "}}],"DroppedAttributeCount":0,"Time":"2025-11-14T05:14:54.8461389-08:00"}],"Links":null,"Status":{"Code":"Unset","Description":""}}
{"verbositylevel":0,"name":"Save","traceid":"3b017e25f5d445a5712d74593b19e705","id":"bde9097ecdbd121b","parentid":"0000000000000000","starttime":"2025-11-14T05:14:54.8461389-08:00","endtime":"2025-11-14T05:14:54.8580181-08:00","duration":"0.01s","entity":"","Annotations":null,"Attributes":[{"Key":"verbositylevel","Value":{"Type":"INT64","Value":0}}],"Events":null,"Links":null,"Status":{"Code":"Unset","Description":""}}

In FCM I could see that the new MOC was deployed, so I was having 2 running MOC VMs.

In the Azure portal, the Arc Resource Bridge showed as “PostUpgrade” and “UpgradeFailed”.

The solution

I found that restarting the WSSDAgent service on the node that was causing the error, did solve the issue and the update could be retried.

Here is how to do it:

Stop-Service WssdAgent
Start-Service WssdAgent

I did not retry the entire solution update, instead I ran a new upgrade of the appliance with these commands:

az login --use-device-codeaz arcappliance upgrade hci --config-file "C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\MocArb\WorkingDirectory\Appliance\hci-appliance.yaml"

After this completed and I could see the Arc Resource Bridge was showing as 1.6 and succeeded, I could skip this step in the solution update and continue the update.