Critical XenDesktop Broker Bug 2

Yesterday afternoon we were struck by a bug that left us unable to enumerate applications, effectively resulting in a near 100% system-down condition.  We’ve been working with Citrix on an intermittent inability to log in with little visible error other than ‘All the Citrix XML Services configured for farm Farm1 failed to respond to this XML Service transaction. ‘ in the Storefront log.  When I received a report that “No desktop was available” I immediately duplicated the issue, then contacted the escalation engineer we were working with.  In short order he put together a team on his side to help review the log files of what was going on. The only change(s) that had been made in the environment were that some applications had been copied earlier, but that change had been tested and seemingly worked without error.

Storefront tracing was alread enabled using the commands, and Scout data was submitted

> Add-PSSnapin Citrix*
> Set-DSTraceLevel -All -TraceLevel Verbose

However, despite our best efforts the logins were continuing to fail.  It became apparenty through examining the Delivery controller event logs that this was a new issue, because we were in fact seeing exceptions in the application event log.

An unexpected exception occurred while the Citrix Broker Service processed an HTTP request. 
 
Error details: 
Request URL: 'http://citrixdc1.contoso.local/scripts/wpnbr.dll' 
Exception Type: 'Citrix.Fma.Sdk.Dal.DALDataStoreException' 
Exception Call Stack: '   at Citrix.Cds.Broker.DAL.BrokerCoreDAL.ReadApplicationResource(IDataRecord row)
   at Citrix.Cds.Broker.DAL.BrokerCoreDAL.ReadResources(DbDataReader query, Func`2 readResource)
   at Citrix.Cds.Broker.DAL.BrokerCoreDAL.EnumerateApplications(DbDataReader query)
   at Citrix.Cds.Broker.DAL.BrokerCoreDAL.EnumerateSelectedResources_Reader(CdsTx _tx, SecurityIdentifier userSid, Boolean enumerateAssignedPrivateDesktops, Boolean enumerateDesktopGroups, Boolean enumerateSharedDesktops, Boolean enumerateApplications, Boolean& applicationsSuppressed, DbDataReader _query, DbParameter _spRetVal)
   at Citrix.Cds.Broker.DAL.GeneratedBrokerDAL.EnumerateSelectedResources_TX(CdsTx _tx, SecurityIdentifier userSid, IEnumerable`1 groupSids, IEnumerable`1 denyOnlySids, IPAddress endpointIPAddress, String endpointName, Nullable`1 isViaAG, IEnumerable`1 smartAccessTags, Boolean enumeratePrivateDesktops, Boolean enumerateAOFUDesktops, Boolean enumerateSharedDesktops, Boolean enumerateApplications, String selectedBrowserName, Boolean& applicationsSuppressed)
   at Citrix.Cds.Broker.DAL.BrokerCoreDAL.EnumerateResources_TXW(CdsTx _tx, UserAccountDetails userDetails, EndpointDetails endpointDetails, Boolean retrieveIconHash, Boolean retrieveIconData)
   at Citrix.Cds.Broker.DAL.GeneratedBrokerDAL.<>c__DisplayClass5f1.b__5ef(CdsTx _tx)
   at Citrix.Fma.Sdk.Dal.BaseTxProvider.RunBatch(CdsTx tx, CdsTxAction txAction, String txName, Int32 retryNumber, Exception& originalException)
   at Citrix.Fma.Sdk.Dal.BaseTxProvider.Execute(String txName, CdsTxOptions txOptions, CdsTxAction txAction)
   at Citrix.Cds.Broker.DAL.GeneratedBrokerDAL.EnumerateResources(UserAccountDetails userDetails, EndpointDetails endpointDetails, Boolean retrieveIconHash, Boolean retrieveIconData)
   at Citrix.Cds.Broker.DAL.GeneratedBrokerDAL._ProcessUnhandledException(String methodName, Exception ex)
   at Citrix.Cds.Broker.DAL.GeneratedBrokerDAL.EnumerateResources(UserAccountDetails userDetails, EndpointDetails endpointDetails, Boolean retrieveIconHash, Boolean retrieveIconData)
   at Citrix.Cds.Broker.XmlHelper.GetResourcesForUser(UserAccountDetails user, EndpointDetails endpoint, Boolean retrieveIconHash, Boolean retrieveIconData)
   at Citrix.Cds.Xms.Wpnbr.AppDataTransaction.HandleRequest(IXmlMultiplexer multiplexer)
   at Citrix.Xms.XmlSupport.XmlPerf.WrapTransaction(Type t, Action transaction)
   at Citrix.Cds.Xms.Wpnbr.WpnbrServer.HandleRequest(HttpListenerRequest request, WindowsIdentity identity, DateTime startTime)'

This error returned nothing of use on the web as most references to DALDataStoreException were in different namespaces, like Cds.  After filtering the DC event logs to find the first occurrence of this error my colleague noted that the time was remarkably close to the time that he copied the applications.

I reviewed the “Configuration Logging” section in Studio and it became increasingly clear that link from copying the applications was likely the culprit.

Seemingly successful commands

Errors began appearing immediately

So what happened? Was some unsupported method of application copying used?  No, in fact.  The commands were tested on a single application, then when that was successful, repeated for all applications in a delivery group.

First,

Get-BrokerApplication "Windows Powershell" | Add-BrokerApplication -DesktopGroup "TestVDI2K8R2"

After launching and testing the app successfully, it was deleted, then the command below was issued.

Get-BrokerApplication | Add-BrokerApplication -DesktopGroup "TestVDI2K8R2"

Reversing this action resolved the errors

Get-BrokerApplication | Remove-BrokerApplication -DesktopGroup "TestVDI2K8R2"

So what happened?  Did we do something wrong, or unsupported?  Apparently not, according to a post from Citrix staff in the official forums.

Yes, the restriction of one delivery group per application is purely cosmetic in Studio’s app publishing wizard.

This was an intentional design, as the multiple association is meant to be an advanced use case. The process for discovering apps from the Start Menu and automatically publishing applications would become much more complex if we supported multiple delivery groups, as the VDA images are likely different between the groups.

The product fully supports publishing apps to multiple delivery groups, but when setting up these associations via PowerShell, it’s up to the administrator to ensure the VDA images are adequately similar so the application can be launched via the same command line in both groups.

Thanks for the feedback on showing the priority–I’ll raise this as an enhancement to the Studio team.

Unfortunately, something went awry with one or more of the applications that was copied in this way.  At this point the precise root cause isn’t known, but if you need to use this functionality, be on the lookout for potential problems!