I’ve been working in a lab lately where I’ve been running into the above problem using AAD Connect’s Staging Mode. We’re at a point where we’re looking to swap out an existing custom FIM 2010 R2 solution (with the soon-to-be deprecated WAAD connector) for the more mainstream solution recommended today by Microsoft for a multi-forest-single-tenant configuration.
I didn’t find the error message (detailed below) all that helpful, and although I did recall seeing it in a FIM context a long time ago, it turns out that the cause (in my case at least) was not something that you would deduce from the above in a million years :). I am including the following for future reference when others might run into the same problem – and as in medicine you shouldn’t assume every symptom that looks the same has the same root cause. However, just like with Doctor Google, I am going to give you my remedy and let you discover the truth for yourself if and when you run into this yourself.
Symptoms
The problem is distinguishable by
- Inbound sync failing with the following:
Unable to persist entry. An error occurred, ..\ObjectNamespace.cpp(636), code 80230306, BAIL: MMS(3160): ..\tower.cpp(10789): 0x80230306 (The dimage has an anchor that is different than the image.)
- An explicit AAD disconnector (i.e. a pending export delete) exists for the matching source anchor of the above object in error
- No AAD connector is joined to the metaverse object for which the error is produced
- cloudFiltered = true (metaverse attribute value)
Cause
To reproduce the circumstances that generated this problem:
- The solution MUST be in STAGING MODE (prevents actual export to AAD)
- There must be an object already in the tenant matching the source anchor of an AD object which is IN SCOPE of AADConnect. In my case for an object to exhibit this error it had to meet the following criteria:
- Must have EA12 set
- Must be in an in-scope OU
- Must NOT be filtered from sync
- LAB: Must be in the corresponding “filter groups” (i.e. specified regional +/or CORP groups)
- LAB and PROD: no other form of filter can be in place (custom sync rule)
The following sequence of steps was found to consistently reproduce the problem:
- Move the object out of scope of AADConnect (in our case taking it out of the corresponding filter groups achieved this, but we could also have moved it out of scope of the selected OU)
- Allow delta import and delta sync to run – which triggers a disconnect (pending export DELETE)
- Allow the AAD MA delta delta sync cycle to run (IMPORTANT: this puts the AAD CS object into an EXPLICIT DISCONNECTOR state – noting that in STAGING mode the export is/can never applied to AAD)
- Move the object back into scope of AADConnect (add back into filter group(s))
- Allow delta import and delta sync to run – which SHOULD trigger a re-provision to AAD (pending export ADD) – but instead we get the above BAIL error.
Online investigation
Old TechNet Instructions existed for DirSync in the past with errors similar to this, but did not resolve the problem (full import/sync did not clear the problem because it was not OU related).
Other reported issues similar to the above both here and here.
Resolution
To resolve the problem there are at least 3 options:
- Full reinstall (the experience of some with this problem in the past – as with the above 2 links) – not what most would find acceptable in PROD
- Delete the AAD CS (possible – this is what I ended up doing today – after confirming we had this problem)
- Delete the AAD object and run a DI/DS to delete the AAD CS “explicit disconnector” object
Explanation (using MIIS/ILM/FIM/MIM sync principles):
- Once an object is in an EXPLICIT DISCONNECTOR state it can only be cleared in one of the following ways:
- It is allowed to be exported to AAD (i.e. the delete performed) – not possible in STAGING MODE
- It is deleted in the tenant directly (Remove-MSOLUser PowerShell commandlet) – not desirable if the object has permissions etc. you wish to retain
- It is commuted from an “explicit disconnector” to a “normal disconnector” – possible in FIM/MIM but not supported in AADConnect
- The entire CS is deleted – cost of 1-2 hours delete/import/full sync (AAD connector only)
- The entire Config is deleted and recreated – for me this came at the cost of approx 3 hours config and 5 hours full sync (all 6 connectors)
- While an explicit disconnector is present for an object matching the same sourceAnchor, any object exhibiting the above BAIL error will continue to do so until it is taken out of scope of AADConnect
Footnote
The outcome of the above is that we need to recognise and avoid the circumstances that lead to the above condition while in STAGING MODE – however this may be unavoidable in PROD during say a pre-go-live phase. As such these BAIL errors are entirely possible, especially with delete/adds happening in parallel with your existing FIM solution that you are about to swap out, and which has no concept of a “staging mode”.
Whenever this occurs in future a CS full delete/reload will most likely be required to resolve the problem – unless Microsoft can provide a means of targeting the disconnectors individually in the AAD CS and allow their conversion from EXPLICIT to NORMAL disconnector. However this is unlikely in the short term.
Nice find Bob!
Right with you Bob, good post. I had something interesting recently where I had this delete O365 user request sitting in the queue for export to AAD. A complicated long story but the CS object associated with that delete request was in a disconnected orphaned state. The solution was either delete the AAD CS or delete that orphaned CS object so that the export gets out of the queue. This was production so delete the whole CS would not be funny, so delete the CS object. I know, not supported and can cause ripples, but it is what it is I guess. Worked fine after that.
Thank you for the excellent post, Bob! (And for linking it in the AAD forum.)
I’m really missing the Joiner tab when using Azure AD Connect. This is just one instance. Prior to this I’ve only really had to delete a connector space twice in eight years in this job. Once was when the miiserver.exe process crashed on an import or sync (database connection hiccup perhaps) and kept doing it every time it tripped over the same garbled object. The other time was because of orphaned, disconnected unconfirmed exports in the old Live@edu MA (remember Hotmail?), which essentially had no import capability. The re-export took several *days* to complete over the December holidays. The technology has improved by leaps and bounds, but it still feels like we’ve gone full circle in terms of its deficiencies.