I had an awkward moment or two today in which my fundamental understanding of outbound FIM declarative sync rules was put to the blow-torch, and for a minute there I thought I was probably had a corrupted Active Directory CS and was facing a complete deletion and reload.  Thankfully I avoided this, but let me take you through the experience in case you find this happening with you one day … and facing that awfully lonely moment when you feel you are in the hot-seat of a live environment and the buck stops with you!  Rest assured that a confidence in your own ability, coupled with a methodical process of elimination will see you through …

Earlier I had successfully set up a heap of pending exports in my AD management agent, but shortly afterwards a dreaded “stopped-database-connectivity” exception on one particular full sync.  I had an overloaded environment to deal with which needed some serious housekeeping … and as soon as I saw the SQL error I knew I could avoid it no longer.  Several service restarts later, as well as reclaiming of disk space and archiving 000s of run histories, I tried re-baselining my sync server, only to find that at the end of it my pending exports had … disappeared!

I retraced my steps and found that I hadn’t missed anything … the pending exports were just gone, even though the metaverse and target CS objects were still just as inconsistent as they were before.  I had renames, updates to title, manager and location, but none of them would manifest themselves as pending updates.  Enter troubleshooting mode …

I located a sample MV user object and observed the ERL contained the correct “applied” ERE, so I tried a full sync preview of the FIM MA connector … nothing!  No “skipped-not-precedent” warnings or anything.  Just no export updates to AD were there at all.  I re-read the section near the end of this reference, but couldn’t see anything missing at all.  I stumbled upon this forum post (and a familiar face therein) but still nothing rang any bells for me.

I decided the ERE must be corrupted, and so I figured I could recreate it specifically for this one user by

1. updating my SR workflow to include a REMOVE SR before an ADD SR (I do this all the time now … would now  argue this is should be best practice, but that’s another story):
2. creating a temporary set with just my user; and
3. creating a temporary set transition MPR to fire my workflow for just this user.

So now I could test and retest with just my user without invalidating my existing sync server baseline – simply by disabling and re-enabling the MPR.  I was sure I would see my EAF appear this time, but alas, nothing.  The ERE was created but went straight to an applied state, and a great big duck egg when it came to the number of pending EAFs.  Not good.  Again I did a preview, but again there was absolutely no clue as to what was wrong … the user/ERL/ERE/SR integrity was all there, and yet no joy.

I was starting to get concerned now.  I had noticed that during the re-baselining process I had set up some pending exports to FIM as well as to AD, and just hadn’t paid much attention to them … just kicked off the export and subsequent DI/DS.  Looking closer now I could see that a heap of DREs had been deleted.  I couldn’t work out why, but I had actually resolved a couple of hundred sync errors by causing a rejoin of incorrectly disconnected AD CS objects, and I figured this was just some sort of delayed clean-up.  Still, I wasn’t comfortable I was on top of this, and checked why we had a need for DREs at all.  I discovered an existence test, and a set of DREs which might have driven an MPR at one stage, but concluded that this was now redundant and I should remove the existence test …

By this time I felt I was clutching at straws really … I couldn’t see anything visible to fault the SR or ERE at all, and yet this DRE thing was enough to make me think that if I perhaps bit the bullet and removed the existence test I would thereby force an update of the SR to import to the sync server.  So I made the change, and followed it by a DI/DS of the FIM MA to bring in the update of the SR.  Given that the first step after doing this is always to run a FS on the FIM MA, I kicked this off and turned my attention to something else.  However, shortly afterwards I turned back to see what I had been hoping for, but had been bracing myself NOT to see … there were the counters for my pending exports to the AD MA happily growing again!  By the time I had run a FS on all my MAs my pending exports were still there, and I could breathe again.

So in summary, if you have problem with a seemingly correct ERE and SR, and you are getting no joy, don’t discount the possibility that it is your SR at fault here.  In the process of restoring some lost joins I had to make a couple of changes to some of the inbound and outbound flows (including a couple of non-declarative attribute flows to restore properties to the metaverse), so it is possible that in the process of doing this I may have invalidated this SR somehow.  Whatever it was, by simply modifying the SR (and I would have faked a change if I hadn’t had the existence test to remove) I was effectively able to flush the entire process out and reload.  Every now and then it seems the ERE process does seem to get into a knot if you move the carpet out from under it.

So my faith is restored, but with the caveat that sometimes FIM is a bit like your PC or laptop … sometimes nothing works like a reboot.  You don’t have to scrub out everything and start again, but sometimes it pays to be prepared to completely reconstruct things selectively.  I guess I had a hunch it would take something like this because I’ve had similar experiences before with a dodgy EAF in a SR … where it didn’t matter how many times I modified the EAF the sync engine seemed to completely ignore my change until I deleted and completely recreated my EAF (not the entire SR).  It is experiences like this which reinforce my opinion that FIM will eventually do the right thing … you just need to be VERY patient at times.