I’ve been thinking a lot lately about the inherent scalability challenges of the FIM sync engine as we seek to push it harder and harder, throwing more and more identities and other related objects at it.  Is there a limit, and how long before we hit it?   We already know to expect long initial load/full sync times for systems with upwards of a million objects under synchronization, and although each service pack seems to include new ways of improving this, at what point do we need to step back and ask ourselves the inevitable question … is there another way?

Inspired by some left field ideas from the wider FIM community @ TEC2012 Europe this week, I’ve started to conceptualize various ways in which one might go about mitigating against the inevitable performance degradation that comes with increased numbers, in particular the number of relationships we’re modelling between them.  I’ve been thinking how maybe the initial load should be achieved by bypassing the sync engine (I recall Blain Checkley posting about doing this not long ago), and then I thought again about the new async FIM MA export mode and thought perhaps that isn’t the place to focus on so much now, given the through-put improvements being seen.  That lead me to the next thought … how do I hedge against the inevitable long initial import/sync times?  That was when it struck me … maybe I already have the key and didn’t realize?

Here’s my thought … take my Replay MA idea, only this time what if we use it not only on the regular instance of the FIM sync engine, but on a completely different one?  What would that do for us?

Thought goes like this … use the FIM MA for all updates to the FIM Portal … as it always should.  However, use a secondary FIM Sync engine without any FIM Portal of its own at all.  We all should know by now that every FIM service can be linked with one and only one sync engine right?  However, there’s nothing to say you can’t take a Replay of the FIM MA and play it through a completely separate FIM sync server instance!  What does that give us?

What this should allow is the off-loading of everything but import flows for the FIM service to (one or more) separate sync services, thereby distributing the load.  While this will still mean that the FIM service will need to be loaded before any downstream sync (sourced from a FIM MA replay) can occur, the time it takes to reach the initial point where the export to the FIM service can begin could be much less than it would otherwise.  What’s more, the subsequent times for “downstream” FIM sync service sync/exports should be much less too.  The only challenge I can see is how to join/project from the FIM Replay MA onto the FIM metaverse of the secondary FIM sync services given we can’t use the mv object ID in the same way as before.  Still, that should be manageable.

Will give this some more thought, but in the meantime I’m excited by the possibilities here.