Ariovis
Expert insight — Netwrix Identity Manager Production Operations

IAM automation must also know when to stop

Safety Thresholds, Simulation Mode, Rollback, Restore Points and controlled deployments in Netwrix Identity Manager Production Operations. A senior Ariovis architect and RUN lead walks through a major anonymised incident and several recurring situations we face on platforms deployed on-premises and in SaaS.

The incident that shapes our RUN practice

We operate Netwrix Identity Manager — formerly Usercube — in demanding environments. The IGA platforms we support feed Active Directory, Microsoft Entra ID, Exchange, ServiceNow, SAP S/4HANA, HR systems, business applications, PowerShell scripts, batch jobs and SQL databases through several agents and Azure DevOps or equivalent CI/CD Pipelines. They compute, provision, reconcile and delete tens of thousands of objects a day. IGA is no longer an administrative tool sitting next to the information system. It is a Production component in its own right.

In a large organisation, a configuration change is prepared and pushed through the deployment pipeline. The Acceptance Environment does not sufficiently reproduce the consequences of the change in Production. A forced cascade-delete mechanism interprets a large set of groups and roles as targets for deletion. The change propagates to Active Directory. More than 33,000 groups then have to be restored in a hurry. Several critical applications are affected, because their access relied directly or indirectly on those groups.

We do not dramatise the episode. We calmly ask the central question: how could a change of this magnitude reach Active Directory without an intermediate stage measuring its Impact Analysis and pausing its application? The point is not to name someone responsible. The point is to understand which protections should have interrupted the chain.

A safe automation is not one that never gets it wrong. It is one whose error can be detected, contained and reversed.

What the incident reveals: a job that succeeds too well

A deployment chain can be technically correct and still be under-protected. Code passes lint checks, gets merged, deploys automatically, starts without error and correctly executes the logic it was given. The logic itself can nevertheless produce a business outcome that is simply implausible. In our case, the cascade delete works, the pipeline works, the provisioning works, Active Directory carries out the requests it receives. That is exactly the problem.

No single component necessarily hits a technical error. The overall outcome is unacceptable all the same. The hardest incidents are not always caused by a job that fails. They can be caused by a job that succeeds at exactly what a bad configuration told it to do.

Netwrix Identity Manager separates computed decisions from their materialisation in target systems. A rule may decide that a group must be removed, an account disabled, a role granted, an attribute changed. Provisioning is the step that applies that decision to Active Directory or to another target. That separation should be used as a guard-rail: before a new job is allowed to write in Production, we run it in a configuration where it can pull data, compute rules, produce diffs, count expected operations and list the impacted objects, without immediately altering the target.

Depending on the context this may be a Blocked Mode, a Simulation, a duplicate job, a copy of the job with no provisioning step, a service account with no write rights, or an Acceptance Environment fed with representative data. We do not immediately grant a new rule the right to act. We first ask it to show us what it intends to do.

After the incident we introduced a duplicate job. It takes the same data, the same rules, the same computations, the same classifications, the same object selections, but does not carry out the destructive operations in Active Directory. Before deployment it produces the number of groups involved, the number of roles removed, the affected populations, the main variations, the expected deletions and any objects shared between several applications. The output is reviewed before the writing job is authorised. This is not a perfect protection: divergence between the two jobs can breed false confidence. Production Operations must confirm that the computation steps are common or comparable and that only the write step differs. A useful simulation reproduces the Production decision without holding the power to execute it.

The guard-rails we anchor into RUN

Separating computation from execution is the first guard-rail. It is not enough on its own. Here are the complementary protections we tune, incident after incident, on our Netwrix Identity Manager RUN engagements.

  1. 01

    Safety Thresholds as a business hypothesis, not a performance dial

    Netwrix Identity Manager can enforce limits on some jobs. A threshold is not just a performance setting; it is a safety hypothesis. A typical job changes a few dozen groups; a real reorganisation may legitimately change several hundred; the deletion of tens of thousands of groups is implausible in the current context. A threshold can Block the job, force a validation, raise an alert or request a Sanity Check. There is no universal number: it depends on the identity and group counts, reorganisation frequency, mass HR events, known peaks, restore capacity and the risk associated with a partial execution. A threshold does not express the maximum volume the platform can handle. It expresses the maximum volume the organisation accepts to let through without a fresh check.

  2. 02

    A threshold can also be too low

    A mass HR event affects more than a thousand users. The existing Batch Limit prevents the full run. Some people are updated, others stay in the old state, department-driven rights become inconsistent, the team splits the operations by hand. The threshold is then raised to 1,000 after analysis and Acceptance testing. Raising it is not an automatic weakening of security: it comes with a Simulation, a volume check, business validation, post-execution monitoring and a mechanism able to block operations that go clearly beyond that value. A poorly calibrated guard-rail can stop an error, but it can also cause a partial update. Production Operations must tell a large volume apart from an implausible one.

  3. 03

    Reconciliation, to avoid mistaking a decision for reality

    After the write, Netwrix Identity Manager can read the target systems back and compare observed state to expected state. The fact that the platform requested an operation does not prove that the operation produced the expected result. Reconciliation exposes an attribute different from the computed value, a missing role, a group still present, an unauthorised account, an entitlement added directly in the target, a provisioning step that did not happen, or a partially applied state. A leaver whose workflow appears completed may coexist with an Entra ID account still active, a mobile session still usable, groups still present and an Exchange step that did not run. Provisioning tells us what the platform asked for. Reconciliation tells us what the information system actually contains.

  4. 04

    Avoiding destructive replays

    A leaver flow sometimes has to be re-run after an incident. It must tell apart someone never processed, partially processed, fully processed, or someone whose new date or situation justifies a fresh run. On a daily leaver script that keeps picking up already-processed people, we see deactivations, out-of-office messages, notifications and downstream updates fire several times, producing double audit trails, inconsistent states and a real difficulty in reconstructing what actually happened. We fix the selection criteria, the end-of-flow state, the replay conditions, the tests in the Acceptance Environment and the post-Production checks. A re-runnable job must know how to resume what is missing without repeating what is already done — an Idempotency property, not just a wish.

  5. 05

    Watching the jobs that never start

    A job can produce no error simply because it never ran. A synchronisation is expected every hour, automatic triggering stops, no new job appears in the history, manual execution still works. The issue is in the Scheduler, its host or its lock mechanism — not in the synchronisation itself. We check the Scheduler host, the service state, the lock files, recent reboots, the execution history and the expected-but-missing jobs. Monitoring cannot only alert on red jobs. It must know that a job was expected at a given time and that it does not exist. A missing job never turns red. Its appointment must be watched, not just its result.

  6. 06

    Testing the differences between environments

    A feature can work in DEV and in Pre-production and then fail in Production. Causes include service-account permissions, Active Directory delegations, a different secret or certificate, an authentication mode, actual volumes, batch size, a Kerberos ticket, the execution server, network access, Scheduler configuration, script or agent version. A job must send passwords to new joiners; it works in Acceptance; in Production some joiners receive nothing. Two independent causes produce the same symptom: the account used lacks the right reset delegation, and a batch limit silently excludes some pending joiners. RUN walks the whole chain: is the person selected, does the data exist, is the password reset, which technical identity performs the action, does it actually hold the right, does the person drop out of the run because of a limit, is the message sent, is the success state recorded only after the actual send. An Acceptance test does not cover Production differences until permissions, volumes and execution components are represented.

  7. 07

    Clearly separating scripts, Pull Requests and CI/CD checks

    We separate scripts used in Production from ad-hoc scripts, migration scripts, test scripts and diagnostic tools. A test or one-off script must not be runnable by accident as a Production job. Every change is tied to a user story, an incident or an approved request, an author, a review and evidence of Acceptance testing. The pipeline can check the configuration structure, the modified files, the presence of required metadata, Pull Request approval, target branch, deployment stages and technical checks. We never claim CI/CD understands the business consequence of a NIM rule. The pipeline confirms that the change follows the delivery process. Simulation confirms what that change will do to identities and rights.

None of these guard-rails is enough on its own. A threshold does not replace Acceptance testing. A Pull Request does not prove that the business consequence is acceptable. Safety comes from their combination.

Back up, restore, roll back — including on SaaS

After the incident, a backup is taken before every sensitive deployment and a rollback plan is written before, not after. On SaaS this discipline takes a specific shape, because the customer or integrator does not directly control the hosted infrastructure or the platform backups. That does not mean there is no way back.

What must be backed up, and why it is not one single object
A backup may cover the Netwrix Identity Manager database, the configuration, the scripts, the agents, parameter files, secrets and certificates, Active Directory, and the groups and roles changed in the target systems. A NIM backup does not necessarily suffice to restore Active Directory if deletions have already been applied there. Conversely, restoring Active Directory without realigning NIM may lead the platform to request the same deletions again on the next run. The Recovery Procedure must state what has to stop, which component is the Restore Point, in what order to restore, how to prevent provisioning from starting again, how to read the target systems back, how to handle business events that happened in the meantime and how to check that consistency is back. Restoring a directory without neutralising the decision that changed it sometimes sets up the same incident to happen again.
Rolling back on Netwrix Identity Manager SaaS
When a file, a synchronisation or an action causes a severe change to the state of the SaaS platform, we can escalate to Netwrix as a partner to study a restore of the SaaS instance to a state prior to the operation. This mechanism is not a self-service button, not every action is always reversible, no Restore Point necessarily exists at any given second, and restoring the NIM instance does not automatically restore Active Directory or downstream applications. We present it as a coordinated operations procedure: we identify the time of the file, job or deployment involved, stop jobs likely to worsen the situation, gather the useful evidence (instance, precise time, job, file, deployed configuration, first observed consequences), open and drive the request with Netwrix, confirm the available Restore Point and its scope with the vendor, assess the impact of data created or modified after that point, and Netwrix then performs the SaaS restore according to the agreed procedure. On Netwrix SaaS, rolling back is not clicking Undo. It is finding the right recovery point, coordinating the restore with the vendor and then rebuilding consistency with the target systems.
Three states to keep apart when rolling back
A wrong HR file is imported at 10:00. It updates several structures in NIM, some reporting relationships, some presence states and the computed rights for a population. Some operations have already been provisioned. The goal is to get back to 09:59. Three states must be told apart. The NIM platform state — the identities, relations, decisions and workflow states in the SaaS database; a restore run by Netwrix may bring it back to an earlier point, within the boundaries confirmed by the vendor. The source state — the HR system or source file must be corrected or neutralised, otherwise the next synchronisation may reintroduce the same situation. The target state — Active Directory, Entra ID and applications may already have received some operations; restoring NIM does not necessarily bring them back. Writes must be stopped, targets restored or fixed, their state read back, reconciled and legitimate events replayed. Getting NIM back to its previous state is not enough if the wrong file is still available or if the target systems have already materialised its consequences.
The partner's role in a SaaS restore
Ariovis is not a simple administrative relay. We qualify the incident, stop the jobs, pin down the reference time, assess the business impact, tell Netwrix what has to be restored, confirm with the customer the potential loss of changes made after the reference time, coordinate the third-party systems, prepare the reconciliation, watch the jobs after resumption and document the timeline and actions. The partner knows the customer's configuration, jobs, sources, target systems, provisioning rules and business priorities. Netwrix owns the SaaS infrastructure and the available restore mechanisms. The vendor can restore the instance; the partner must know which state to recover to, and how to reconnect that state to the rest of the information system.
The rollback plan must precede the deployment
Before a sensitive change we define which event will halt the deployment, which volume will be considered abnormal, which jobs will be paused, which backup exists, which SaaS Restore Point could be requested, who calls Netwrix, which targets will need to be restored separately, which post-Restore-Point data may be lost, how legitimate events will be replayed and who authorises the resumption of jobs. On SaaS we also confirm upfront with partner and vendor the scope of a possible restore, the available time granularity, response times, the data covered, the exclusions and the information needed to open the incident. A rollback that only exists in the project team's head is not yet a Recovery Procedure.
Deploying at night is not enough
We schedule sensitive deployments in the evening to reduce user exposure and allow immediate observation. We are careful not to treat the time slot as a sufficient guard-rail. Evening deployments cut simultaneous activity, keep a dedicated team on the change, preserve time to restore and let us watch the first jobs before resumption. They do not replace Simulation, thresholds, backup, Impact Analysis, the rollback plan, the presence of authorised decision-makers or a real ability to reach the vendor. Evening buys time to react. It does not prevent a bad decision from being executed.
Post-deployment checks
A deployment is not finished when the pipeline turns green. We run a Sanity Check covering the number of objects added, changed and deleted, the affected AD groups, errors and warnings, expected jobs, operations blocked by thresholds, gaps between NIM and targets, main populations, joiners and leavers of the day, critical applications, scheduled scripts, Scheduler state, service accounts and source freshness. It also includes precise functional scenarios: a representative joiner, a leaver, a mover, a rehire, a user with several positions, a contractor with an end date, a critical application entitlement. The pipeline confirms that the deployment ran. The Sanity Check confirms that the IAM service still produces the right decisions.
On-premises and SaaS: same principles, different responsibilities
On-premises the customer controls the server, the database, the agents, the backups, the scripts, the scheduling and the recovery procedures; they must be able to restore these in the right order. On SaaS, Netwrix owns the hosted infrastructure and the instance backups; the customer and Ariovis own the local agents, the files, the sources, the exported configurations, the scripts, the targets, the pipeline and the incident timeline, and coordinate with Netwrix a potential restore of the instance to an earlier point. In both cases, restoring NIM must be followed by reconciliation with Active Directory, Entra ID and applications. The difference between on-premises and SaaS changes who performs the restore. It does not remove the need to understand what must be restored and what must be reconciled.

What our clients recognise, and where we stand

An IAM lead or a Production Operations lead running Netwrix Identity Manager usually recognises several of these situations: an HR file has been processed but contained bad data; a rule produced far more changes than expected; a job never fired; a leaver was replayed; a job worked only in Acceptance; a service account lacked the right delegation in Production; a batch limit silently excluded some people; a synchronisation succeeded but the target stayed different; a direct change in AD created Drift; a NIM restore risks triggering another provisioning wave; nobody had defined in advance how to request a SaaS rollback; the backup exists but the full procedure has never been rehearsed.

These situations do not mean the platform is badly designed. They show that an IGA becomes a full-fledged Production component and requires operational practices in proportion with its power to act. We surface with precision the native capabilities of Netwrix Identity Manager — expected-rights computation, the separation between synchronisation and provisioning, configurable jobs, workflow states, volume limits, execution history, error visibility, correlation, resource classification, property and role reconciliation, provisioning verification, unauthorised-account detection, comparison of theoretical and observed state — and we surround them with separated environments, simulations, no-write jobs, calibrated thresholds, backups, recovery procedures, Pull Requests, CI/CD checks, a representative Acceptance testbed, rollback plans, post-deployment checks and monitoring that also watches the jobs that never fired.

We do not try to slow every change with process for its own sake. Not every change deserves a vendor-coordinated restore, an evening deployment, a full Simulation or a crisis validation. The level of control depends on how destructive the change is, on the number of objects concerned, on the criticality of the applications, on reversibility, on the quality of Acceptance testing, on the ability to restore and on user exposure. Editing an email template is not treated like a new group-deletion rule. We do not protect every change in the same way. We reinforce control when the automation is given the power to change many objects or to remove access.

What we take away

In the original incident, the platform had been given a configuration asking it to delete a very large number of groups. Provisioning applied that decision. RUN then restored the objects and hardened the chain: backup before deployment, duplicate job, no-write computation, volume checks, calibrated thresholds, better Acceptance testing, evening deployment, Sanity Check, rollback plan. On SaaS, we can also coordinate with Netwrix a restore of the instance to a state prior to the file or the action involved, within the Restore Point available and the scope confirmed by the vendor. That restore is only one step. We then have to neutralise the cause, reconcile the target systems, replay legitimate events, verify decisions and gradually resume jobs.

An IGA platform does not become safe because it automates changes correctly. It becomes operable when we can see what it is about to do, prevent it from going too far, and return to a controlled state when reality does not match the expected decision.