Managing custom Oak indexes in AEM as a Cloud Service has traditionally been one of the more complex areas of AEM development.

Every time Adobe released a new version of an out-of-the-box (OOTB) index, teams had to manually merge their customizations into the latest index version, maintain compatibility, and validate deployments.

If you’re tired of continuously maintaining versioned custom indexes such as damAssetLucene-8-custom-1, cqPageLucene-2-custom-3, or manually merging OOTB index changes, then Simplified Index Management using Diff Index is the feature you’ve been waiting for.

Introduced as part of recent Oak enhancements, Diff Indexes allow developers to define only the differences (diffs) from Adobe’s OOTB indexes instead of maintaining complete index definitions.

Why Use Simplified Index Management?

Traditional AEMaaCS index customization requires:

• Copying entire OOTB index definitions.
• Creating new versioned custom indexes.
• Repeating merges whenever Adobe upgrades product indexes.
• Maintaining large and difficult-to-review .content.xml files.

With Diff Indexes:

✅ Only changes are stored.
✅ Adobe automatically merges changes with the latest OOTB index.
✅ Future upgrades become significantly easier.
✅ Smaller, cleaner, and easier-to-maintain index definitions.
✅ Reduced risk during AEM upgrades.

How Does Diff Index Work?

A special index node named diff.index is created under /oak:index.

/oak:index
   └── diff.index
         └── diff.json

The diff.json file contains both:

• Customizations to existing OOTB indexes.
• Fully custom indexes.

Note:- All the custom index entries must be present under single diff.json file. You should not create different diff.json files for different indexes.

During deployment, AEM automatically merges these changes with the latest available OOTB index definition and creates a new versioned index if required.

Migrating Existing Custom Indexes to Diff Index

Step 1: Create the diff.index Definition

Create the file, if it does not exit:

ui.apps/src/main/content/jcr_root/_oak_index/diff.index/.content.xml

add below code to .content.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root
    xmlns:jcr="http://www.jcp.org/jcr/1.0"
    xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
    jcr:primaryType="nt:unstructured"
    type="lucene"
    includedPaths="/same"
    queryPaths="/same"
    async="async">

    <diff.json jcr:primaryType="nt:file"/>
</jcr:root>

Step 2: Create an Empty diff.json

Create:

ui.apps/src/main/content/jcr_root/_oak_index/diff.index/diff.json

Step 3: Download Existing Index Definitions

1. Open the AEM as a Cloud Service Developer Console.
2. Select one specific Author instance (avoid selecting All Authors).
3. Choose:

Status Dump → Oak Indexes
Output Format → JSON

4. Click Get Status.
5. Click Download.

A raw JSON representation of all indexes will open in a new browser tab.

Copy the complete JSON.

Step 4: Generate the Diff Index

Open the Oak Index Definition Analyzer:

https://oak-indexing.github.io/oakTools/indexDefAnalyzer.html

Paste the downloaded JSON into → Index Definitions (JSON) → Click → Analyze

The tool generates the corresponding Diff Index definitions.

Step 5: Add the Migrated Index to diff.json

Suppose you’re migrating:

aemcq5tuorials.slingFolderDamLucene-custom-1

Locate:

aemcq5tuorials.slingFolderDamLucene

inside the generated output from Step 4 Diff Index. Copy that entry into diff.json.

For easier verification, consider renaming it:

aemcq5tuorials.slingFolderDamLucene → aemcq5tuorials.slingFolderDamLuceneDiffIndex

Please refer below example on how your diff.json should look like:

{
"aemcq5tuorials.slingFolderDamLuceneDiffIndex": {
    "type": "lucene",
    "evaluatePathRestrictions": true,
    "includedPaths": "/content/dam/library/workfront",
    "queryPaths": "/content/dam/library/workfront",
    "async": [
      "async",
      "nrt"
    ],
    "compatVersion": 2,
    "aggregates": {
      "sling:Folder": {
        "include0": {
          "path": "jcr:content"
        }
      }
    },
    "indexRules": {
      "sling:Folder": {
        "properties": {
          "wfReferenceNumber": {
            "useInSuggest": true,
            "name": "jcr:content/metadata/wfReferenceNumber",
            "nodeScopeIndex": true,
            "useInSpellcheck": true,
            "propertyIndex": true
          }
        }
      }
    }
  }

This makes it easy to confirm that your queries are actually using the migrated index and not the old one. This we are going to validate using query performance tool in coming steps.

Step 6: Validate the Diff Index

Before deployment, validate the JSON using: https://oak-indexing.github.io/oakTools/diffIndexVerifier.html

The verifier checks:

• JSON syntax
• Index naming conventions
• Structural correctness

Always validate before committing code.

Step 7: Remove the Old Index Filter

After migrating successfully, remove the old index entry from:

ui.apps/src/main/content/META-INF/vault/filter.xml

Example:

<filter root="/oak:index/aemcq5tuorials.slingFolderDamLucene-custom-1"/>

This ensures the old index is no longer deployed. The beauty of diff index is that, it remove all old versioned index from the repository.

Step 8: Deploy the Code

Deploy the updated package using your normal Cloud Manager pipeline.

During deployment, AEM automatically:

• Merges the diff with the latest OOTB index.
• Creates a new versioned index.
• Triggers indexing if necessary.

Step 9: Verify the Migrated Index with Query Performance Tool

Use your favorite AI assistant (ChatGPT, Adobe Agent, etc.) and provide the original .content.xml of the index. Ask it to generate a query that attempts to use every indexed property or you can using your existing query if you have it handy.

Important: Do not use the generated diff.json as input. Use the original index definition so that verification remains unbiased.

Run the generated query using below URL if you are testing in local:

/libs/granite/operations/content/diagnosistools/queryPerformance.html

or below path if you are using cloud instance

/ui#/aem/libs/granite/operations/content/diagnosistools/queryPerformance.html

Success Criteria

✅ Query plan references the newly created Diff Index.

Failure Criteria

❌ Query plan still references the old custom index.

If the old index is being used, revisit the migration and validation steps.

Creating Brand-New Custom Indexes with Diff Index

One of the biggest advantages of Simplified Index Management is support for fully custom indexes.

Important

When creating a completely new custom index:Because you don’t any base index definition, that’s why you need to add all properties inside diff.json. You don’y need old oak:index definition with .content.xml for this one.

Example:

{
"custom.aemcq5tuorialsUserActionIndexDiffIndex": {
  "evaluatePathRestrictions": true,
  "includedPaths": ["/var/dam/custom-useraction-node"],
  "queryPaths": ["/var/dam/custom-useraction-node"],
  "type": "lucene",
  "async": "async",
  "compatVersion": 2,
  "indexRules": {
    "nt:unstructured": {
      "properties": {
        "processed": {
          "propertyIndex": true,
          "type": "Boolean",
          "name": "processed"
        }
      }
    }
  }
 }
}

Key Rules for Fully Custom Indexes

• Index name must contain a dot (.).

Example: custom.aemcq5tuorialsUserActionIndexDiffIndex

• includedPaths must not be empty.
• includedPaths must not point to /apps or /libs.
• queryPaths should typically match includedPaths.

Known Limitations

The current implementation of Simplified Index Management has a few limitations:

• Simplified Index Management is available only in supported versions of AEM as a Cloud Service and recent AEM SDKs.
• Simplified Index Management is currently not available for indexes that include /apps or /libs in their includedPaths configuration.
• Indexes Without includedPaths Cannot Use Simplified Index Management
• Indexes should have queryPaths and queryPaths should typically match includedPaths.

Considerations and Troubleshooting Tips

Although Diff Indexes simplify index management considerably, there are some limitations and considerations.

1. Supported Only on Recent Oak Versions

Diff Index support requires newer Oak versions available in recent AEM as a Cloud Service releases and recent AEM SDK versions. Older SDK versions do not support this feature.

2. Fully Custom Indexes Require Complete Definitions

For brand-new indexes, there is no OOTB index to merge with. You must provide the entire index definition inside diff.json.

3. Merge Warnings Can Occur

If:

• JSON is malformed
• Unsupported properties are used
• Invalid paths are configured
• Merge conflicts exist

Oak writes warning properties (warn.01, warn.02, etc.) under diff.index.

Always inspect warnings after deployment.

4. Local/RDE Reindexing Can Be Expensive

When using local SDK or RDE, modifying diff.json immediately creates new index versions and starts reindexing.

For large repositories, this can be time-consuming. Avoid excessive index modifications in large shared environments.

Final Thoughts

Simplified Index Management using Diff Index is a major improvement for AEM developers working on AEM as a Cloud Service.

By storing only the differences instead of full index definitions, teams can:

• Reduce maintenance overhead.
• Simplify upgrades.
• Improve readability.
• Minimize merge conflicts.
• Future-proof custom indexing implementations.

If your project still maintains versioned custom indexes manually, now is a great time to migrate to Diff Indexes.

Happy indexing!