Duplicated OSGI Configurations after Upgrade

Duplicated OSGI Configurations is one of the most common issue that each developer face while performing upgrade from AEM 6.0 to higher version. I have also faced duplicated osgi configurations after upgrade multiple times so decided to write this post on this. Recently i have upgraded AEM 6.3.3 instance to AEM 6.5 and faced duplicated osgi configurations, in this post i will cover steps to fix it.

Analysis of duplicated osgi configurations:-

I went to felix console and saw many factory configuration got duplicated like Job Queue configurations, Logger configurations, CryptoDistributionTransportSecretProvider configurations, LoginModuleImpl ,  TokenLoginModule , GuestLoginModule etc.

One thing also you will notice that most of these configurations do not exist in the customer packaging or code base. These are out of the box.

Impact of duplicated osgi configurations:-

Ideally duplicate factory configurations should not have any (negative) effect. Since configurations are sorted (by ranking) and when a configuration is used, the first matching configuration is used.

System would not be affected if you have more duplicate factory configs, like for example logger configurations.

Note:- Only drawback is, it is hard for administrator to identify which one to edit and on start up all duplicated services gets executed.

Removing duplicated osgi configurations:-

The primary cause of this issue is due to a corrupt installer folder under launchpad. The simplest fix is to try this:

  1. Stop AEM instance
  2. Rename crx-quickstart/launchpad to launchpad.bak
  3. Start AEM – it will rebuild the whole launchpad folder and its installer subfolder

If you still see duplicate OSGI configurations, follow below steps:-

Removing duplicated osgi configurations is not a straight forward approach, i have followed below three steps to completely remove duplicated osgi factory configurations.

Thanks to Andrew Khoury, he has written a utility to find and remove duplicates.
Visit https://github.com/cqsupport/felix-osgi-utils/tree/master/felix-osgi-config-repair to download the utility and follow below commands:-

Usage:
    java -jar felix-osgi-config-repair.jar /path/to/config/directory [--delete]

  e.g. output duplicate config files without deletion:
    java -jar felix-osgi-config-repair.jar launchpad/config

  e.g. output duplicate config files with deletion:
    java -jar felix-osgi-config-repair.jar launchpad/config --delete

It will remove most of the duplicate configuration, but this utility doesnot guarantee 100% removal of duplicates. When i ran this utility in my upgraded instance it has removed around 380 duplicates out of around 420.

In order to remove remaining duplicates adobe team has provided one script which removes corrupted serialization file from launchpad config. Below is the shell script that i run after this:-

#!/bin/bash

# This script identifies and deletes (if used with '-x' parameter) duplicate OSGI factory configurations.
# Before executing this script please check the following requirements:
# - Update 'launchpadDir' variable.
# - Create a full backup of AEM.
# - Shutdown AEM. Never ever execute this script when AEM is running! For example check 'ps uax | grep [j]ava'.


# Full path to the AEM launchpad directory. Shell wildcards are allowed.
launchpadDir="/opt2/work/wlp-usr/servers/cq?/sling/_*/launchpad"


# We need a temporary file
file="$(mktemp)"

if [ "$1" != "-x" ]; then
  dryrun="Dry run: "
fi
# List all .config files and search for factory configs (i.e. foo/bar/190bf5a9-231a-4388-8354-018b4ec4d155.config)
find ${launchpadDir}/config -name "*\.config" | grep "[a-z0-9]\{8\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{12\}\.config$" | while read config; do
  # Identify uuid
  uuid="$(echo "${config}" | sed "s/.*\([a-z0-9]\{8\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{12\}\)\.config$/\1/")"
  # Identify factory path
  factoryDir="$(echo "${config}" | sed "s/\(.*\)\/\([a-z0-9]\{8\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{12\}\)\.config$/\1/")"
  # Create a checksum of the config without the uuid. This will create identical checksums for identical configurations
  checksum="$(cat "${factoryDir}/${uuid}.config" | sed "s/${uuid}//g" | sort | md5sum | sed "s/ .*//")"

  # This will create a list of checksum and filename
  echo "${checksum} ${config}" >> "${file}"
done

# Sort the list, count checksums them and list duplicates only.
# Eventually we get a list of checksums (of all duplicated configs).
cat "${file}" | sort | awk '{print $1}' | uniq -c | grep -v "^[ ]\+1 " | awk '{print $2}' | while read checksum; do
  cnt=1
  # Search for a checksum in the temp file, print the filenames and do a filesystem list of them, sorted by time (oldest first)
  ls -tr $(grep "${checksum}" "${file}" | awk '{print $2}') | while read configfile; do
    # Get the last modify time
    timestamp="$(stat "${configfile}" | grep ^Modify | sed "s/^Modify: //")"
    # We count the amount of duplicates. If our counter is 2 and higher we can delete those files. If not it's the first configfile and we keep it.
    if [ $cnt -ge 2 ]; then
      # Identify uuid
      uuid="$(echo "${configfile}" | sed "s/.*\([a-z0-9]\{8\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{12\}\)\.config$/\1/")"
      # Identify factory path
      factoryDir="$(echo "${configfile}" | sed "s/\(.*\)\/\([a-z0-9]\{8\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{4\}-[a-z0-9]\{12\}\)\.config$/\1/")"

      # Delete the duplicate
      echo "${dryrun}Deleting ($timestamp): ${configfile}"
      if [ "$1" = "-x" ]; then
        rm -f "${configfile}"
      fi

      # Remove the duplicate configuration from factory.config
      echo "${dryrun}Cleaning ${uuid} from ${factoryDir}/factory.config"
      if [ ! -f "${factoryDir}/factory.config" ]; then
        echo "Error: ${factoryDir}/factory.config not found. Can't clean ${uuid}."
      fi
      if [ "$1" = "-x" ]; then
        # Search for the the classname.uuid and replace it with nothing.
        # If the cleanup was between two other configs we have to add a comma between two (now) empty quotes.
        sed -i -e "s/[,]\?\"[a-zA-Z0-9\.]\+${uuid}\"[,]\?//g" -e "s/\"\"/\",\"/g" "${factoryDir}/factory.config"
      fi

    else
      echo "${dryrun}Keeping  ($timestamp): ${configfile}"
    fi
    # Increase the counter
    ((cnt++))
  done
  # Separate the duplicate files in our shell output with an empty space
  echo ""
done

# Delete the temporary file
rm -f "${file}"

After running this script it has removed 24 more duplicate osgi configuration. For removing remaining duplicate osgi configuration in have to follow manual clean up approach as described below:-

Navigate to http://localhost:4502/system/console/status-Configurations. You will see list of all OSGI factory configuration available.

check-osgi-conf

Download it as text and compare using and text editor.

Most times there is 1 configuration entry being loaded from ‘jcrinstall:’ or ‘launchpad:’ and the other entries exist just in the OSGI configuration without a match in repository or bundles.

Below is the list of duplicate configs from the local Author (the number before each line is the number of duplicates (min 2, sometimes more)). 
See details in attached author-duplicate-factory-configs.txt. You can also run below command to check which factory config is repeated how many number of time.
Note:- Running shell script and below command is completely optional, you can remove all these via manual deletion also.

$ fgrep -A1 'Found ' author-duplicate-factory-configs.txt | grep -v 'Found' | grep -v -- '--' | perl -pe 's/\.[0-9a-z-]+$//' | sort | uniq -c

1 com.adobe.cq.hc.ContentPackagesHealthCheck
1 com.adobe.cq.social.datastore.as.impl.ASResourceProviderFactory
1 com.adobe.cq.social.datastore.op.impl.SocialMSResourceProviderFactory
1 com.adobe.cq.social.datastore.rdb.impl.SocialRDBResourceProviderFactory
2 com.adobe.cq.social.messaging.client.endpoints.impl.MessagingOperationsServiceImpl
109 org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.amended

Note:- In order to minimize number of duplicated osgi configurations after upgrade. Make sure to remove duplicate configurations from /apps/system/config.

Note:- As per AEM 6.5 SP2 release notes https://helpx.adobe.com/experience-manager/6-5/release-notes/sp-release-notes.html . This issue is fixed with Hotfix for CQ-4274016 but i have tried with SP2 also and facing the same.

You can also track this issue on sling jira below:-https://issues.apache.org/jira/browse/SLING-6313

Spread the love

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.