JIRA Upgrades are bread and butter work for us. Over the last few months (Jan-April 2017) we've experienced problems upgrading various companies to JIRA 7.3.x. JIRA's plugin system intermittently (say, 50% of the time) fails to load:

In the JIRA logs, near the top we find:

2017-04-12 20:51:17,450 ThreadPoolAsyncTaskExecutor::Thread 6 ERROR [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.upm.plugin-license-storage-plugin'

followed by dozens of plugins complaining about missing licensing-related dependencies.

2017-04-12 20:51:17,450 ThreadPoolAsyncTaskExecutor::Thread 6 ERROR      [c.a.p.osgi.factory.OsgiPlugin] Unable to start the plugin container for plugin 'com.atlassian.upm.plugin-license-storage-plugin'
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseReceiptValidator' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [com.atlassian.upm.license.internal.PluginLicenseRepository]: : Error creating bean with name 'pluginLicenseRepository' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [com.atlassian.upm.license.internal.HostLicenseProvider]: : Error creating bean with name 'jiraHostLicenseProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 1 of type [com.atlassian.upm.license.internal.LicenseManagerProvider]: : Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jiraHostLicenseProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 1 of type [com.atlassian.upm.license.internal.LicenseManagerProvider]: : Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pluginLicenseRepository' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [com.atlassian.upm.license.internal.HostLicenseProvider]: : Error creating bean with name 'jiraHostLicenseProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 1 of type [com.atlassian.upm.license.internal.LicenseManagerProvider]: : Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jiraHostLicenseProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Unsatisfied dependency expressed through constructor argument with index 1 of type [com.atlassian.upm.license.internal.LicenseManagerProvider]: : Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'licenseManagerProvider' defined in URL [bundle://166.0:0/META-INF/spring/atlassian-plugins-components.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.upm.license.internal.impl.LicenseManagerProviderImpl]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: com.google.common.collect.ImmutableSet.of([Ljava/lang/Object;)Lcom/google/common/collect/ImmutableSet;
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1139)

What's going on?

The culprit is this jar:

$JIRAHOME/plugins/installed-plugins/plugin-license-storage-plugin-2.13.3.jar

(or plugin-license-storage-plugin-2.12.jar or other low versions).

Plugin key
com.atlassian.upm.plugin-license-storage-plugin
Plugin name

Atlassian Universal Plugin Manager - Plugin License Storage plugin

This plugin is very obsolete, and is no longer compatible with JIRA's API.

How does it get there in the first place? Say that back when we ran JIRA 6.4.9, we installed the Dynamic Forms plugin, version 1.7.11: 

The Dynamic Forms 1.7.11 depended on the Plugin License Storage jar, and bundles it in the .obr file. After installing, our installed-plugins directory would have two jars:

dynamic-forms-1.7.11.jar
plugin-license-storage-plugin-2.12.jar

Then later, we clicked 'Update' to upgrade Dynamic Forms to 1.8.9, which does not depend on 'Plugin License Storage' any more. What jars do we now have?

dynamic-forms-1.8.9.jar
plugin-license-storage-plugin-2.12.jar

Oops. The plugin upgraded, but left its old dependency there. The jar then sits like a time bomb, harmless up till the point where JIRA's API changes enough, or other plugins are installed that interact with it.

The problem of plugins leaving their obsolete dependencies is fairly well-known. I frequently see plugins called RB - Filtering,  RB - Support etc, which are obsolete dependencies. The problem with  plugin-license-storage-plugin-*.jar is that it is marked as a 'system' plugin, so never appears in the list of user-installed plugins.

What's the fix?

Obviously, if you run JIRA 7.x, check your /var/atlassian/application-data/jira/plugins/installed-plugins directory for plugin-license-storage-plugin-*.jar.

But there is a broader problem here that Atlassian need to address. Upgrading a plugin should not leave its unused dependencies lying about. When you click 'Install' in the UPM, the plugin .obr file is downloaded, and all dependent jars are just dumped into plugins/installed-plugins. The interdepdency information (from obr.xml) is lost, meaning there's no possibility later of the plugin system noticing "hey, nobody depends on this jar any more, let's delete it".

Until Atlassian fix this (ha), the only sure way to avoid dangling plugin dependencies is to delete all your plugins, then install them again.