Audun Nes' tech blog

Monday, March 14, 2016

Oracle® WebLogic managed server reaches ADMIN state during startup due to name collision

The problem

When starting a Oracle® WebLogic managed server it halts when reaching the ADMIN state.

Looking at the managed server's out file I saw the following error:

<Mar 15, 2016 8:45:40 AM UTC> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application "MyDataSource".

weblogic.application.ModuleException: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289 [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 [6058417782957188722S:my-server:soa_domain:soa_server1/288])/288 [weblogic.jdbc.common.internal.RemoteDataSource])

at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:411)

at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:114)

at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:195)

at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:190)

at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)

Truncated. see log file for complete stacktrace

Caused By: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289 [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:

The cause

The error above says I have a problem starting the MyDataSource data source due to a name collision. Logging into the WebLogic console, and looking at my data sources, I realized I had two data sources with the same JNDI location. The reason was that I had prepared a new data source long time ago, and forgotten all about it. During the startup of the managed server, my new data source had started up, but the old one could of course not start since it had the same JNDI location.

The solution

Removed the old and obsolete data source with the conflicting JNDI location.
Checked the managed server's out file for other errors, and there were none.
Resumed startup of the managed server so the server state changed from ADMIN to RUNNING.

Thursday, March 10, 2016

Oracle® SOA Suite 12.1.3 startup fails with CoherenceException due to ensureWKAAddresses

The problem

After a fresh install of Oracle® SOA Suite 12.1.3 with BPM and BAM enabled, both the SOA server and the BAM server failed to startup due to the following exception:

Mar 09, 2016 9:49:36 AM oracle.dms.servlet.DMSServletFilter setEagerlySetContextValues

INFO: The setting that controls the eager fetching of some types of execution context data has been set to true.

<Mar 9, 2016 9:49:37 AM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.cacheprovider.coherence.CoherenceException:

at weblogic.cacheprovider.coherence.CoherenceClusterManager.ensureWKAAddresses(CoherenceClusterManager.java:510)

at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:236)

at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)

at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)

at weblogic.cacheprovider.CacheProviderServerService.initialize(CacheProviderServerService.java:71)

at weblogic.cacheprovider.CacheProviderServerService.start(CacheProviderServerService.java:65)

at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)

at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)

at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)

at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)

at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)

at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)

at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)

at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)

at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)

at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)

at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)

at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)

at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

The cause

As stated in the exception, the issue is that the servers are listening on localhost instead of the actual hostname while the Oracle® WebLogic Server is running in Production mode.

The solution

There are multiple solutions to this one, but the two most obvious ones are to either run Oracle® WebLogic Server is running in Development mode, or to listen to the actual hostname instead of localhost. I would not recommend the former. Not unless it is a single user development system.

Run the hostname command to find the actual hostname. The hostname command works on both Linux, Unix and Windows.

Edit the managed servers through the Oracle® WebLogic Server console, and set the value returned from the hostname command as the Listen Address,

Tuesday, March 1, 2016

Could not initialize class sun.awt.X11GraphicsEnvironment for frevvo module in Oracle® BPM Suite 12.2.1

The problem

BPM Composer in Oracle® BPM Suite 12.2.1 fails when open enterprise maps, and the SOA server's out file says:

<Mar 1, 2016 7:31:43 AM UTC> <Error> <HTTP> <BEA-101017> <[ServletContext@881844748[app:frevvo module:/frevvo path:null spec-version:3.1]] Root cause of ServletException.

java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:103)

at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:82)

at sun.awt.X11FontManager.isHeadless(X11FontManager.java:511)

Truncated. see log file for complete stacktrace

The cause

The JVM was not running headless, and it did not have a DISPLAY variable set to point to a X server.

The solution

Added -Djava.awt.headless=true to the soa server, and restarted it.

Friday, February 12, 2016

Error "ORA-01882 timezone region not found" when creating new domain in Oracle® WebLogic

The problem

During domain configuration I received an error saying ORA-01882 timezone region not found when I clicked on the Get RCU Configuration button.

The cause

This is because your operating system is either using a invalid timezone value, or using the Etc/UTC or Universal timezone. The two latter examples are due to a bug when connecting to the database.

The solution

Exit the domain configuration assistant
Set a timezone value accepted by the database: TZ=UTC; export TZ
Start the domain configuration assistant again by running config.sh
To prevent the same issue from happening again (assuming you hit the bug), you should add the corrected TZ to the software owner's profile, e.g. in $HOME/.bash_profile if you are using the Bourne Again Shell. However, if your Linux server clock is configured with an invalid timezone, you should fix it at Linux level by using a IANA compliant timezone.

Thursday, February 11, 2016

OPatch failed with error code 73

The problem

I came across a sandbox server which had an interesting problem with oraInventory. When running opatch I received the following error:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Possible causes are:

Oracle Home dir. path does not exist in Central Inventory

Oracle Home is a symbolic link

Oracle Home inventory is corrupted

LsInventorySession failed: Oracle Home inventory cannot be loaded.

OPatch failed with error code 73

The cause

The immediate cause was that /u01/app/oraInventory was empty besides the "locks" directory that got created when I ran $ORACLE_HOME/OPatch/opatch lsinventory.

/etc/oraInst.loc pointed to the /u01/app/oraInventory directroy, and so did $ORACLE_HOME/oraInst.loc.

I looked for other copies of oraInventory on the server, but there were none.

I can only speculate on why /u01/app/oraInventory was empty. Perhaps someone deleted the contents by mistake, or by some misguided effort to free up disk space.

The solution

Had this been a production system it would have made sense to check the file system backup, and do a restore of /u01/app/oraInventory. But his was just a sandbox, and no file system backup existed.

The solution was to run $ORACLE_HOME/oui/bin/attachHome.sh to populate the oraInventory again.

Now the command $ORACLE_HOME/OPatch/opatch lsinventory worked fine.

Friday, February 5, 2016

Server subsystem failed. Reason: A MultiException has 10 exceptions when starting Oracle® WebLogic managed server

The problem

After starting the Oracle® WebLogic AdminServer in a sandbox environment, I attempted to start the managed servers, but got the following errors in the $DOMAIN_HOME/servers/<ms-server>/logs/<ms-server>.out log file:

<Feb 5, 2016 8:31:03 AM UTC> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 10 exceptions. They are:

1. java.lang.AssertionError: Assertion violated

2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.ldap.EmbeddedLDAP

3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.security.PreSecurityService errors were found

4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.security.PreSecurityService

5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.security.SecurityService errors were found

6. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.security.SecurityService

7. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.RemoteNamingService errors were found

8. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.RemoteNamingService

9. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.ForeignJNDIManagerService errors were found

10. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.ForeignJNDIManagerService

A MultiException has 10 exceptions. They are: