Audun Nes' tech blog: WebLogic

Showing posts with label WebLogic. Show all posts

Wednesday, May 11, 2016

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage

The problem

We discovered that an Oracle® WebLogic managed server was in status FAILED on a development environment. The apparent reason for this was the following error:

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a storage

The cause

The can be multiple reasons or the above error, but the logical guess was that our /data mount point had run out of disk space, because that mount point was where our domain configuration (and persistent stores) were saved. That theory turned out to be wrong. The /data mount point was only 12% utilized on a 100GB partition, so there was plenty of available disk.

Then I tried to create an empty file on the mount point:

# touch /data/dummy

# touch: cannot touch `/data/dummy': Read-only file system

When checking the /var/log/messages file, I found the reason for that:

May 11 07:37:45 <hostname> kernel: JBD2: I/O error detected when updating journal superblock for dm-25-8.

May 11 07:37:45 <hostname> kernel: EXT4-fs error (device dm-25): ext4_journal_start_sb: Detected aborted journal

May 11 07:37:45 <hostname> kernel: EXT4-fs (dm-25): Remounting filesystem read-only

Comparing the commands cat /etc/fstab and ls -l /dev/mapper/ | grep dm-25 also confirmed that device dm-25 was indeed the source for the /data mount point.

The solution

After shutting down everything as cleanly as I could, I checked if there were still processes trying to use the /data mount point:

lsof | grep "/data"

It turned out that there was a few processes I was unable to shutdown cleanly, so I had to kill those.

Then I unmounted the /data mount point:

umount /data

Finally I fixed the issue by running fsck and mounting the partition again.

Caution: Running fsck on a partition can in worst case scenario be destructive, so make sure you have a valid backup!

fsck /data

mount /data

Now I could start up Oracle® WebLogic Server without any issues.

Thursday, April 28, 2016

RDA-10913: Unknown profile "FM12c_SoaMin" when running Remote Diagnostic Agent on Oracle® SOA Suite 12.1.3

The problem

You are trying to collect information about the Oracle® SOA Suite 12.1.3 using the Remote Diagnostic Agent (RDA), but it fails with a RDA-10913: Unknown profile "FM12c_SoaMin" error, like shown below:

source $DOMAIN_HOME/bin/setDomainEnv.sh

cd $SOA_ORACLE_HOME

cd ../oracle_common/rda

./rda.sh -s soa_issue -p FM12c_SoaMin

RDA-00013: Error reported by the command "setup":

RDA-00014: Request error "Setup":

RDA-10913: Unknown profile "FM12c_SoaMin"

The cause

There is something wrong with the RDA supplied with Oracle® SOA Suite 12.1.3, and this is how I discovered that:

The command ./rda.sh -V told me I was running RDA 8.02, which looked fine.

By reading the full output from ./rda.sh -V I could also see that the required modules where installed, according to Oracle Support Doc ID 391983.1.

Using ./rda.sh -h told me which options RDA is accepting, And in particular I noted the -L flag:

-L List the available modules, profiles, and conversion groups

./rda.sh -L | grep Soa

FM11g_Soa Oracle SOA Suite 11g problems

FM11g_SoaMin Oracle SOA Suite 11g problems (minimum)

As you can see, the RDA shipped with Oracle® SOA Suite 12.1.3 was only supporting Oracle® SOA Suite 11g diagnostics.

The solution

Downloaded the latest version of the RDA tool for Oracle® Fusion Middleware. At the time of writing this, the patch number was 22783073.
Used opatch apply to install the patch I just downloaded.
Verified which SOA profiles were now available by running:
./rda.sh -L | grep Soa
OFM_SoaMax OFM SOA Suite problems (11g/12c+) : full
OFM_SoaMin OFM SOA Suite problems (11g/12c+) : minimum
Ran the RDA with the new module for Oracle® SOA Suite 11g and 12c:
./rda.sh -s soa_issue -p OFM_SoaMin

Monday, March 14, 2016

Oracle® WebLogic managed server reaches ADMIN state during startup due to name collision

The problem

When starting a Oracle® WebLogic managed server it halts when reaching the ADMIN state.

Looking at the managed server's out file I saw the following error:

<Mar 15, 2016 8:45:40 AM UTC> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application "MyDataSource".

weblogic.application.ModuleException: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289 [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 [6058417782957188722S:my-server:soa_domain:soa_server1/288])/288 [weblogic.jdbc.common.internal.RemoteDataSource])

at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:411)

at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:114)

at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:195)

at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:190)

at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)

Truncated. see log file for complete stacktrace

Caused By: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289 [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:

The cause

The error above says I have a problem starting the MyDataSource data source due to a name collision. Logging into the WebLogic console, and looking at my data sources, I realized I had two data sources with the same JNDI location. The reason was that I had prepared a new data source long time ago, and forgotten all about it. During the startup of the managed server, my new data source had started up, but the old one could of course not start since it had the same JNDI location.

The solution

Removed the old and obsolete data source with the conflicting JNDI location.
Checked the managed server's out file for other errors, and there were none.
Resumed startup of the managed server so the server state changed from ADMIN to RUNNING.

Thursday, March 10, 2016

Oracle® SOA Suite 12.1.3 startup fails with CoherenceException due to ensureWKAAddresses

The problem

After a fresh install of Oracle® SOA Suite 12.1.3 with BPM and BAM enabled, both the SOA server and the BAM server failed to startup due to the following exception:

Mar 09, 2016 9:49:36 AM oracle.dms.servlet.DMSServletFilter setEagerlySetContextValues

INFO: The setting that controls the eager fetching of some types of execution context data has been set to true.

<Mar 9, 2016 9:49:37 AM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.cacheprovider.coherence.CoherenceException:

at weblogic.cacheprovider.coherence.CoherenceClusterManager.ensureWKAAddresses(CoherenceClusterManager.java:510)

at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:236)

at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)

at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)

at weblogic.cacheprovider.CacheProviderServerService.initialize(CacheProviderServerService.java:71)

at weblogic.cacheprovider.CacheProviderServerService.start(CacheProviderServerService.java:65)

at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)

at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)

at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)

at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)

at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)

at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)

at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)

at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)

at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)

at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)

at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)

at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)

at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

The cause

As stated in the exception, the issue is that the servers are listening on localhost instead of the actual hostname while the Oracle® WebLogic Server is running in Production mode.

The solution

There are multiple solutions to this one, but the two most obvious ones are to either run Oracle® WebLogic Server is running in Development mode, or to listen to the actual hostname instead of localhost. I would not recommend the former. Not unless it is a single user development system.

Run the hostname command to find the actual hostname. The hostname command works on both Linux, Unix and Windows.

Edit the managed servers through the Oracle® WebLogic Server console, and set the value returned from the hostname command as the Listen Address,

Tuesday, March 1, 2016

Could not initialize class sun.awt.X11GraphicsEnvironment for frevvo module in Oracle® BPM Suite 12.2.1

The problem

BPM Composer in Oracle® BPM Suite 12.2.1 fails when open enterprise maps, and the SOA server's out file says:

<Mar 1, 2016 7:31:43 AM UTC> <Error> <HTTP> <BEA-101017> <[ServletContext@881844748[app:frevvo module:/frevvo path:null spec-version:3.1]] Root cause of ServletException.

java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:103)

at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:82)

at sun.awt.X11FontManager.isHeadless(X11FontManager.java:511)

Truncated. see log file for complete stacktrace

The cause

The JVM was not running headless, and it did not have a DISPLAY variable set to point to a X server.

The solution

Added -Djava.awt.headless=true to the soa server, and restarted it.

Friday, February 12, 2016

Error "ORA-01882 timezone region not found" when creating new domain in Oracle® WebLogic

The problem

During domain configuration I received an error saying ORA-01882 timezone region not found when I clicked on the Get RCU Configuration button.

The cause

This is because your operating system is either using a invalid timezone value, or using the Etc/UTC or Universal timezone. The two latter examples are due to a bug when connecting to the database.

The solution

Exit the domain configuration assistant
Set a timezone value accepted by the database: TZ=UTC; export TZ
Start the domain configuration assistant again by running config.sh
To prevent the same issue from happening again (assuming you hit the bug), you should add the corrected TZ to the software owner's profile, e.g. in $HOME/.bash_profile if you are using the Bourne Again Shell. However, if your Linux server clock is configured with an invalid timezone, you should fix it at Linux level by using a IANA compliant timezone.

Thursday, February 11, 2016

OPatch failed with error code 73

The problem

I came across a sandbox server which had an interesting problem with oraInventory. When running opatch I received the following error:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Possible causes are:

Oracle Home dir. path does not exist in Central Inventory

Oracle Home is a symbolic link

Oracle Home inventory is corrupted

LsInventorySession failed: Oracle Home inventory cannot be loaded.

OPatch failed with error code 73

The cause

The immediate cause was that /u01/app/oraInventory was empty besides the "locks" directory that got created when I ran $ORACLE_HOME/OPatch/opatch lsinventory.

/etc/oraInst.loc pointed to the /u01/app/oraInventory directroy, and so did $ORACLE_HOME/oraInst.loc.

I looked for other copies of oraInventory on the server, but there were none.

I can only speculate on why /u01/app/oraInventory was empty. Perhaps someone deleted the contents by mistake, or by some misguided effort to free up disk space.

The solution

Had this been a production system it would have made sense to check the file system backup, and do a restore of /u01/app/oraInventory. But his was just a sandbox, and no file system backup existed.

The solution was to run $ORACLE_HOME/oui/bin/attachHome.sh to populate the oraInventory again.

Now the command $ORACLE_HOME/OPatch/opatch lsinventory worked fine.

Friday, February 5, 2016

Server subsystem failed. Reason: A MultiException has 10 exceptions when starting Oracle® WebLogic managed server

The problem

After starting the Oracle® WebLogic AdminServer in a sandbox environment, I attempted to start the managed servers, but got the following errors in the $DOMAIN_HOME/servers/<ms-server>/logs/<ms-server>.out log file:

<Feb 5, 2016 8:31:03 AM UTC> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 10 exceptions. They are:

1. java.lang.AssertionError: Assertion violated

2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.ldap.EmbeddedLDAP

3. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.security.PreSecurityService errors were found

4. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.security.PreSecurityService

5. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.security.SecurityService errors were found

6. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.security.SecurityService

7. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.RemoteNamingService errors were found

8. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.RemoteNamingService

9. java.lang.IllegalArgumentException: While attempting to resolve the dependencies of weblogic.jndi.internal.ForeignJNDIManagerService errors were found

10. java.lang.IllegalStateException: Unable to perform operation: resolve on weblogic.jndi.internal.ForeignJNDIManagerService

A MultiException has 10 exceptions. They are: