Oracle® WebLogic managed server reaches ADMIN state during startup due to name collision

The problem

When starting a Oracle® WebLogic managed server it halts when reaching the ADMIN state. 

Looking at the managed server's out file I saw the following error:

<Mar 15, 2016 8:45:40 AM UTC> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application "MyDataSource".

weblogic.application.ModuleException: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289        [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 [6058417782957188722S:my-server:soa_domain:soa_server1/288])/288        [weblogic.jdbc.common.internal.RemoteDataSource])

        at weblogic.jdbc.module.JDBCModule.activate(JDBCModule.java:411)

        at weblogic.application.internal.flow.ModuleListenerInvoker.activate(ModuleListenerInvoker.java:114)

        at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:195)

        at weblogic.application.internal.flow.ModuleStateDriver$2.next(ModuleStateDriver.java:190)

        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)

        Truncated. see log file for complete stacktrace

Caused By: weblogic.common.ResourceException: Failed to bind remote object (ClusterableRemoteRef(6058417782957188722S:my-server:soa_domain:soa_server1 null)/289   [weblogic.jdbc.common.internal.RemoteDataSource]) to replica aware stub at MY_DS(ClusterableRemoteRef(6058417782957188722S:my-server:

The cause

The error above says I have a problem starting the MyDataSource data source due to a name collision. Logging into the WebLogic console, and looking at my data sources, I realized I had two data sources with the same JNDI location. The reason was that I had prepared a new data source long time ago, and forgotten all about it. During the startup of the managed server, my new data source had started up, but the old one could of course not start since it had the same JNDI  location.

The solution

1. Removed the old and obsolete data source with the conflicting JNDI location.
2. Checked the managed server's out file for other errors, and there were none. 
3. Resumed startup of the managed server so the server state changed from ADMIN to RUNNING. 

Oracle® 12c Database Configuration Assistant hangs on Oracle® Linux 7 in VirtualBox 5

I recently installed Oracle® Linux 7.2 on my VirtualBox 5.0.14 installation on Windows 7 in order to have a private playground for evaluating software. Afterwards I installed Oracle® 12c database version 12.1.0.2 on my new virtual Linux server. This went fine until I fired up Database Configuration Assistant (DBCA) in order to create a playground database. DBCA just hanged forever without consuming noticeable CPU. I kept it running for an entire night, but it never started fully up. It kept hanging at the splash screen as shown below:

Investigations

The log file $ORACLE_BASE/cfgtoollogs/dbca/trace.log_OraDB12Home1_<timestamp> showed the following:

[main] [ 2016-02-04 10:06:33.737 CET ] [Host.<init>:1090]  Begin tracing..

[main] [ 2016-02-04 10:06:34.971 CET ] [UIHost.createHelpSet:485]  HelpSetParseExceptionjava.lang.NullPointerException

[main] [ 2016-02-04 10:06:36.703 CET ] [Host.checkIfBigClusterAndHubNode:1710]  Not a cluster environment: exiting BigCluster Check

[main] [ 2016-02-04 10:06:36.708 CET ] [InventoryUtil.getOUIInvSession:349]  setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS

[main] [ 2016-02-04 10:06:36.708 CET ] [InventoryUtil.isCRSHome:386]  Homeinfo /u01/app/oracle/product/12.1.0/dbhome_1,1

[main] [ 2016-02-04 10:06:36.922 CET ] [Host.validateGridHome:3878]  Validation false

[main] [ 2016-02-04 10:06:36.922 CET ] [Host.startOperation:2395]  Source db null

[main] [ 2016-02-04 10:06:36.922 CET ] [Host.startOperation:2396]  GDB Name null

[main] [ 2016-02-04 10:06:36.957 CET ] [Host.startOperation:2397]  MgmtDB sid -MGMTDB

[main] [ 2016-02-04 10:06:36.957 CET ] [Host.startOperation:2398]  MgmtDB name _mgmtdb

[main] [ 2016-02-04 10:06:36.998 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: null

[main] [ 2016-02-04 10:06:37.000 CET ] [InventoryUtil.getOUIInvSession:349]  setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS

[main] [ 2016-02-04 10:06:37.000 CET ] [OracleHome.getVersion:1010]  Homeinfo /u01/app/oracle/product/12.1.0/dbhome_1,1

[main] [ 2016-02-04 10:06:37.181 CET ] [OracleHome.getVersion:1038]  OracleHome.server.getVersion Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.182 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.182 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.182 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.183 CET ] [CommonUtils.createPasswordFile:1243]  calling new orapwd for 11.1 or higher

[main] [ 2016-02-04 10:06:37.183 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.184 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.184 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.184 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.184 CET ] [CommonUtils.getPasswordFileCreateCmd:1182]  for new orapwd for 11.1 or higher

[main] [ 2016-02-04 10:06:37.189 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.190 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.190 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.190 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.191 CET ] [CommonUtils.getPasswordFileCreateCmd:1213]  /u01/app/oracle/product/12.1.0/dbhome_1/bin/orapwd

[main] [ 2016-02-04 10:06:37.191 CET ] [CommonUtils.getPasswordFileCreateCmd:1213]  file=/u01/app/oracle/product/12.1.0/dbhome_1/dbs/orapwDBUA0636958

[main] [ 2016-02-04 10:06:37.191 CET ] [CommonUtils.getPasswordFileCreateCmd:1213]  force=y

[main] [ 2016-02-04 10:06:37.191 CET ] [CommonUtils.getPasswordFileCreateCmd:1213]  format=12

[main] [ 2016-02-04 10:06:37.192 CET ] [OsUtilsBase.execProg:2123]  beginning execProg with input array.

[main] [ 2016-02-04 10:06:37.377 CET ] [OsUtilsBase.execProg:2160]  finished execProg with input array. Status:0

[main] [ 2016-02-04 10:06:37.378 CET ] [OracleHome.initOptionsStopOnError:1356]  Initializing Database Options with  for dummy sid=DBUA0636958 using initfile=/u01/app/oracle/product/12.1.0/dbhome_1/dbs/initDBUA0636958.ora using pwdfile=/u01/app/oracle/product/12.1.0/dbhome_1/dbs/orapwDBUA0636958

[main] [ 2016-02-04 10:06:37.398 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.399 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.399 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.400 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.400 CET ] [OracleHome.getVersion:991]  OracleHome.getVersion called.  Current Version: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.401 CET ] [OracleHome.getVersion:1059]  Current Version From Inventory: 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.402 CET ] [SQLPlusEngine.getCmmdParams:225]  m_home 12.1.0.2.0

[main] [ 2016-02-04 10:06:37.402 CET ] [SQLPlusEngine.getCmmdParams:226]  version > 112 true

[main] [ 2016-02-04 10:06:37.403 CET ] [SQLEngine.getEnvParams:612]  NLS_LANG: AMERICAN_AMERICA.AL32UTF8

[main] [ 2016-02-04 10:06:37.421 CET ] [SQLEngine.initialize:358]  Execing SQLPLUS/SVRMGR process...

[main] [ 2016-02-04 10:06:37.467 CET ] [SQLEngine.initialize:395]  m_bReaderStarted: false

[main] [ 2016-02-04 10:06:37.467 CET ] [SQLEngine.initialize:399]  Starting Reader Thread...

[main] [ 2016-02-04 10:06:42.068 CET ] [OracleHome.initOptionsStopOnError:1370]  executing: startup nomount pfile='/u01/app/oracle/product/12.1.0/dbhome_1/dbs/initDBUA0636958.ora'

The 'ps -ef' command also confirms that sqlplus is stuck in "startup nomount"

Some unsuccessful attempts

- Tried with CentOS 7.1 instead of Oracle® Linux 7.2. Replicated the issue 100%.

- Tried to run DBCA as part of the Oracle® 12c Database installation (Software and starter database) and after a software only install. Same problem in both cases. 

Searching for an answer

I found a similar issue described in https://community.oracle.com/message/13249304#13249304, but no solution was offered, 

The solution

After various unsuccessful attempts to fix this at Linux level, I turned my attention to VirtualBox. After trying to tweak several options, I found the culprit to be the Paravirtualization Interface for acceleration. Changing that from Default to None fixed the issue.

I did not try all the various options for Paravirtualization Interface, but in my setup both with Default and Legacy I faced issues with DBCA.

ORA-01017: invalid username/password; logon denied when updating WebLogic data source and you know the username and password are correct

Background

The title might sound a bit confusing, because everyone that has experienced an ORA-01017 knows it only happens if either the database username or password is incorrect.

I recently had to change the password for all of the infrastructure component schemas in a Oracle® SOA Suite 12.1.3 installation. How to do it is documented at https://docs.oracle.com/middleware/1213/core/ASADM/repos.htm#ASADM637

After running the modifyBootStrapCredential for the OPSS configuration I expected everything to be dandy except a few ORA-01017 errors for the data sources I had not yet updated with the new password. Everything I needed to do was to edit each data source and provide the new password. Or so I thought...

The problem

When I tried to apply the changes I got the following ORA-01017 error for each data source I tried to edit:

I was a bit thunderstruck at first because the username and passwords worked fine in both SQL*Plus and in SQL Developer.

The AdminServer out file contained errors like this:
<Jan 19, 2016 1:15:54 PM UTC> <Error> <Deployer> <BEA-149205> <Failed to initialize the application "LocalSvcTblDataSource" due to error weblogic.application.ModuleException: weblogic.common.resourcepool.ResourceSystemException:
 Could not create connection for datasource '<data source name>'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in
 the configuration or that the database is not available.
weblogic.application.ModuleException: weblogic.common.resourcepool.ResourceSystemException:
 Could not create connection for datasource '<data source name>'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in
 the configuration or that the database is not available.
        at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:350)
        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)
        at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:175)
        at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:170)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.common.resourcepool.ResourceSystemException:
 Could not create connection for datasource '<data source name>'.

 The returned message is: ORA-01017: invalid username/password; logon denied

 It is likely that the login or password is not valid.
 It is also possible that something else is invalid in
 the configuration or that the database is not available.
        at weblogic.jdbc.common.internal.JDBCUtil.parseException(JDBCUtil.java:322)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection0(ConnectionEnvFactory.java:492)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory.access$000(ConnectionEnvFactory.java:20)
        at weblogic.jdbc.common.internal.ConnectionEnvFactory$1.run(ConnectionEnvFactory.java:388)
        at java.security.AccessController.doPrivileged(Native Method)
        Truncated. see log file for complete stacktrace
>

The workaround

For each data source I had to do the following to work around this problem:

1. Clicked the Lock & Edit button.
2. Had to note which targets the data source was using.
3. Removed the targets from the data source, and clicked on the Save button.
4. Clicked on the Activate Changes button.
5. Clicked the Lock & Edit button.
6. Updated the password for the data source, and clicked on the Save button.
7. Clicked on the Activate Changes button.
8. Clicked the Lock & Edit button.
9. Added the targets that I noted in step 2 to the data source, before I clicked on the Save button.
10. Clicked the Activate Changes button.

After all data sources had been updated with the new passwords, and the data sources had been re-targeted, I did a full domain restart. 

While I started the AdminServer up, I looked out for errors like these which could indicate I had forgotten to update some data sources:

<Jan 19, 2016 1:32:34 PM UTC> <Error> <Deployer> <BEA-149205> <Failed to initialize the application "<data source name>" due to error weblogic.application.ModuleException: weblogic.common.resourcepool.ResourceSystemException

How to troubleshoot and fix "ORA-01882 timezone region not found" while creating WebLogic data source

Symptoms

While creating a data source in WebLogic you get the error "ORA-01882 timezone region not found" while saving the new configuration. 

Troubleshooting

This troubleshooting was performed on a Red Hat Linux 6.6 server running WebLogic 12.1.3

Finding the timezone used by the JVM

Even if the user account running WebLogic does not have a TZ environment variable set, the JVM will pickup a timezone as specified by the sysconfig clock on Linux. By running $JAVA_HOME/bin/jvisualvm you will be able to find out which timezone it is using by looking for the user.timezone property in System properties:

In my case this was originally set to Etc/UTC. 

Which timezones are supported by the Oracle database

Log on to the database to which you attempted to create a data source, and then run the following query:

SELECT * FROM V$TIMEZONE_NAMES ORDER BY 1;

In my case there was no such thing as TZNAME = Etc/UTC. But of course just UTC was a valid value. 

Force WebLogic to use UTC timezone

Edit $DOMAIN_HOME/bin/startWebLogic.sh and go to the bottom of the file where the commands to startup WebLogic reside. After each of the three ${MEM_ARGS} add -Dtimezone=UTC. After your change the section will look like this:

echo "starting weblogic with Java version:"

${JAVA_HOME}/bin/java ${JAVA_VM} -version


if [ "${WLS_REDIRECT_LOG}" = "" ] ; then

echo "Starting WLS with line:"

echo "${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Duser.timezone=UTC -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}"

${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Duser.timezone=UTC -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}

else

echo "Redirecting output from WLS window to ${WLS_REDIRECT_LOG}"

${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Duser.timezone=UTC -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WLS_POLICY_FILE} ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS} >"${WLS_REDIRECT_LOG}" 2>&1
fi

Finally restart WebLogic.

Verify that the JVM is using the new timezone

The quick way to check it is of course "ps -ef | grep timezone" and make sure there is one hit per managed server plus the AdminServer. 

Of you can check with JVisualVM as explained above.  

Finding grants from data dictionary

Unless you make an full export with grants you won’t get all grants when doing a export/import with Oracle. So if you don’t have a list of all grants across those schemas you imported, you will have a huge job of compiling and establishing which grants are missing.

The solution to this is to find those grants through the data dictionary on the source system. For instance, if you want to find all grants made by the PORTAL schema, the query would be like this:


SET HEADING OFF
SET PAGES 999
SPOOL grants.sql
SELECT 'GRANT '||privilege||' ON '||owner||'.'||table_name||' TO '||grantee||DECODE(grantable,'YES',' WITH GRANT OPTION;',';')
FROM DBA_TAB_PRIVS
WHERE grantor = 'PORTAL';
SPOOL OFF
EXIT


Now you have everything you need in the grants.sql and this script can now be run on the target system.

Oracle Q-Quote

To avoid quoting quotes in string, Oracle 10g offers the Q-Quote technique.

Let’s say yoy want to select the following from dual: I’m into rock’n’roll

This would normally mean you should quote the three quotes, but with the Q-Quote technique it’s much simpler:

SQL> SELECT q'[I'm into rock'n'roll]' FROM dual;

Schedule a job using DBMS_SCHEDULER

In this article I show how you can setup automatic rebuild of Portal indexes every day at 04.00 am.

Create your PL/SQL procedure

sqlplus portal/<password>

CREATE OR REPLACE PROCEDURE my_analyze_portal IS
BEGIN
wwsbr_stats.delete_stats;
wwsbr_stats.gather_stats;
wwsbr_stats.enable_monitoring;
wwsbr_stats.gather_stale;
commit;
END;
/

Submit the job

sqlplus portal/<password>

BEGIN
	DBMS_SCHEDULER.create_job (
    job_name        => 'job_my_analyze_portal',
    job_type        => 'STORED_PROCEDURE',
    job_action      => 'my_analyze_portal',
    start_date      => SYSTIMESTAMP,
    repeat_interval => 'freq=daily; byhour=4; byminute=0; bysecond=0',
    end_date        => NULL,
    enabled         => TRUE,
    comments        => 'Refresh Portal indexes.');
END;
/

COMMIT;

EXIT

View the job

You can view the current status of this job at any time.

SELECT job_name, enabled, last_start_date,next_run_date,comments
FROM user_scheduler_jobs
WHERE job_action='my_analyze_portal';

Put DBMS_SCHEDULER to the test

Imagine you have a job that should run every third minute, monday to friday and sunday, but only between 18.00 and midnight. How would you do that with DBMS_JOB ? The short answer is: You wouldn’t! The longer answer would be to have a chain of jobs enabling, disabling and redefining eachothers.

With DBMS_SCHEDULER however, this is just a walk in the park, and can be achieved with one single statement in a block:

BEGIN
dbms_scheduler.create_schedule (
schedule_name => ‘WOW_SCHEDULE’,
repeat_interval => ‘FREQ=DAILY; BYDAY=1,2,3,5,7; BYHOUR=18,19,20,21,22,23; BYMINUTE=0,3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48,51,54,57; BYSECOND=0′,
comments => ‘Every third minut between 18.00 and midnight monday to friday and sundays’);
END;
/