OC4J_SECURITY will not start

I'm always a bit nervous when restarting the Oracle Application server. When it works - all is good, but when there is a problem - identifing whats wrong and then resolving seems to take forever.

We have a forms application that runs on 10giAS. Starting up the mid tier gave the following error:

oracle@oraapp> $ORACLE_HOME/opmn/bin/opmnctl startall
opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=oraapp:6201
5 of 8 processes started.

ias-instance id=mid.oraapp
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
OC4J/home/default_island

Error
--> Process (pid=0)
oid dependency failed
OID
failed to start a managed process because a dependency check failed
Log:
none

--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_BI_Forms/default_island

Error
--> Process (pid=0)
oid dependency failed
OID
failed to start a managed process because a dependency check failed
Log:
none

--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_Portal/default_island

Error
--> Process (pid=0)
oid dependency failed
OID
failed to start a managed process because a dependency check failed
Log:
none

The reason is that the Infrastructure tier was not started and it has a dependencing on OID starting. So I ensure that the database is up and the listener is up and start up the infrastructure layer to receive the following:

oracle@oraapp> $ORACLE_HOME/opmn/bin/opmnctl startall
opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=oraapp:6200
0 of 1 processes started.

ias-instance id=infr.oraapp
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
OC4J/OC4J_SECURITY/default_island

Error
--> Process (pid=19176)
time out while waiting for a managed process to start
Log:
/u01/oracle/product/10gAS/infr/opmn/logs/OC4J~OC4J_SECURITY~default_island~1

Once again not a lot of info and not much in the logs. I found the solution on Metalink DocId 331969.1

In a nutshell, the infrastructure layer had crashed out, and previous state information was causing problems on restart.

1. Ensure all processes are shut down
2. cd $ORACLE_HOME/opmn/logs
3. mv states states.old
4. mkdir states

Restart the infrastructure layer. It quickly returned a failed state so I issued the start command again and it started up.

Start the mid tier and all is fine.