ORA-00376: file 2 cannot be read at this time

I didn't know to write anything in my blog, So I used Tittle, that's "ORA-00376".

Actually, Enterprise Manager failed on Test Server(11G), I tried to find the problem and recreated EM.

I found something wrong in oc4j config DIR.

>>>
$ ls -ltr /oracle/11g/oc4j/j2ee/OC4J_DBConsole/config/

-rw-r--r-- 1 oracle oinstall 2845 Oct 31 15:08 system-application.xml
-rw-r--r-- 1 oracle oinstall 0 Apr 8 15:45 server.xml
>>>

Why server.xml file is no size. I didn't sure about the problem.
So, I think I should recreate EM.

>>>
$ emca -config dbcontrol db
.
.
.
Apr 8, 2008 4:55:49 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_16_55_19.log.
Apr 8, 2008 4:55:50 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Apr 8, 2008 4:55:52 PM oracle.sysman.emcp.EMReposConfig invoke
SEVERE: Failed to unlock all EM-related accounts
Apr 8, 2008 4:55:52 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Failed to unlock all EM-related accounts
Refer to the log file at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_16_55_19.log for more details.
Could not complete the configuration. Refer to the log file at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_16_55_19.log for more details.
>>>

Perhaps something wrong on DB
Checked alert file (rdbms).

>>>
Tue Apr 08 16:59:46 2008
Errors in file /oracle/11g/diag/rdbms/testdb/testdb/trace/testdb_smon_2396.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: '+DATA/testdb/datafile/sysaux.1087.651074273'
>>>

After that, recovery that datafile.

>>>
SQL> select FILE#, status from v$datafile;

FILE# STATUS
---------- -------
1 SYSTEM
2 RECOVER
3 ONLINE
4 ONLINE
5 ONLINE
6 ONLINE

SQL> select * from v$recover_file;

FILE# ONLINE ONLINE_ ERROR CHANGE# TIME
---------- ------- ------- ----------------------------------------------------------------- ---------- ---------
2 OFFLINE OFFLINE 2861632 07-APR-08

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

$ /oracle/11g/bin/rman target /

Recovery Manager: Release 11.1.0.6.0 - Production on Tue Apr 8 17:01:49 2008

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: TESTDB (DBID=2434402578)

RMAN> recover datafile 2;

Starting recover at 08-APR-08
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=14 device type=DISK

starting media recovery

archived log for thread 1 with sequence 64 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_64.1054.651456629
archived log for thread 1 with sequence 65 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_65.1055.651462047
archived log for thread 1 with sequence 66 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_66.1056.651468027
archived log for thread 1 with sequence 67 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_67.1057.651473947
archived log for thread 1 with sequence 68 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_68.1058.651475295
archived log for thread 1 with sequence 69 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_69.1059.651475889
archived log for thread 1 with sequence 70 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_70.1060.651475903
archived log for thread 1 with sequence 71 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_71.1061.651475913
archived log for thread 1 with sequence 72 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_72.1063.651477283
archived log for thread 1 with sequence 73 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_73.1064.651510549
archived log for thread 1 with sequence 74 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_74.1065.651510651
archived log for thread 1 with sequence 75 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_75.1066.651510735
archived log for thread 1 with sequence 76 is already on disk as file +DATA/testdb/archivelog/2008_04_08/thread_1_seq_76.1067.651516283
archived log file name=+DATA/testdb/archivelog/2008_04_08/thread_1_seq_64.1054.651456629 thread=1 sequence=64
media recovery complete, elapsed time: 00:00:00
Finished recover at 08-APR-08

RMAN> sql 'alter database datafile 2 online';

sql statement: alter database datafile 2 online

RMAN> exit

Recovery Manager complete.

SQL> select * from v$recover_file;

no rows selected
>>>

Try to create again

>>>
$ emca -config dbcontrol db

STARTED EMCA at Apr 8, 2008 5:06:05 PM
EM Configuration Assistant, Version 11.1.0.5.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Database SID: testdb
Database Control is already configured for the database testdb
You have chosen to configure Database Control for managing the database testdb
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: y
Listener port number: 1521
Password for SYSMAN user: Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
ASM ORACLE_HOME [ /oracle/11g ]:
ASM SID [ +ASM ]:
ASM port [ 1521 ]:
ASM user role [ SYSDBA ]:
ASM username [ SYS ]:
ASM user password: -----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/11g

Local hostname ................ test01
Listener port number ................ 1521
Database SID ................ testdb
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
ASM ORACLE_HOME ................ /oracle/11g
ASM SID ................ +ASM
ASM port ................ 1521
ASM user role ................ SYSDBA
ASM username ................ SYS

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
Apr 8, 2008 5:06:37 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_17_06_05.log.
Apr 8, 2008 5:06:37 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Apr 8, 2008 5:06:40 PM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
Apr 8, 2008 5:08:07 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Apr 8, 2008 5:08:10 PM oracle.sysman.emcp.util.DBControlUtil configureSoftwareLib
INFO: Software library is already configured.
Apr 8, 2008 5:08:10 PM oracle.sysman.emcp.util.DBControlUtil configureSoftwareLib
INFO: EM_SWLIB_STAGE_LOC (value) will be ignored.
Apr 8, 2008 5:08:10 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Apr 8, 2008 5:08:12 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /oracle/11g/bin/emctl config emkey -repos
Apr 8, 2008 5:08:12 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
WARNING: Configuring EM-Key failed.
Apr 8, 2008 5:08:12 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
WARNING: Error securing Database control.
Apr 8, 2008 5:08:12 PM oracle.sysman.emcp.EMDBPostConfig setWarnMsg
INFO: Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):

1) Set the environment variable ORACLE_SID to testdb
2) /oracle/11g/bin/emctl stop dbconsole
3) /oracle/11g/bin/emctl config emkey -repos -sysman_pwd <>
4) /oracle/11g/bin/emctl secure dbconsole -sysman_pwd <>
5) /oracle/11g/bin/emctl start dbconsole

To secure Em Key, run /oracle/11g/bin/emctl config emkey -remove_from_repos -sysman_pwd <>
Apr 8, 2008 5:08:12 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Apr 8, 2008 5:08:13 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /oracle/11g/bin/emctl start dbconsole
Apr 8, 2008 5:08:13 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_17_06_05.log for more details.
Could not complete the configuration. Refer to the log file at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_17_06_05.log for more details.
$ /oracle/11g/bin/emctl stop dbconsole
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
OC4J Configuration issue. /oracle/11g/oc4j/j2ee/OC4J_DBConsole_Error occurred during initialization of VM
java.lang.Error: Properties init: Could not determine current working directory._testdb not found.
$ /oracle/11g/bin/emctl config emkey -repos -sysman_pwd pass
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
EM Configuration issue. /oracle/11g/Error occurred during initialization of VM
java.lang.Error: Properties init: Could not determine current working directory._testdb not found.

$ /oracle/11g/bin/emctl secure dbconsole -sysman_pwd pass

Oracle Enterprise Manager 11g Database Control Release 11.1.0.6.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
http://test01:1158/em/console/aboutApplication
DBCONSOLE already stopped... Done.
Agent is already stopped... Done.
Securing dbconsole... Started.
Securing dbconsole... Failed.
EMKey is misconfigured.
>>>

Mission failed, So try drop RESOS and then create again: Yo!


>>>
$ emca -repos drop

STARTED EMCA at Apr 8, 2008 5:27:30 PM
EM Configuration Assistant, Version 11.1.0.5.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Apr 8, 2008 5:27:30 PM oracle.sysman.emcp.util.GeneralUtil initSQLEngine
SEVERE: No SID specified
Database SID: testdb
Listener port number: 1521
Password for SYSMAN user:
Do you wish to continue? [yes(Y)/no(N)]: y
Apr 8, 2008 5:27:45 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_17_27_30.log.
Apr 8, 2008 5:27:45 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...

Apr 8, 2008 5:31:09 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 8, 2008 5:31:09 PM

$ emca -repos create

STARTED EMCA at Apr 8, 2008 5:31:55 PM
EM Configuration Assistant, Version 11.1.0.5.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Apr 8, 2008 5:31:55 PM oracle.sysman.emcp.util.GeneralUtil initSQLEngine
SEVERE: No SID specified
Database SID: testdb
Listener port number: 1521
Password for SYSMAN user:
Do you wish to continue? [yes(Y)/no(N)]: y
Apr 8, 2008 5:32:09 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/11g/cfgtoollogs/emca/testdb/emca
Apr 8, 2008 5:32:09 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...

Apr 8, 2008 5:42:13 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 8, 2008 5:42:14 PM

$ emca -config dbcontrol db

STARTED EMCA at Apr 8, 2008 5:43:25 PM
EM Configuration Assistant, Version 11.1.0.5.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Database SID: testdb
Database Control is already configured for the database testdb
You have chosen to configure Database Control for managing the database testdb
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: y
Listener port number: 1521
Password for SYSMAN user: Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
ASM ORACLE_HOME [ /oracle/11g ]:
ASM SID [ +ASM ]:
ASM port [ 1521 ]:
ASM user role [ SYSDBA ]:
ASM username [ SYS ]:
ASM user password: -----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/11g

Local hostname ................ test01
Listener port number ................ 1521
Database SID ................ testdb
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
ASM ORACLE_HOME ................ /oracle/11g
ASM SID ................ +ASM
ASM port ................ 1521
ASM user role ................ SYSDBA
ASM username ................ SYS

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
Apr 8, 2008 5:43:51 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/11g/cfgtoollogs/emca/testdb/emca_2008_04_08_17_43_25.log.
Apr 8, 2008 5:43:51 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Apr 8, 2008 5:43:56 PM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
Apr 8, 2008 5:45:00 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Apr 8, 2008 5:45:04 PM oracle.sysman.emcp.util.DBControlUtil configureSoftwareLib
INFO: Software library configured successfully.
Apr 8, 2008 5:45:04 PM oracle.sysman.emcp.EMDBPostConfig configureSoftwareLibrary
INFO: Deploying Provisioning archives ...
Apr 8, 2008 5:45:15 PM oracle.sysman.emcp.EMDBPostConfig configureSoftwareLibrary
INFO: Provisioning archives deployed successfully.
Apr 8, 2008 5:45:15 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Apr 8, 2008 5:45:26 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Database Control secured successfully.
Apr 8, 2008 5:45:26 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Apr 8, 2008 5:45:58 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Apr 8, 2008 5:45:58 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is https://test01:1158/em <<<<<<<<<<<
Apr 8, 2008 5:46:02 PM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING:
************************ WARNING ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted. The encryption key has been placed in the file: /oracle/11g/timeten01_testdb/sysman/config/emkey.ora. Please ensure this file is backed up as the encrypted data will become unusable if this file is lost.

***********************************************************
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 8, 2008 5:46:02 PM

>>>

Wow! done.


Enjoy!