OS Watcher

OSWatcher (oswbb) is a downloadable utility to capture performance metrics from the operating system. When you install and run oswbb as part of a performance diagnostic data collection best practice, you can aid in a quicker resolution of your SR with support and development.

Installation
1.       Download the tool from metalink
2.       Create folder for each node separately.
3.       Unzip file oswbb601.tar
4.       Set up file private.net by modifying the file ../oswbb/Exampleprivate.net
5.       Output log will be placed under ../oswbb/archive/

Starting OSWbb
To start the OSWbb utility execute the startOSWbb.sh shell script. This script has 2 arguments which control the frequency that data is collected and the number of hours of data to archive.

ARG1 = snapshot interval in seconds.
ARG2 = the number of hours of archive data to store.
If you do not enter any arguments the script runs with default values of 30 and 48 meaning collect data every 30 seconds and store the last 48 hours of data in archive files.

Example:
./startOSWbb.sh 60 10

This would start the tool and collect data at 60 second intervals and log the last 10 hours of data to archive files.

Stopping OSWbb:
To stop the OSWbb utility execute the stopOSWbb.sh command. This terminates all the processes associated with the tool.

Example:
./stopOSWbb.sh

Reference
OSWatcher Black Box (Includes: [Video]) (Doc ID 301137.1)

When patching, you have two entities in your environment
  - Oracle Database Software
  - Oracle Database

You can check the patches installed using opatch utility which you can find in $ORACLE_HOME/OPatch:

./opatch lsinv -bugs_fixed | grep -i 'database cpu'

Following patching your database with a PSU/CPU, you want to keep a history of the patches applied.
In 11g (actually this started since 10.2.0.4), there is a new post patch installation script which does the job.
It is catbundle.sql, and is located in $ORACLE_HOME/rdbms/admin. (In previous releases this was done by catcpu.sql)
This shall be invoked with 2 parameters, patch type (PSU/CPU) and action (apply or rollback).
The purpose of the script is to update the entry in registry$history.

SQL> @catbundle.sql CPU APPLY
PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

1. PSU (and CPU) patch README already contains information about when to execute post install step (i.e loading modified .sql files, ex: catbundle.sql) on databases created using DBCA.

2. PSU/CPU entry in "registry$history" gets updated by the patch post installation step (i.e catbundle.sql). Skipping this step (where it is not actually MANDATORY in above mentioned case) results no PSU/CPU entry in registry$history. Even though it is not mandatory to run PSU/CPU post install step (i.e catbundle.sql), there is no harm in running as it helps to update the registry$history.


References:
How to confirm that a Critical Patch Update (CPU) has been installed in Linux / UNIX [ID 821263.1]
Is it required to run PSU/CPU patch post installation step after creating the database using DBCA in already patched Oracle Home ? [ID 1388555.1]
Best approach for Oracle database patching sequence to latest/required patchset along with CPU/PSU/any-other-one-off patch [ID 865255.1]

error message in alert.log after dnfs is enabled for a database...

Database Alert Log entries: Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1 (Doc ID 1352886.1)    

   

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later

Information in this document applies to any platform.

Symptoms

Database alert log shows these 'Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1' entries:

alert_ORCL.log

~~~~~~~~~~~

...

Mon Aug 08 21:04:00 2011

Adjusting the default value of parameter parallel_max_servers

from 320 to 185 due to the value of parameter processes (200)

Starting ORACLE instance (normal)

...

Starting up:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

...

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

...

Mon Aug 08 21:05:19 2011

Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1        <-- here

...

Cause

tcp_max_buf: is set too small.

Solution

The error can be fixed by increasing the value of tcp_max_buf:

# /usr/sbin/ndd -set /dev/tcp tcp_max_buf 1056768

To display its value:

# /usr/sbin/ndd /dev/tcp tcp_max_buf

Also check what are tcp_xmit_hiwat and tcp_recv_hiwat set to and if set low, see if increasing these helps:

# /usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 1056768

# /usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 1056768

To display their values:

# /usr/sbin/ndd /dev/tcp tcp_xmit_hiwat

# /usr/sbin/ndd /dev/tcp tcp_recv_hiwat

Cannot start oms EM 12c

The OMS is failing to start and the problem seems to the the connectivity to the repository DB. I have observe that the connection to the DB from SQL Plus is failing. To check if the OMS is able to connect, perform the connectivity check using the rcuJDBCEngine.

1. Get the connection details by running the command
./emctl config oms -list_repos_details

2. Please refer to the following note for details on checking the connectivity from OMS to repository using rcuJDBCEngine.
12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine (Doc ID 1448007.1)

Solution

1) stop any running processes using:-

$/bin/emctl stop oms -all -force

2) Check whether there are any remaining processes:-

ps -ef | grep EMGC_ADMINSERVER
ps -ef | grep EMGC_OMS1
ps -ef | grep java

kill any of the above processes which are running from the OMS $ORACLE_HOME using 'kill -9 '

3) Remove the *.DAT and *.lok files:

$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/store/default/_WLS_EMGC_OMS1000000.DAT
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/store/default/_WLS_EMGC_ADMINSERVER000000.DAT

and

[oracle@hostname gc_inst]$ find . -name "*.lok"
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/config/config.lok
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/ldap/ldapfiles/EmbeddedLDAP.lok
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/EMGC_OMS1.lok
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/data/ldap/ldapfiles/EmbeddedLDAP.lok
$EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/tmp/EMGC_ADMINSERVER.lok

4) start the OMS

$/bin/emctl start OMS

 11g/12c OMS failed to start "Unable to obtain lock on EMGC_OMS1.lok" [ID 1302822.1]

Recreate EM 12C repository

&gt; ./emca -config dbcontrol db -repos recreate

STARTED EMCA at Dec 8, 2012 10:33:43 AM

EM Configuration Assistant, Version 11.2.0.3.0 Production

Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:

Database SID: pacloud

Listener port number: 1521

Listener ORACLE_HOME [ /opt/oracle/app/oracle/product/11.2.0/db_1 ]:

Password for SYS user:

Password for DBSNMP user:

Password for SYSMAN user:

Email address for notifications (optional): dbasupport@protectair.com

Outgoing Mail (SMTP) server for notifications (optional):

-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /opt/oracle/app/oracle/product/11.2.0/db_1

Local hostname ................ culp-dbmgmt01.pa.parsonsvic.com

Listener ORACLE_HOME ................ /opt/oracle/app/oracle/product/11.2.0/db_1

Listener port number ................ 1521

Database SID ................ pacloud

Email address for notifications ............... dbasupport@protectair.com

Outgoing Mail (SMTP) server for notifications ...............

-----------------------------------------------------------------

----------------------------------------------------------------------

WARNING : While repository is dropped the database will be put in quiesce mode.

----------------------------------------------------------------------

Do you wish to continue? [yes(Y)/no(N)]: Y

Dec 8, 2012 10:34:51 AM oracle.sysman.emcp.EMConfig perform

INFO: This operation is being logged at /opt/oracle/app/oracle/cfgtoollogs/emca/pacloud/emca_2012_01_08_10_33_43.log.

Dec 8, 2012 10:34:52 AM oracle.sysman.emcp.EMReposConfig invoke

INFO: Dropping the EM repository (this may take a while) ...

http://docs.oracle.com/cd/B28359_01/server.111/b28319/emca.htm

reCreate Dataguard Broker

Step By Step How to Recreate Dataguard Broker Configuration.
Fix

Step 1. Stop broker in both of primary and standby databases (in all instances in case of RAC):


SQL&gt;ALTER SYSTEM SET DG_BROKER_START=FALSE;
Step 2. Drop old broker configuration files from both primary and standby:

Check the location of files using : "show parameter dg_broker_config_file"
By default the broker configuration files will be in "$ORACLE_HOME/dbs" in Unix and "$ORACLE_HOME/database" in Windows.
SQL&gt; show parameter dg_broker_config_file

NAME                   TYPE   VALUE
---------------------- ------ ----------------------------------------
dg_broker_config_file1 string \oracle\product\10.2.0\dbs\dr1PROD10G.dat
dg_broker_config_file2 string \oracle\product\10.2.0\dbs\dr2PROD10G.dat
Drop the files using ASMCMD if files are on ASM:


ASMCMD&gt; cd DATA/PROD10G/
ASMCMD&gt; rm dr1PROD10G.dat
ASMCMD&gt; rm dr2PROD10G.dat
Drop the files using :

Unix


$cd $ORACLE_HOME/dbs
$ rm dr1PROD10G.dat
$ rm dr2PROD10G.dat

Windows
c:\&gt;cd $ORACLE_HOME/database
D:\oracle\product\10.2.0\db_2\database&gt; del dr1PROD10G.dat
D:\oracle\product\10.2.0\db_2\database&gt; del dr2PROD10G.dat
Step 3. Start broker in both of primary and standby (in all instances in case of RAC):


SQL&gt;ALTER SYSTEM SET DG_BROKER_START=TRUE;
Step 4. Connect to DGMGRL on primary: (from instance one in case of RAC)


$dgmgrl
DGMGRL for Linux: Version 10.2.0.3.0 - Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL&gt; CONNECT sys/;
Connected.
Step 5. Create configuration on primary:


DGMGRL&gt; CREATE CONFIGURATION 'PRODCONF' AS PRIMARY DATABASE IS 'PROD10G' CONNECT IDENTIFIER IS PROD10G;

Configuration "PRODCONF" created with primary database "PROD10G"
Step 6. Add standby in the configuration:


DGMGRL&gt; ADD DATABASE 'STDBY10G' AS CONNECT IDENTIFIER IS STDBY10G MAINTAINED AS PHYSICAL;

Database "STDBY10G" added
Step 7. Enable Configuration:


DGMGRL&gt; ENABLE CONFIGURATION;
Enabled.
Use the SHOW command to verify that the configuration and its databases were successfully enabled and brought online:


DGMGRL&gt; SHOW CONFIGURATION;

Configuration
Name: PRODCONF
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED

Databases:
PROD10G - Primary database
STDBY10G - Physical standby database

Current status for "PRODCONF"
SUCCESS
References

NOTE:454418.1 - Unable To Recreate Data Guard Fast Start Failover Configuration With DGMGRL
Doc ID 808783.1- Step By Step How to Recreate Dataguard Broker Configuration ()

Drop and Cleanup a Non-RAC Database

To drop and cleanup a Non-RAC Oracle database, implement following steps as user oracle.


Step 1 – Drop OEM

Set ORACLE_SID and ORACLE_UNQNAME as follows:
$>. oraenv <= set ORACLE_SID
$>cd
$>. .profile  <= set ORACLE_UNQNAME using .profile

Try to drop OEM (if exists) as follows:
$>emca -deconfig dbcontrol db -repos drop

If OEM is dropped by emca, followings will be deleted . Otherwise delete them manually:

Directories:
$><ORACLE_HOME>/<HOSTNAME>_<ORACLE_SID>
$><ORACLE_HOME>/oc4j/j2ee/OC4J_DBConsole_<HOSTNAME>_<ORACLE_SID>

ORACLE_SID related entries (if exists ) in following files:
$><ORACLE_HOME>/install/ReadMe.txt         <= oem port entries - optional
$><ORACLE_HOME>/install/portlist.txt       <= oem port entries


Step 2 – Drop Database

$>sqlplus / as sysdba
sqlplus>shutdown immediate;
sqlplus>starup mount exclusive restrict;
sqlplus>drop database;


Step 3 – Cleanp Residues

Delete following directories manually if exists:
$><ORACLE_BASE>/diag/rdbms/<ORACLE_SID>    <= diagnostic directory

Delete following files manually if exists:
$><ORACLE_HOME>/dbs/init<ORACLE_SID>.ora  <= init.ora file
$><ORACLE_HOME>/dbs/orapw<ORACLE_SID>      <= passwd file

Delete ORACLE_SID related entries (if exists ) from following files:
$>/var/opt/oracle/oratab                    <= oratab
$><ORACLE_HOME>/network/admin/tnsnames.ora <= tnsnames entries
$><ORACLE_HOME>/network/admin/listener.ora <= listener entries