Tips about Troubleshooting RAC

1. Installation Log Files
   software installation log files inside the logs directory of Oracle central inventory(designed by oraInst.loc);log file for the root.sh configuration script is the $GRID_HOME/cfgtools/crsconfig/rootcrs_.log.
2. log directory location for Oracle RDBMS
   1)database alert.log:$ORACLE_BASE/diag/rdbms/<dbname>/<sid>/trace/
   2)all diagnostic data is strored in $ORACLE_BASE/diag/.
3. log directory location for Oracle Grid Infrastructure
   1)cluster node alert.log:$GRID_HOME/log/<hostname>/
   2)all diagnostic data is stored in $GRID_HOME/log/<hostname>/
4. troubleshooting rac steps
   1)database alert.log-->grid alert.log-->specific log file
5. RAC ON and OFF
   1)RAC ON
   [oracle@rac1~]$ cd $ORACLE_HOME/lib
   [oracle@rac1 lib]$ make -f ins_rdbms.mk rac_off
   [oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle
   2)RAC OFF
   [oracle@rac1~]$ cd $ORACLE_HOME/lib
   [oracle@rac1 lib]$ make -f ins_rdbms.mk rac_on
   [oracle@rac1 lib]$ make -f ins_rdbms.mk ioracle
6. Hung Database
   1)systemstate dump.
   2)hanganalyze.
7. Debugging Node eviction Issues
   1)The node is not able to complete the network heartbeat (NHB).
   2)The node is not able to complete the disk heartbeat (DHB).
   3)The node does not have enough cpu to perform either of the heartbeat operations.
8. Cluster Health Monitor
   1)Check the clock synchronization between cluster nodes.
   2)Check the network connectivity between cluster nodes.
   3)Check and ensure that all cluster nodes can access the shared storage where the OCR and voting disks are stored.
   4)Check that there are enough OS resources (such as cpu and memory) available even when there is peak activity in the Oracle RAC database. Cluster Health Monitor will be helpful in analyzing the historical cpu data.
9. Cluster Reconfiguration Steps(IMR)
   1)The name service is frozen. The CGS contains an internal database of all the members/instances in the cluster with all their configuration and servicing details. The name service provides a mechanism to address this configuration data in a structured and synchronized manner.
   2)The lock database (IDLM) is frozen. The lock database is frozen to prevent processes from
   obtaining locks on resources that were mastered by the departing/dead instance.
   3) Determination of membership and validation and IMR occur.
   4) Bitmap rebuild takes place,including instance name and uniqueness verification. CGS must
   synchronize the cluster to be sure that all members get the reconfiguration event and that they
   all see the same bitmap.
   5)Delete all dead instance entries and republish all names newly configured.
   6)Unfreeze and release the name service for use.
   7)Hand over reconfiguration to GES/GCS.
10. Advanced Debugging for Oracle Clusterware Modules
    # crsctl get log -h
    # crsctl start crs
    # crsctl get log css all
    # crsctl set log css OLR:5
    # crsctl set log res ora.backup.vip:3
    --注：
    1）高级调试能是0~5，值越大记录的诊断信息越多，最高级别可能会泄露一些源码片段，这些也许只能有甲骨文官方支持进行分析才行。
11. Debugging VarIoUs Utilities in Oracle RAC using SRVM_TRACE(true or false,Oracle10g or later);