Friday, November 4, 2011

Paul Elbow's crsstat script: Improved Formatting of crs_stat on 10g and 11g

Today I found a very helpful script from Paul Elbow's blog to format crsctl output:
My crsstat script: Improved Formatting of crs_stat on 10g and 11g


Labels: , , ,

Tuesday, November 1, 2011

Differences between "crsctl .. crs" and "crsctl ... cluster"

I had been puzzled by the differences between "crsctl ... crs" and "crsctl ... cluster", so I decided to look into the differences between them, it turns out that they are clearly explained in the documentation:
Oracle® Clusterware Administration and Deployment Guide
11g Release 2 (11.2)

Part Number E16794-14

E CRSCTL Utility Reference

According to the above doc:

CRSCTL Overview

CRSCTL is an interface between you and Oracle Clusterware, parsing and calling Oracle Clusterware APIs for Oracle Clusterware objects.
Oracle Clusterware 11g release 2 (11.2) introduces cluster-aware commands with which you can perform check, start, and stop operations on the cluster. You can run these commands from any node in the cluster on another node in the cluster, or on all nodes in the cluster, depending on the operation.
You can use CRSCTL commands to perform several operations on Oracle Clusterware, such as:
  • Starting and stopping Oracle Clusterware resources
  • Enabling and disabling Oracle Clusterware daemons
  • Checking the health of the cluster
  • Managing resources that represent third-party applications
  • Integrating Intelligent Platform Management Interface (IPMI) with Oracle Clusterware to provide failure isolation support and to ensure cluster integrity
  • Debugging Oracle Clusterware components



Clusterized (Cluster Aware) Commands

You can run clusterized commands on one node to perform operations on another node in the cluster. These are referred to as remote operations. This simplifies administration because, for example, you no longer have to log in to each node to check the status of the Oracle Clusterware on all of your nodes.
Clusterized commands are completely operating system independent; they rely on the OHASD (Oracle High Availability Services daemon). If this daemon is running, then you can perform remote operations, such as the starting, stopping, and checking the status of remote nodes.
Clusterized commands include the following:
  • crsctl check cluster
  • crsctl start cluster
  • crsctl stop cluster

---crsctl command help shows the following .
crsctl stop cluster -h
Usage:
crsctl stop cluster [[-all]|[-n [...]]] [-f]
Stop CRS stack
where
Default Stop local server
-all Stop all servers
-n Stop named servers
server [...] One or more blank-separated server names
-f Force option


In simple words it work like this.

crsctl stop cluster is equal to crsctl stop crs
crsctl stop cluster -all --> stops clsuterware on ALL nodes.
crsctl stop cluster -n racnode1 stops clusterware on node racnode1.

Labels: , , ,

How to change the public interface name of 11202 RAC


Recently I was asked to work SA to change the public interface name of our 11202 two nodes RAC on Linux 5.5.  After researching Oracle Support notes and confirmation with Oracle Support, following are the steps required to perform this change:
Note: unfortunately at last minute, this change was cancelled, so I didn't have a chance to really test it out. I tested another set of similar steps but failed due to wrong steps execution.

1) set new public network interface name with oifcfg setif -global <..ifname.>
$ $ORA_CRS_HOME/bin/oifcfg setif -global eth0/192.168.100.0:public

2) delete old public network interface name with oifcfg delif -global
$ $ORA_CRS_HOME/bin/oifcfg delif -global bond0

3) voting disks permission needs to stay permenent to be "oracle:oinstall", even after server reboot

4) SHUTDOWN crs
crsctl stop crs

5) perform changes at OS level

6) reboot both servers
Note: Reboot is not essentail. Needed only if SA need that.

7) After server reboot, check voting disks permission to make sure they belong to "oracle:oinstall".
If not, change it back to oracle:oinstall.
Note: this step is not necessary. I put it here because after SA made the interface name change on OS level and rebooted the server, CRS couldn't be started.  After long hours of hard work, it turned out to be the voting disks permission issue. 

8) Restart the clusterware crsctl start crs
Check crs status, if not all online, shutdown force: crsctl stop crs -f, then start it up node by node: crsctl start crs
Note: what are the differences between "crsctl stop/start crs" and "crsctl stop/start cluster"? 
"crsctl stop/start crs" only stop or start one node. 

9) To modify the VIP change while all cluster related services are online (CRS, ASM, db)
a) Gather the existing setup
$ srvctl config nodeapps -a

b) Verify VIP status
$ crsctl stat res -t
- it should show VIPs are ONLINE

$ ifconfig -a

- VIP logical interface is bind to the public network interface

c) Stop the nodeapps resources and all dependent resources (stop ASM/DB instance only required for full outage):
$ srvctl stop instance -d -n
$ srvctl stop vip -n -f

d) Verify VIP is now OFFLINE and the interface is no longer bound to the public network interface

$ crsctl stat res -t

$ ifconfig -a

e) Modifying VIP and Its Associated Attributes
Determine the new VIP IP/subnet/netmask or VIP hostname, make the network change on OS first, ensure the new VIP is registered in DNS or modified in /etc/hosts
(for Unix/Linux). If the network interface is changed, ensure the new interface is available on the server before proceeding with the modification.

f) Modify the VIP resource
network resource can be modified directly via srvctl modify network command.

as root user:
# srvctl modify network -k ] [-S /[/if1[|if2...]]

For example:
New VIP is: 110.11.70.11 racnode1-nvip
new subnet is 110.11.70.0
new netmask is 255.255.255.0
new interface is eth0

# srvctl modify network -k 1 -S 110.11.70.0/255.255.255.0/eth0

Note:
How to tell if VIP is bound or not bound to the public interface from the output of "ifconfig -a"?
Answer: If the public interface is 'eth0' then vip should bound as 'eth0:1'

g) Verify the change
$ srvctl config nodeapps -a

h) Start the vip and other resources
$ srvctl start vip { -n
$ srvctl start instance -d -i

i) Verify the new VIP is ONLINE and bind to the public network interface

$ crsctl stat res -t
$ ifconfig -a

Related Oracle Support Notes:

  1. How to Change Interconnect/Public Network (Interface or Subnet) in Oracle Clusterware [ID 283684.1]
  2. How to Modify Private Network Interface in 11.2 Grid Infrastructure [ID 1073502.1] 
  3. Troubleshooting CRSD Start up Issue [ID 1323698.1]
  4. CRS can not Start After Node Reboot [ID 733260.1]
  5. 10gR2, 11gR1 and 11gR2 Oracle Clusterware (CRS / Grid Infrastructure) & RAC Command (crsctl, srvctl, cluvfy etc) Syntax and Reference [ID 1332452.1]



Labels: , ,

Friday, October 14, 2011

Reviewed Oracle Notes on My Recent 11202 RAC Installation with ASMLib

ASM
  1. ASM 11.2 Configuration KIT (ASM 11gR2 Installation & Configuration, Deinstallation, Upgrade, ASM Job Role Separation (Standalone Only). [ID 1092213.1]
  2. How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices or Block Devices? [ID 580153.1]
  3. Device Checks for ASM Fails with PRVF-5150: Path ORCL: is not a valid path [ID 1210863.1
  4. ASM Technical Best Practices [ID 265633.1]
  5. Master Note for Automatic Storage Management (ASM) [ID 1187723.1]
  6. How to Prepare Storage for ASM [ID 452924.1]
  7. How To Install ASMLIB API via the Unbreakable Linux Network [ID 461718.1]
  8. FAQ ASMLIB CONFIGURE,VERIFY, TROUBLESHOOT [ID 359266.1]
Grid/CRS/VIP
  1. Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure [ID 1312225.1]
  2. Oracle De-install Utility - 11g R2 [ID 886184.1]
  3. How To Deinstall/Uninstall Oracle Home In 11gR2? [ID 883743.1]
  4. 11.2.0.2 Patchset Location For Grid Infrastructure And RDBMS. [ID 1223673.1]
  5. Support of Linux and Oracle Products on Linux [ID 266043.1]
  6. Grid And Database Runinstaller - Precheck Fails For Run-Level [ID 1307948.1]
  7. Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure [ID 1312225.1]
  8. RAC: Frequently Asked Questions [ID 220970.1]
  9. Linux: PRVF-5449 : Check of Voting Disk location "ORCL:(ORCL:)" failed [ID 1267569.1]
  10. root.sh Fails as the ora.asm resource is not ONLINE or PROTL-16 Reports [ID 1259874.1]
  11. How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]
  12. The listener stops or goes offline randomly due to VIP issue [ID 985170.1]
  13. VIP does not relocate back to the original node starting from 10.2.0.4 and 11.1 even after the public network problem is resolved. [ID 805969.1]
  14. Troubleshooting CRSD Start up Issue [ID 1323698.1]
  15. VIP does not Failover or Network Resource Stays Online after Disconnected Cables for Bond Public Network [ID 1276737.1]
  16. VIP Fails Over and Listener Stops After Short Public Network Hiccup [ID 1333165.1]
  17. Crs Components Are Not Starting After Server Reboot [ID 1152653.1]
  18. Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]
  19. CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage [ID 783456.1]
  20. CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide [ID 330358.1]
  21. RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) [ID 810394.1]
  22. Introducing Cluster Health Monitor (IPD/OS) [ID 736752.1]

Labels: , , , ,