Paul Elbow's crsstat script: Improved Formatting of crs_stat on 10g and 11g

Today I found a very helpful script from Paul Elbow's blog to format crsctl output:
My crsstat script: Improved Formatting of crs_stat on 10g and 11g

ORA-12520: TNS:listener could not find available handler for requested type of server

Today I encountered a strange intermittent database connection problem when trying to connect to a database via SQL*Plus, the error I got is:
ORA-12520: TNS:listener could not find available handler for requested type of server

According to this Oracle note: Intermittent TNS-12516 or TNS-12519 Errors When Connecting Via Oracle Net [ID 240710.1] , it's likely that lack of server processes is the cause of the problem.  So I checked the system parameters settings of this database:
SQL> show parameters process;


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
processes                            integer     150        


SQL> select * from v$resource_limit where resource_name in ('processes','sessions');


RESOURCE_NAME                  CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_ALLOCATION                       LIMIT_VALUE
------------------------------ ------------------- --------------- ---------------------------------------- ----------------------------------------
processes                                       78             150        150                                      150
sessions                                        93             172        264                                      264


The process value is set to the default one by another DBA who created this database.  And also at the same time, there is an import running in the database, and it's using parallelism of 8. Do the settings of parallel related parameters have impact on generating this error? I don't know, I need to do more research about that.

SQL> show parameters parallel


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
fast_start_parallel_rollback         string      LOW
parallel_adaptive_multi_user         boolean     TRUE
parallel_automatic_tuning            boolean     FALSE
parallel_degree_limit                string      CPU
parallel_degree_policy               string      MANUAL
parallel_execution_message_size      integer     16384
parallel_force_local                 boolean     FALSE
parallel_instance_group              string
parallel_io_cap_enabled              boolean     FALSE
parallel_max_servers                 integer     135
parallel_min_percent                 integer     0


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
parallel_min_servers                 integer     0
parallel_min_time_threshold          string      AUTO
parallel_server                      boolean     FALSE
parallel_server_instances            integer     1
parallel_servers_target              integer     384
parallel_threads_per_cpu             integer     2
recovery_parallelism                 integer     0


Anyway, it seems the low value of "process" parameter a likely cause of this problem.

Check 11202 RAC system recorded network interface name

Recently I was required to update our 11202 two nodes RAC on OEL 5.5 public interface name. After interface name change, here are the 2 steps I did to check if the system got the interface name correctly:
[oracle@node2 ~]$
[oracle@node2 ~]$ oifcfg getif
eth3  192.168.206.20  global  cluster_interconnect
bond0  192.168.100.0  global  public


[oracle@node2 ~] more /opt/app/11.2.0/grid/gpnp/linkin2/profiles/peer/profile.xml
IP="192.168.206.20" Adapter="eth3" Use="cluster_interconnect"/>Network id="net2" IP="192.168.100.0" Adapter="bond0" Use="public"/>
:ASM-Profile id="asm" DiscoveryString="/dev/mapper/*" SPFile="+OCR_VOTE/linkincluster/asmparameterfile/registry.253.764469115"/>
CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
ithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
t="gpnp orcl xsi"/>DO5mYZjrYv+5t15UH8zedF0o3O4=
nfo>fyCiD46yDKsjcjpyoFYIuoZ1+sEvE72XPKAOu5gSfLjllUFVgNNM3EW8EiyUhvbmhHgxLcQ5ZL8iTdmhDtun53gbcNVc5RVP82B32WH1qzJUaDLx3MXQsZcO3/wmkyxUo7azsCosrk2WWwYeSqUl4YiArTbv3WPMXwfU9TdFyHM=
reValue>

Differences between "crsctl .. crs" and "crsctl ... cluster"

I had been puzzled by the differences between "crsctl ... crs" and "crsctl ... cluster", so I decided to look into the differences between them, it turns out that they are clearly explained in the documentation:
Oracle® Clusterware Administration and Deployment Guide
11g Release 2 (11.2)
Part Number E16794-14
E CRSCTL Utility Reference

According to the above doc:

CRSCTL Overview

CRSCTL is an interface between you and Oracle Clusterware, parsing and calling Oracle Clusterware APIs for Oracle Clusterware objects.
Oracle Clusterware 11g release 2 (11.2) introduces cluster-aware commands with which you can perform check, start, and stop operations on the cluster. You can run these commands from any node in the cluster on another node in the cluster, or on all nodes in the cluster, depending on the operation.
You can use CRSCTL commands to perform several operations on Oracle Clusterware, such as:

• Starting and stopping Oracle Clusterware resources
• Enabling and disabling Oracle Clusterware daemons
• Checking the health of the cluster
• Managing resources that represent third-party applications
• Integrating Intelligent Platform Management Interface (IPMI) with Oracle Clusterware to provide failure isolation support and to ensure cluster integrity
• Debugging Oracle Clusterware components

---

Clusterized (Cluster Aware) Commands

You can run clusterized commands on one node to perform operations on another node in the cluster. These are referred to as remote operations. This simplifies administration because, for example, you no longer have to log in to each node to check the status of the Oracle Clusterware on all of your nodes.
Clusterized commands are completely operating system independent; they rely on the OHASD (Oracle High Availability Services daemon). If this daemon is running, then you can perform remote operations, such as the starting, stopping, and checking the status of remote nodes.
Clusterized commands include the following:

• crsctl check cluster
• crsctl start cluster
• crsctl stop cluster
  

---crsctl command help shows the following .
crsctl stop cluster -h
Usage:
crsctl stop cluster [[-all]|[-n [...]]] [-f]
Stop CRS stack
where
Default Stop local server
-all Stop all servers
-n Stop named servers
server [...] One or more blank-separated server names
-f Force option

In simple words it work like this.

crsctl stop cluster is equal to crsctl stop crs
crsctl stop cluster -all --> stops clsuterware on ALL nodes.
crsctl stop cluster -n racnode1 stops clusterware on node racnode1.

How to change the public interface name of 11202 RAC

Recently I was asked to work SA to change the public interface name of our 11202 two nodes RAC on Linux 5.5.  After researching Oracle Support notes and confirmation with Oracle Support, following are the steps required to perform this change:
Note: unfortunately at last minute, this change was cancelled, so I didn't have a chance to really test it out. I tested another set of similar steps but failed due to wrong steps execution.

1) set new public network interface name with oifcfg setif -global <..ifname.>
$ $ORA_CRS_HOME/bin/oifcfg setif -global eth0/192.168.100.0:public

2) delete old public network interface name with oifcfg delif -global
$ $ORA_CRS_HOME/bin/oifcfg delif -global bond0

3) voting disks permission needs to stay permenent to be "oracle:oinstall", even after server reboot

4) SHUTDOWN crs
crsctl stop crs

5) perform changes at OS level

6) reboot both servers
Note: Reboot is not essentail. Needed only if SA need that.

7) After server reboot, check voting disks permission to make sure they belong to "oracle:oinstall".
If not, change it back to oracle:oinstall.
Note: this step is not necessary. I put it here because after SA made the interface name change on OS level and rebooted the server, CRS couldn't be started.  After long hours of hard work, it turned out to be the voting disks permission issue. 

8) Restart the clusterware crsctl start crs
Check crs status, if not all online, shutdown force: crsctl stop crs -f, then start it up node by node: crsctl start crs
Note: what are the differences between "crsctl stop/start crs" and "crsctl stop/start cluster"? 
"crsctl stop/start crs" only stop or start one node. 

9) To modify the VIP change while all cluster related services are online (CRS, ASM, db)
a) Gather the existing setup
$ srvctl config nodeapps -a

b) Verify VIP status
$ crsctl stat res -t
- it should show VIPs are ONLINE

$ ifconfig -a

- VIP logical interface is bind to the public network interface

c) Stop the nodeapps resources and all dependent resources (stop ASM/DB instance only required for full outage):
$ srvctl stop instance -d -n
$ srvctl stop vip -n -f

d) Verify VIP is now OFFLINE and the interface is no longer bound to the public network interface

$ crsctl stat res -t

$ ifconfig -a

e) Modifying VIP and Its Associated Attributes
Determine the new VIP IP/subnet/netmask or VIP hostname, make the network change on OS first, ensure the new VIP is registered in DNS or modified in /etc/hosts
(for Unix/Linux). If the network interface is changed, ensure the new interface is available on the server before proceeding with the modification.

f) Modify the VIP resource
network resource can be modified directly via srvctl modify network command.

as root user:
# srvctl modify network -k ] [-S /[/if1[|if2...]]

For example:
New VIP is: 110.11.70.11 racnode1-nvip
new subnet is 110.11.70.0
new netmask is 255.255.255.0
new interface is eth0

# srvctl modify network -k 1 -S 110.11.70.0/255.255.255.0/eth0

Note:
How to tell if VIP is bound or not bound to the public interface from the output of "ifconfig -a"?
Answer: If the public interface is 'eth0' then vip should bound as 'eth0:1'

g) Verify the change
$ srvctl config nodeapps -a

h) Start the vip and other resources
$ srvctl start vip { -n
$ srvctl start instance -d -i

i) Verify the new VIP is ONLINE and bind to the public network interface

$ crsctl stat res -t
$ ifconfig -a

Related Oracle Support Notes:

1. How to Change Interconnect/Public Network (Interface or Subnet) in Oracle Clusterware [ID 283684.1]
2. How to Modify Private Network Interface in 11.2 Grid Infrastructure [ID 1073502.1] 
3. Troubleshooting CRSD Start up Issue [ID 1323698.1]
4. CRS can not Start After Node Reboot [ID 733260.1]
5. 10gR2, 11gR1 and 11gR2 Oracle Clusterware (CRS / Grid Infrastructure) & RAC Command (crsctl, srvctl, cluvfy etc) Syntax and Reference [ID 1332452.1]