PRCR-1079,CRS-2674,CRS-5804,CRS-2632 when "srvctl start database -d orcl

Environment
2 node 11.2.0.2 Oracle RAC database on OEL 5.5 platform

Problem
Today I tried to enable archive log mode in these 2 node 11.2.0.2 RAC database, so I followed this article and my previous post to change the database to archive log mode:

1. Login to one of the nodes (i.e. linux1) and disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:
   
   $ sqlplus "/ as sysdba"
   $ sqlplus alter system set cluster_database=false scope=spfile sid='orcl1';
2. Shutdown all instances accessing the clustered database:
   
   $ srvctl stop database -d orcl
3. Using the local instance, MOUNT the database:
   
   $ sqlplus "/ as sysdba"
   $ sqlplus startup mount
4. Enable archiving and set archive location and log_archive_format
   
   $ sqlplus alter database archivelog;
$ sqlplus alter system set log_archive_format='arch_%t_%s.arc'   scope=spfile;
System altered.
$ sqlplus ALTER SYSTEM SET log_archive_dest_1='location=/u11/archive' scope=spfile;
System altered.
$ sqlplus ALTER SYSTEM SET log_archive_dest_2='location=/u12/archive' scope=spfile;
System altered.
Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance:

$ sqlplus alter system set cluster_database=true scope=spfile sid='orcl1';
Shutdown the local instance:

$ sqlplus shutdown immediate 
Bring all instance back up using srvctl:

$ srvctl start database -d orcl 
(Optional) Bring any services (i.e. TAF) back up using srvctl: 

After I ran this "srvctl start database -d racdb" command, I got following errors:
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-2674: Start of 'ora.orcl.db' on 'orcl1' failed
CRS-5804: Communication error with agent process
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy

There is only orcl2 instance is up and running, the "orcl1" was not started up:

[oracle@node2~]$ srvctl status database -d orcl
Instance orcl1 is not running on node node1
Instance orcl2 is running on node node2



Solution
The above error messages are not helpful in troubleshooting the cause of this problem, after several google and metalink search, I found this support note:  ORA-01078 when starting RAC instance within a private domain [ID 889845.1], I got a hint that I might be able to startup the "orcl1" instance at node1, so I did the following:
In node 1, which is the once couldn't be started up by "srvctl"

1 SQL> startup nomount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORA-19905: log_archive_format must contain %s, %t and %r

2 So I created a pfile out of the spfile while the orcl1 instance was in "nomount" state:
SQL> create pfile='/home/oracle/orcl1_init.ora" from spfile;

3 Edited the "orcl1_init.ora" file to set the right "log_archive_format" parameter value, it turned out that I missed "%r" in my original change, as the red color highlighted text shown in the above:

4 Restart the database by using the newly updated "orcl1_init.ora" file:
SQL> startup nomount pfile='/home/oracle/orcl1_init.ora";

5 Create new spfile out of the pfile:
SQL> create spfile from pfile=/home/oracle/orcl1_init.ora';

6 Shutdown the orcl1 instance
SQL> shutdown immediate;

7 After the above changes, I was able to successfully start up the "orcl" database:

[oracle@node1~]$ srvctl stop database -d orcl
Instance orcl1 is not running on node node1
Instance orcl2 is not running on node node2


[oracle@node1~]$ srvctl start database -d orcl

Instance orcl1 is running on node node1

Instance orcl2 is running on node node2

8 After successfully restarted the database, I checked the service status I created for "Server Side Transparent Application Failover" purpose:

[oracle@node1~]$ srvctl status service -d orcl
Service orcl_taf is running on instance(s) orcl1,orcl2

9 I checked both instances to see if they recognize the 2 log_archive_dest_n location I set in the above steps:
In node1:



NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1                   string      location=/u11/archive
log_archive_dest_2                   string      location=/u12/archive

In node2:

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_1                   string      location=/u11/ora_arch

log_archive_dest_2                   string      location=/u12/ora_arch

As you see that in node2, the instance still kept its original log destination values I set months ago, that's not correct, I want it to point to the new /u11/archive and /u12/archive locations. 

10 To fix the step 9 problem, I did the same steps listed in the "Problem" section, except changing "orcl1" to "orcl2", and running those commands in node2 rather than node1.  After those changes, both instances point to the same archive locations:

In node1:

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_1                   string      location=/u11/archive

log_archive_dest_2                   string      location=/u12/archive

In node2:

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

log_archive_dest_1                   string      location=/u11/archive

log_archive_dest_2                   string      location=/u12/archive