2.1.1 OPatch Utility Information 
每個節點的gi home和db home執行,version 11.2.0.3.6 or later. 
$ unzip <OPATCH-ZIP> -d <ORACLE_HOME> 
$ <ORACLE_HOME>/OPatch/opatch version 

#su - root 
chmod 777 /tmp/p6880880_112000_Linux-x86-64.zip 
chown grid:oinstall /tmp/p6880880_112000_Linux-x86-64.zip 
su - grid 
unzip /tmp/p6880880_112000_Linux-x86-64.zip -d /u01/app/11.2.0/grid/ 
replace /u01/app/11.2.0/grid/OPatch/jlib/oracle.opatch.classpath.jar? [y]es, [n]o, [A]ll, [N]one, [r]ename: A 

#su - root 
chown oracle:oinstall /tmp/p6880880_112000_Linux-x86-64.zip 
su - oracle 
unzip /tmp/p6880880_112000_Linux-x86-64.zip -d /u01/app/oracle/product/11.2.0/dbhome_1/ 
rm /tmp/p6880880_112000_Linux-x86-64.zip 



2.1.2 OCM(Oracle Configuration Manager) Configuration 
創建ocm README沒直接給出，而是引用了一篇文檔 
示例中ORACLE_HOME是db_home，此次主要是gi PSU所以安裝用gi_home

./emocmrsp后可不加參數，將在在當前目錄下生成ocm.rsp
節點2同樣執行：

[root@rac-node1 ~]# su - grid 
[grid@rac-node1 ~]$ /u01/app/11.2.0/grid/OPatch/ocm/bin/emocmrsp -no_banner -output /tmp/ocm_my.rsp 
Provide your email address to be informed of security issues, install and 
initiate Oracle Configuration Manager. Easier for you if you use your My 
Oracle Support Email address/User Name. 
Visit http://www.oracle.com/support/policies.html for details. 
Email address/User Name: 

You have not provided an email address for notification of security issues. 
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y 
The OCM configuration response file (/tmp/ocm_my.rsp) was successfully created. 
[grid@rac-node1 ~]$ ll /tmp/ocm_my.rsp 
-rw-r--r-- 1 grid oinstall 621 Oct 31 07:59 /tmp/ocm_my.rsp 
[grid@rac-node1 ~]$ chmod 777 /tmp/ocm_my.rsp 
[grid@rac-node1 ~]$ ll /tmp/ocm_my.rsp 
-rwxrwxrwx 1 grid oinstall 621 Oct 31 07:59 /tmp/ocm_my.rsp

2.1.3 Validation of Oracle Inventory 
每個節點的gi home和db home執行： 
$ <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME> 

su - grid 
/u01/app/11.2.0/grid/OPatch/opatch lsinventory -detail -oh /u01/app/11.2.0/grid/ 

su - oracle 
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -detail -oh /u01/app/oracle/product/11.2.0/dbhome_1/ 




2.1.4 Download and Unzip the Patch 
所有節點都online可以訪問 
grid用戶解壓到共享目錄<UNZIPPED_PATCH_LOCATION>（可以非共享） 
目錄不能有其他文件，（不能是/tmp根目錄下，/tmp下建子目錄可以） 
ORA_INSTALL group需要有讀的權限 
su - grid

2.1.5 Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch 
oracle用戶停止em: 
$ <ORACLE_HOME>/bin/emctl stop dbconsole 



2.3 OPatch auto for GI 
root運行會同時patch gi_home和db_home。 
如果GI home or Oracle RAC database home is in non-shared storage.，要在每個節點手動執行，并且不能同時執行。 


2.4 Patch Installation 

補丁安裝的2種情形: 
Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured. 
Case 2: GI Home is not shared, Database Home is shared, ACFS may be used. 

Patching Oracle RAC Database Homes and GI Together 
Case 1: GI Home and the Database Homes that are not shared and ACFS file system is not configured. 

As root user, execute the following command on each node of the cluster: 
# opatch auto <UNZIPPED_PATCH_LOCATION>/19380115 -ocmrf <ocm response file> 
同時打gi_home和db_home 需讀取ocm.rsp權限 
chmod 777 /tmp/ocm_my.rsp 
/u01/app/11.2.0/grid/OPatch/opatch auto /tmp/patch/19380115 -ocmrf /tmp/ocm.rsp 


對于auto patch失敗的部分，無需rollback，再次auto即可 
Verifying environment and performing prerequisite checks... 
All of the constituents of the composite patch are already installed in the Oracle Home. No need to apply this patch. 
Log file location: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch3014-10-30_23-06-06PM_1.log 


--其他節點執行同樣的操作 



2.5 Patch Post-Installation Instructions安裝后期指令 

2.5.2 Loading Modified SQL Files into the Database 
1.在任一節點執行 

SQL> startup

SQL> @?/rdbms/admin/catbundle.sql psu apply
更新dba_registry_history view以便使用psu 
--沒提示編輯無效對象，psu的過程應該包括 


2.檢查日志： $ORACLE_BASE/cfgtoollogs/catbundle for any errors: 
catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log 
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log 


2.5.3 更新rman Catalog 
If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it: 
$ rman catalog username/password@alias 
RMAN> UPGRADE CATALOG; 



2.6 Patch Post-Installation Instructions for Databases Created or Upgraded after Installation of Patch in the Oracle Home 
安裝db_home補丁后,使用模版創建的database需要執行"Loading Modified SQL Files into the Database" 

These instructions are for a database that is created or upgraded after the installation of the patch. 

You must execute the steps in Section 2.5.2, "Loading Modified SQL Files into the Database" for any new database only if it was created by any of the following methods: 

Using DBCA (Database Configuration Assistant) to select a sample database (General, Data Warehouse, Transaction Processing) 

Using a script that was created by DBCA that creates a database from a sample database 

There are no actions required for databases that have been upgraded. 

查詢DB補丁記錄 
set linesize 300 
col action_time for a30 
col version for a15 
col id for 99 
col action for a10 
col namespace for a10 
col COMMENTS for a20 
col BUNDLE_SERIES for a15 
select * from registry$history; 
查詢補丁狀態grid、oracle皆可 
$ $ORACLE_HOME/OPatch/opatch lspatches 
$ $ORACLE_HOME/OPatch/opatch lsinventory 

2.7 Patch Deinstallation 

Roll Back the Oracle RAC Database Homes and GI Together 

Case 1: GI Home and Database Homes that are not shared and ACFS file system is not configured. 

As root user, execute the following command on each node of the cluster. 

# opatch auto <UNZIPPED_PATCH_LOCATION>/19380115 -rollback -ocmrf <ocm response file> 

/u01/app/11.2.0/grid/OPatch/opatch auto /tmp/patch/19380115 -rollback -ocmrf /tmp/ocm.rsp

節點2同樣執行

If the message, "A system reboot is recommended before using ACFS" is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

Case 2: GI Home is not shared, Database Home is shared and ACFS may be used. 

From the Oracle database home, make sure to stop the Oracle RAC databases running on all nodes. As the database home owner execute: 

$ <ORACLE_HOME>/bin/srvctl stop database –d <db-unique-name> 
On the 1st node, unmount the ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. 

On the 1st node, roll back the patch from the GI Home using the opatch auto command. As root user, execute the following command: 

# opatch auto <UNZIPPED_PATCH_LOCATION>/19380115 -oh <GI_HOME> -rollback -ocmrf <ocm response file> 
If the message, "A system reboot is recommended before using ACFS” is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

On the 1st node, remount ACFS file systems. See My Oracle Support Document 1494652.1 for mounting ACFS file systems. 

On the 1st node, roll back the patch to the Database home using the opatch auto command. This operation will rollback the patch to the Database home across the cluster given that it is a shared ACFS home. Note that a USM only patch cannot be applied to a Database home. As root user, execute the following command: 

# opatch auto <UNZIPPED_PATCH_LOCATION>/19380115 -oh <DATABASE_HOME> -rollback -ocmrf <ocm response file> 
On the 1st node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute: 

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename> 
On the 2nd (next) node, unmount the ACFS file systems. See My Oracle Support Document 1494652.1 for unmounting ACFS file systems. 

On the 2nd node, roll back the patch to GI Home using the opatch auto command. As root user, execute the following command: 

# opatch auto <UNZIPPED_PATCH_LOCATION>/19380115 -oh <GI_HOME> -rollback -ocmrf <ocm response file> 
If the message, "A system reboot is recommended before using ACFS” is shown, then a reboot must be issued before continuing. Failure to do so will result in running with an unpatched ACFS\ADVM\OKS driver. 

On the 2nd node, running the opatch auto command in Step 9 will restart the stack. 

On the 2nd node, remount ACFS file systems. See My Oracle Support Document 1494652.1 for mounting ACFS file systems. 

On the 2nd node only, restart the Oracle instance, which you have previously stopped in Step 1. As the database home owner execute: 

$ <ORACLE_HOME>/bin/srvctl start instance –d <db-unique-name> -n <nodename> 
Repeat Steps 8 through 13 for all remaining nodes of the cluster. 

2.8 Patch Post-Deinstallation Instructions for an Oracle RAC Environment 

Follow these steps only on the node for which the steps in Section 2.5.2, "Loading Modified SQL Files into the Database" were executed during the patch application.: 

只需在一個節點上執行，升級時執行了"Loading Modified SQL Files into the Database"的節點上

Start all database instances running from the Oracle home. (For more information, see Oracle Database Administrator's Guide.) 
啟動所有的instance

For each database instance running out of the ORACLE_HOME, connect to the database using SQL*Plus as SYSDBA and run the rollback script as follows: 

sqlplus / AS SYSDBA 
SQL> STARTUP 
SQL> @?/rdbms/admin/catbundle_PSU_<database SID PREFIX>_ROLLBACK.sql 

 SQL> @?/rdbms/admin/catbundle_PSU_CLUSDB_ROLLBACK.sql

In an Oracle RAC environment, the name of the rollback script will have the format catbundle_PSU_<database SID PREFIX>_ROLLBACK.sql. 

SQL> set linesize 300

SQL> col action_time for a30

SQL> col version for a15

SQL> col id for 99

SQL> col action for a10

SQL> col namespace for a10

SQL> col COMMENTS for a20

SQL> col BUNDLE_SERIES for a15

SQL> select * from registry$history;

ACTION_TIME ACTION NAMESPACE VERSION ID COMMENTS BUNDLE_SERIES

------------------------------ ---------- ---------- --------------- --- -------------------- ---------------

30-OCT-14 04.02.28.027746 PM APPLY SERVER 11.2.0.4 0 Patchset 11.2.0.2.0 PSU

31-OCT-14 08.44.46.147995 AM APPLY SERVER 11.2.0.4 4 PSU 11.2.0.4.4 PSU

31-OCT-14 02.01.00.767507 PM ROLLBACK SERVER 11.2.0.4 4 PSU 11.2.0.4.4 PSU

31-OCT-14 02.01.09.369049 PM APPLY SERVER 11.2.0.4 0 Patchset 11.2.0.2.0 PSU

Check the log file for any errors. The log file is found in $ORACLE_BASE/cfgtoollogs/catbundle and is named catbundle_PSU_<database SID>_ROLLBACK_<TIMESTAMP>.log where TIMESTAMP is of the form YYYYMMMDD_HH_MM_SS. If there are errors, see Section 3, "Known Issues". 

Ensure that you verify the Oracle Inventory and compare the output with the one you ran in Section 2.1.3, "Validation of Oracle Inventory" and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command: 

$ opatch lsinventory 
All other instances can be started and accessed as usual while you are executing the deinstallation steps. 

/u01/app/11.2.0/grid/OPatch/opatch lsinventory

/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory

3 Known Issues

This section includes the following known issues: 

Issue 1

When Trace File Analyzer (TFA) is patched as part of a PSU, any rollback of the PSU will not roll back any patch applied to TFA. This is intended behavior, but if there are any issues with patching TFA itself, then manual steps will be required to recover.

回退psu不會回退Trace File Analyzer (TFA)，這是有意的行為。

If a PSU has been rolled back and due to some issue with patching TFA it has to be reinstalled, then you should:

Remove TFA by executing the following command on each node as the root user:

GIHOME/tfa/<nodename>/tfa_home/bin/uninstalltfa.sh -local 

Reinstall the previous TFA version by executing the following command on each node as the root user:

GIHOME/crs/install/tfa_setup.sh -crshome <path to GIHOME> -silent 

Issue 2

After rolling back using opatch auto from the Oracle Grid infrastructure Patch Set Update 11.2.0.4.4, database and listener resources are going to be offline.

You should start the database and listener resources using the srvctl command.

When performing any further patching actions, databases and listeners may not be started up automatically for the home being patched on a node. Start up these explicitly by executing srvctl start database and srvctl start listener commands.

排錯：

2014-10-30 23:03:10: status of apply patch is 18688 
2014-10-30 23:03:10: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.6 
Copyright (c) 2013, Oracle Corporation. All rights reserved. 


Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1 
Central Inventory : /u01/app/oraInventory 
from : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc 
OPatch version : 11.2.0.3.6 
OUI version : 11.2.0.4.0 
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch3014-10-30_23-03-00PM_1.log 

Verifying environment and performing prerequisite checks... 
Prerequisite check "CheckActiveFilesAndExecutables" failed. 
The details are: 


Following executables are active : 
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed. 
Log file location: /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch3014-10-30_23-03-00PM_1.log 

OPatch failed with error code 73 



libclntsh.so.11.1文件busy，剛開始沒想到殺進程，重啟了系統，之后fuser 文件沒提示有人用，再次auto opatch成功。 
fuser /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 


如果有則執行如下： 
oracle@mydb1>/sbin/fuser /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: 12195m 
oracle@mydb1> 
oracle@mydb1>ps -ef |grep 12195 
oracle 12195 11944 0 Jan19 pts/7 00:00:00 adrci 
oracle 22813 3717 0 21:18 pts/13 00:00:00 grep 12195 
oracle@mydb1>kill -9 12195