4 Configuring Oracle Clusterware Storage

4.1.1 Overview of Storage Options

Use the information in this overview to help you select your storage option.

http://www.oracle.com/technology/support/metalink/

https://metalink.oracle.com/

https://metalink.oracle.com

4.1.2 Checking for Available Shared Storage with CVU

To check for all shared file systems available across all nodes on the cluster with GPFS, use the following command:

/mountpoint/runcluvfy.sh comp ssa -n node_list

If you want to check the shared accessibility of a specific shared storage type to specific nodes in your cluster, then use the following command syntax:

/mountpoint/runcluvfy.sh comp ssa -n node_list -s storageID_list

In the preceding syntax examples, the variable mountpoint is the mountpoint path of the installation media, the variable node_list is the list of nodes you want to check, separated by commas, and the variable storageID_list is the list of storage device IDs for the storage devices managed by the file system type that you want to check.

For example, if you want to check the shared accessibility from node1 and node2 of storage devices /dev/rhdisk8 and /dev/rhdisk9, and your mountpoint is /dev/dvdrom/, then enter the following command:

/dev/dvdrom/runcluvfy.sh comp ssa -n node1,node2 -s /dev/rhdisk8,/dev/rhdisk9

If you do not specify specific storage device IDs in the command, then the command searches for all available storage devices connected to the nodes on the list.

4.2.1 Requirements for Using a File System for Oracle Clusterware Files

To use a file system for Oracle Clusterware files, the file system must comply with the following requirements.

• To use a cluster file system on AIX, you must use GPFS.

• If you choose to place your Oracle Cluster Registry (OCR) files on a shared file system, then one of the following must be true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device that implements file redundancy),

  • At least two file systems are mounted, and use the features of Oracle Database 11g Release 1 (11.1) to provide redundancy for the OCR.

• If you intend to use a shared file system to store database files, then use at least two independent file systems, with the database files on one file system, and the recovery files on a different file system.

• The Oracle Clusterware owner (-, oracle) must have write permissions to create the files in the path that you specify.

Use Table 4-2 to determine the partition size for shared file systems.

In Table 4-2, the total required volume size is cumulative. For example, to store all Oracle Clusterware files on the shared file system with normal redundancy, you should have at least 1.3 GB of storage available over a minimum of three volumes (two separate volume locations for the OCR and OCR mirror, and one voting disk on each volume).

4.2.2 Checking NFS Mount Buffer Size Parameters for Oracle Clusterware

If you are using NFS, then you must set the values for the NFS buffer size parameters rsize and wsize to at least 16384. Oracle recommends that you use the value 32768. Update the /etc/fstab file on each node with an entry similar to the following:

nfs_server:/vol/DATA/oradata  /home/oracle/netapp     nfs\   
cio,rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600

4.2.3 Creating Required Directories for Oracle Clusterware Files on Shared File Systems

Use the following instructions to create directories for Oracle Clusterware files. You can also configure shared file systems for the Oracle Database and recovery files.

To create directories for the Oracle Clusterware files on separate file systems from the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   Note:

   The mount point that you use for the file system must be identical on each node. Make sure that the file systems are configured to mount automatically when a node restarts.

2. Use the df -k command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems that you want to use:

   File Type	File System Requirements
   Oracle Clusterware files	Choose a file system with at least 560 MB of free disk space (one OCR and one voting disk, with external redundancy).

   
   

   If you are using the same file system for more than one type of file, then add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. If the user performing installation has permissions to create directories on the disks where you plan to install Oracle Clusterware, then OUI creates the Oracle Clusterware file directory.

   If the user performing installation does not have write access, then you must create these directories manually using commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on them:

   • Oracle Clusterware file directory:

     # mkdir /mount_point/oracrs
     # chown oracle:oinstall /mount_point/oracrs
     # chmod 775 /mount_point/oracrs
     

When you have completed creating subdirectories in each of the mount point directories, and set the appropriate owner, group, and permissions, you have completed GPFS configuration.

4.3.1 Identifying Required Raw Partitions for Clusterware Files

Table 4-3 lists the number and size of the raw partitions that you must configure for Oracle Clusterware files.

4.3.2 Configuring Raw Disk Devices for Oracle Clusterware Without HACMP or GPFS

If you are installing Oracle RAC on an AIX cluster without HACMP or GPFS, then you must use shared raw disk devices for the Oracle Clusterware files. You can also use shared raw disk devices for database file storage. However, Oracle recommends that you use Automatic Storage Management to store database files in this situation.

This section describes how to configure the shared raw disk devices for Oracle Clusterware files (Oracle Cluster Registry and Oracle Clusterware voting disk). It also describes how to configure shared raw devices for Oracle ASM and for Database files, if you intend to install Oracle Database, and you need to create new disk devices.

To configure shared raw disk devices for Oracle Clusterware files:

1. Identify or configure the required disk devices.

   The disk devices must be shared on all of the cluster nodes.

2. As the root user, enter the following command on any node to identify the device names for the disk devices that you want to use:

   # /usr/sbin/lspv | grep -i none 
   

   This command displays information similar to the following for each disk device that is not configured in a volume group:

   hdisk17         0009005fb9c23648                    None  
   

   In this example, hdisk17 is the device name of the disk and 0009005fb9c23648 is the physical volume ID (PVID).

3. If a disk device that you want to use does not have a PVID, then enter a command similar to the following to assign one to it:

   # chdev -l hdiskn -a pv=yes
   

4. On each of the other nodes, enter a command similar to the following to identify the device name associated with each PVID on that node:

   # /usr/sbin/lspv | grep -i "0009005fb9c23648"
   

   The output from this command should be similar to the following:

   hdisk18         0009005fb9c23648                    None  
   

   In this example, the device name associated with the disk device (hdisk18) is different on this node.

5. If the device names are the same on all nodes, then enter commands similar to the following on all nodes to change the owner, group, and permissions on the character raw device files for the disk devices:

   • OCR device:

     # chown root:oinstall /dev/rhdiskn
     # chmod 640 /dev/rhdiskn
     

   • Other devices:

     # chown oracle:dba /dev/rhdiskn
     # chmod 660 /dev/rhdiskn
     

6. If the device name associated with the PVID for a disk that you want to use is different on any node, then you must create a new device file for the disk on each of the nodes using a common unused name.

   For the new device files, choose an alternative device file name that identifies the purpose of the disk device. The previous table suggests alternative device file names for each file. For database files, replace dbname in the alternative device file name with the name that you chose for the database in step 1.

   Note:

   Alternatively, you could choose a name that contains a number that will never be used on any of the nodes, for example hdisk99.

   To create a new common device file for a disk device on all nodes, perform these steps on each node:

   1. Enter the following command to determine the device major and minor numbers that identify the disk device, where n is the disk number for the disk device on this node:

      # ls -alF /dev/*hdiskn
      

      The output from this command is similar to the following:

      brw------- 1 root system    24,8192 Dec 05 2001  /dev/hdiskn
      crw------- 1 root system    24,8192 Dec 05 2001  /dev/rhdiskn
      

      In this example, the device file /dev/rhdiskn represents the character raw device, 24 is the device major number, and 8192 is the device minor number.

   2. Enter a command similar to the following to create the new device file, specifying the new device file name and the device major and minor numbers that you identified in the previous step:

      Note:

      In the following example, you must specify the c flag to create a character raw device file.

      # mknod /dev/ora_ocr_raw_256m c 24 8192
      

   3. Enter commands similar to the following to change the owner, group, and permissions on the character raw device file for the disk:

      • OCR:

        # chown root:oinstall /dev/ora_ocr_raw_256m
        # chmod 640 /dev/ora_ocr_raw_256m
        

      • Oracle Clusterware voting disk:

        # chown oracle:dba /dev/ora_vote_raw_256m
        # chmod 660 /dev/ora_vote_raw_256m
        

   4. Enter a command similar to the following to verify that you have created the new device file successfully:

      # ls -alF /dev | grep "24,8192"
      

      The output should be similar to the following:

      brw------- 1 root   system   24,8192 Dec 05 2001  /dev/hdiskn
      crw-r----- 1 root   oinstall 24,8192 Dec 05 2001  /dev/ora_ocr_raw_256m
      crw------- 1 root   system   24,8192 Dec 05 2001  /dev/rhdiskn
      

7. To enable simultaneous access to a disk device from multiple nodes, you must set the appropriate Object Data Manager (ODM) attribute listed in the following table to the value shown, depending on the disk type:

   Disk Type	Attribute	Value
   SSA, FAStT, or non-MPIO-capable disks	reserve_lock	no
   ESS, EMC, HDS, CLARiiON, or MPIO-capable disks	reserve_policy	no_reserve

   
   

   To determine whether the attribute has the correct value, enter a command similar to the following on all cluster nodes for each disk device that you want to use:

   # /usr/sbin/lsattr -E -l hdiskn
   

   If the required attribute is not set to the correct value on any node, then enter a command similar to one of the following on that node:

   • SSA and FAStT devices

     # /usr/sbin/chdev -l hdiskn  -a reserve_lock=no
     

   • ESS, EMC, HDS, CLARiiON, and MPIO-capable devices

     # /usr/sbin/chdev -l hdiskn  -a reserve_policy=no_reserve
     

8. Enter commands similar to the following on any node to clear the PVID from each disk device that you want to use:

   # /usr/sbin/chdev -l hdiskn -a pv=clear
   

   When you are installing Oracle Clusterware, you must enter the paths to the appropriate device files when prompted for the path of the OCR and Oracle Clusterware voting disk, for example:

   /dev/rhdisk10
   

4.3.3 Configuring HACMP Multinode Disk Heartbeat (MNDHB) for Oracle Clusterware

This section contains the following topics:

• Overview of Requirements for Using HACMP with Oracle Clusterware

• Deploying HACMP and MDNDHB for Oracle Clusterware

• Upgrading an Existing Oracle Clusterware and HACMP Installation

4.3.4 Configuring Raw Logical Volumes for Oracle Clusterware

This section describes how to configure raw logical volumes for Oracle Clusterware and database file storage. The procedures in this section describe how to create a new volume group that contains the logical volumes required for both types of files.

Before you continue, review the following guidelines which contain important information about using volume groups with this release of Oracle RAC:

• You must use concurrent-capable volume groups for Oracle Clusterware.

• The Oracle Clusterware files require less than 560 MB of disk space, with external redundancy. To make efficient use of the disk space in a volume group, Oracle recommends that you use the same volume group for the logical volumes for both the Oracle Clusterware files and the database files.

• If you are upgrading an existing Oracle9i release 2 Oracle RAC installation that uses raw logical volumes, then you can use the existing SRVM configuration repository logical volume for the OCR and create a new logical volume in the same volume group for the Oracle Clusterware voting disk. However, you must remove this volume group from the HACMP concurrent resource group that activates it before you install Oracle Clusterware.

  See Also:

  The HACMP documentation for information about removing a volume group from a concurrent resource group.

  Note:

  If you are upgrading a database, then you must also create a new logical volume for the SYSAUX tablespace. Refer to the "Configuring Raw Logical Volumes in the New Oracle Clusterware Volume Group" section for more information about the requirements for the Oracle Clusterware voting disk and SYSAUX logical volumes.

• You must use a HACMP concurrent resource group to activate new or existing volume groups that contain only database files (not Oracle Clusterware files).

  See Also:

  The HACMP documentation for information about adding a volume group to a new or existing concurrent resource group.

• All volume groups that you intend to use for Oracle Clusterware must be activated in concurrent mode before you start the installation.

• The procedures in this section describe how to create basic volumes groups and volumes. If you want to configure more complex volumes, (using mirroring, for example), then use this section in conjunction with the HACMP documentation.

4.3.5 Creating a Volume Group for Oracle Clusterware

To create a volume group for the Oracle Clusterware files:

1. If necessary, install the shared disks that you intend to use.

2. To ensure that the disks are available, enter the following command on every node:

   # /usr/sbin/lsdev -Cc disk
   

   The output from this command is similar to the following:

   hdisk0 Available 1A-09-00-8,0  16 Bit LVD SCSI Disk Drive
   hdisk1 Available 1A-09-00-9,0  16 Bit LVD SCSI Disk Drive
   hdisk2 Available 17-08-L       SSA Logical Disk Drive
   

3. If a disk is not listed as available on any node, then enter the following command to configure the new disks:

   # /usr/sbin/cfgmgr
   

4. Enter the following command on any node to identify the device names and any associated volume group for each disk:

   # /usr/sbin/lspv
   

   The output from this command is similar to the following:

   hdisk0     0000078752249812   rootvg
   hdisk1     none               none
   hdisk4     00034b6fd4ac1d71   ccvg1
   

   For each disk, this command shows:

   • The disk device name

   • Either the 16 character physical volume identifier (PVID) if the disk has one, or none

   • Either the volume group to which the disk belongs, or none

   The disks that you want to use may have a PVID, but they must not belong to existing volume groups.

5. If a disk that you want to use for the volume group does not have a PVID, then enter a command similar to the following to assign one to it:

   # /usr/sbin/chdev -l hdiskn -a pv=yes
   

6. To identify used device major numbers, enter the following command on each node of the cluster:

   # ls -la /dev | more
   

   This command displays information about all configured devices, similar to the following:

   crw-rw----   1 root     system    45,  0 Jul 19 11:56 vg1
   

   In this example, 45 is the major number of the vg1 volume group device.

7. Identify an appropriate major number that is unused on all nodes in the cluster.

8. To create a volume group, enter a command similar to the following, or use SMIT (smit mkvg):

   # /usr/sbin/mkvg -y VGname -B -s PPsize -V majornum -n \
   -C PhysicalVolumes
   

9. The following table describes the options and variables used in this example. Refer to the mkvg man page for more information about these options.

   Command Option	SMIT Field	Sample Value and Description
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or it could specify the name of the database that you intend to create.
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or for a database volume group, it could specify the name of the database that you intend to create.
   -B	Create a big VG format Volume Group	Specify this option to create a big VG format volume group. Note: If you are using SMIT, then choose yes for this field.
   -s PPsize	Physical partition SIZE in megabytes	32 Specify the size of the physical partitions for the database. The sample value shown enables you to include a disk up to 32 GB in size (32 MB * 1016).
   -V Majornum	Volume Group MAJOR NUMBER	46 Specify the device major number for the volume group that you identified in Step 7.
   -n	Activate volume group AUTOMATICALLY at system restart	Specify this option to prevent the volume group from being activated at system restart. Note: If you are using SMIT, then choose no for this field.
   -C	Create VG Concurrent Capable	Specify this option to create a concurrent capable volume group. Note: If you are using SMIT, then choose yes for this field.
   PhysicalVolumes	PHYSICAL VOLUME names	hdisk3 hdisk4 Specify the device names of the disks that you want to add to the volume group.

   
   

10. Enter a command similar to the following to vary on the volume group that you created:

    # /usr/sbin/varyonvg VGname
    

4.3.6 Configuring Raw Logical Volumes in the New Oracle Clusterware Volume Group

To create the required raw logical volumes in the new Oracle Clusterware volume group:

1. Identify the logical volumes that you must create.

2. If you prefer, you can also use the command smit mklv to create raw logical volumes.

   The following example shows the command used to create a logical volume for the ocr volume group in the SYSAUX tablespace with a physical partition size of 114 MB (1792/7 = 256):

   # /usr/sbin/mklv -y test_sysaux_raw_1792m -T O -w n -s n -r n ocr 7
   

3. Change the owner, group, and permissions on the character device files associated with the logical volumes that you created, as follows:

   Note:

   The device file associated with the Oracle Cluster Registry must be owned by root. All other device files must be owned by the Oracle software owner user (oracle).

   # chown oracle:dba /dev/rora_vote_raw_256m
   # chmod 660 /dev/rora_vote_raw_256m
   # chown root:oinstall /dev/rora_ocr_raw_256m
   # chmod 640 /dev/rora_ocr_raw_256m
   

4.3.7 Importing the Volume Group on the Other Cluster Nodes

To make the volume group available to all nodes in the cluster, you must import it on each node, as follows:

1. Because the physical volume names may be different on the other nodes, enter the following command to determine the PVID of the physical volumes used by the volume group:

   # /usr/sbin/lspv
   

2. Note the PVIDs of the physical devices used by the volume group.

3. To vary off the volume group that you want to use, enter a command similar to the following on the node where you created it:

   # /usr/sbin/varyoffvg VGname
   

4. On each cluster node, complete the following steps:

   1. Enter the following command to determine the physical volume names associated with the PVIDs you noted previously:

      # /usr/sbin/lspv
      

   2. On each node of the cluster, enter commands similar to the following to import the volume group definitions:

      # /usr/sbin/importvg -y VGname -V MajorNumber PhysicalVolume
      

      In this example, MajorNumber is the device major number for the volume group and PhysicalVolume is the name of one of the physical volumes in the volume group.

      For example, to import the definition of the oracle_vg1 volume group with device major number 45 on the hdisk3 and hdisk4 physical volumes, enter the following command:

      # /usr/sbin/importvg -y oracle_vg1 -V 45 hdisk3
      

   3. Change the owner, group, and permissions on the character device files associated with the logical volumes you created, as follows:

      # chown oracle:dba /dev/rora_vote_raw_256m
      # chmod 660 /dev/rora_vote_raw_256m
      # chown root:oinstall /dev/rora_ocr_raw_256m
      # chmod 640 /dev/rora_ocr_raw_256m
      

   4. Enter the following command to ensure that the volume group will not be activated by the operating system when the node starts:

      # /usr/sbin/chvg -a n VGname
      

4.3.8 Activating the Volume Group in Concurrent Mode on All Cluster Nodes

To activate the volume group in concurrent mode on all cluster nodes, enter the following command on each node:

# /usr/sbin/varyonvg -c VGname