2 Pre-Installation Tasks for Installing RAC on AIX-Based Systems

Network Hardware Requirements

Each node in the cluster must meet the following requirements:

• Each node must have at least two network adapters; one for the public network interface and one for the private network interface (the interconnect).

• The interface names associated with the network adapters for each network must be the same on all nodes.

• For increased reliability, you can configure redundant public and private network adapters for each node.

• For the public network, each network adapter must support TCP/IP.

• For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better recommended).

  
  

  Note: UDP is the default interconnect protocol for RAC and TCP is the interconnect protocol for Oracle CRS. Token-Ring is not supported for the interconnect.

  
  

IP Address Requirements

Before starting the installation, you must identify or obtain the following IP addresses for each node:

• An IP address and an associated host name registered in the domain name service (DNS) for each public network interface

• One unused virtual IP address and an associated virtual host name registered in DNS that you will configure for the primary public network interface

  The virtual IP address must be in the same subnet as the associated public interface. After installation, you can configure clients to use the virtual host name or IP address. If a node fails, its virtual IP address fails over to another node.

• A private IP address and optional host name for each private interface

  Oracle recommends that you use private network IP addresses for these interfaces, for example: 10.*.*.* or 192.168.*.*. You can use the /etc/hosts file on each node to associate private host names with private IP addresses.

For example, if each node has two public and two private interfaces, you might have the following host names and IP addresses on one of the nodes (rac1) and similar host names and IP addresses on the other nodes:

Checking the Network Requirements

To verify that each node meets the requirements, follow these steps:

1. If necessary, install the network adapters for the public and private networks and configure them with either public or private IP addresses.

2. Register the host names and IP addresses for the public network interfaces in DNS.

3. For each node, register one virtual host name and IP address in DNS.

4. For each private interface on every node, add a line similar to the following to the /etc/hosts file on all nodes, specifying the private IP address and associated private host name:

   10.0.0.1     rac1-priv1
   
   

5. Verify that the entry for the loopback address 127.0.0.1 in the /etc/hosts file is similar to the following:

   127.0.0.1    localhost  loopback
   
   

   The name localhost must be the first host name specified on this line.

6. To identify the interface name and associated IP address for every network adapter, enter the following command:

   # /etc/ifconfig -a
   
   

   From the output, identify the interface name and IP address for all network adapters that you want to specify as public or private network interfaces.

   
   

   Note: When you install Oracle CRS and RAC, you will require this information.

   
   

Check for Required Software

Depending on the products that you intend to install, verify that the following software is installed on the system. The procedure following the table describes how to check these requirements.

bos.adt.base
bos.adt.lib
bos.adt.libm
bos.perf.libperfstat
bos.perf.perfstat
bos.perf.proctools

X11.motif.lib

mqm.Client.Bnd
mqm.Server.Bnd

To ensure that the system meets these requirements, follow these steps:

1. To determine which version of AIX is installed, enter the following command:

   # oslevel -r
   
   

   If the operating system version is lower than AIX 5.2.0.0 Maintenance Level 1 (5200-01), upgrade your operating system to this level. AIX 5L version 5.2 maintenance packages are available from the following Web site:

   https://techsupport.services.ibm.com/server/aix.fdc
   
   

2. To determine whether the required file sets are installed and committed, enter a command similar to the following:

   # lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.perfstat \ bos.perf.libperfstat bos.perf.proctools
   
   

   If a file set is not installed and committed, then install it. See your operating system or software documentation for information about installing file sets.

Check for Required Patches

Depending on the products that you intend to install, verify that the following patches are installed on the system. The procedure following the table describes how to check these requirements.

To ensure that the system meets these requirements, follow these steps:

1. To determine whether an APAR is installed, enter a command similar to the following:

   # /usr/sbin/instfix -i -k "IY22854 IY26778 ..."
   
   

   If an APAR is not installed, download it from the following Web site and install it:

   https://techsupport.services.ibm.com/server/aix.fdc
   
   

   If you are using GPFS and require efix IY52454, see the following FTP site for download instructions:

   ftp://service.software.ibm.com/aix/efixes/iy52454/
   
   

2. To determine whether a PTF is installed, enter a command similar to the following:

   # lslpp -l -B U489726 U485561 ...
   
   

   If a PTF is not installed, download it from the following Web site and install it:

   https://techsupport.services.ibm.com/server/aix.fdc
   
   

3. If you require a CSD for WebSphere MQ, see the following Web site for download and installation information:

   http://www.ibm.com/software/integration/mqfamily/support/summary/aix.html
   
   

4. To continue completing pre-installation tasks, go to the "Create Required UNIX Groups and User" section.

Creating the Oracle Inventory Group

You must create the Oracle Inventory group if it does not already exist. The following subsections describe how to determine the Oracle Inventory group name, if it exists, and how to create it if necessary.

# more /etc/oraInst.loc

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

Creating the OSDBA Group

You must create an OSDBA group in the following circumstances:

• An OSDBA group does not exist, for example, if this is the first installation of Oracle Database software on the system

• An OSBDA group exists, but you want to give a different group of UNIX users database administrative privileges in a new Oracle installation

If the OSDBA group does not exist or if you require a new OSDBA group, create it as follows, using the group name dba unless a group with that name already exist:

1. Enter the following command:

   # smit security
   
   

2. Choose the appropriate menu items to create the dba group.

3. Press F10 to exit.

Creating an OSOPER Group (Optional)

Create an OSOPER group only if you want to identify a group of UNIX users with a limited set of database administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group. If you want to use an OSOPER group, you must create it in the following circumstances:

• If an OSOPER group does not exist, for example, if this is the first installation of Oracle Database software on the system

• If an OSOPER group exists, but you want to give a different group of UNIX users database operator privileges in a new Oracle installation

If you require an new OSOPER group, create it as follows, using the group name oper unless a group with that name already exists:

1. Enter the following command:

   # smit security

2. Choose the appropriate menu items to create the oper group.

3. Press F10 to exit.

Creating the Oracle Software Owner User

You must create an Oracle software owner user in the following circumstances:

• If an Oracle software owner user does not exist, for example, if this is the first installation of Oracle software on the system

• If an Oracle software owner user exists, but you want to use a different UNIX user, with different group membership, to give database administrative privileges to those groups in a new Oracle Database installation

# id oracle

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

Verifying That the UNIX User nobody Exists

Before installing the software, verify that the UNIX user nobody exists on the system:

1. To determine whether the user exists, enter the following command:

   # id nobody
   
   

   If this command displays information about the nobody user, you do not have to create that user.

2. If the nobody user does not exist, enter the following command to create it:

   # smit security
   
   

   Specify the appropriate options to create an unprivileged nobody user, then press F10 to exit.

3. Repeat this procedure on all of the other cluster nodes.

Create Identical Users and Groups on Other Cluster Nodes

The Oracle software owner user and the Oracle Inventory, OSDBA, and OSOPER groups must exist and be identical on all cluster nodes. To create these identical users and groups, you must identify the user ID and group IDs assigned them on the node where you created them, then create the user and groups with the same name and ID on the other cluster nodes.

Set Up User Equivalence for rsh and rcp on All Cluster Nodes

Before you install and use Oracle Real Application clusters, you must set up user equivalence on all cluster nodes, as follows:

1. On all cluster nodes, edit the /etc/hosts.equiv file in any text editor and add entries similar to the following for all nodes in the cluster:

   node1-public oracle
   node1-public.domain oracle
   node2-public oracle
   node2-public.domain oracle
      .
      .
      .
   

   In this example:

   • noden_public is the host name associated with the primary public network interface

   • domain is the domain name of the system, for example mydomain.com

   
   

   Note: Alternatively, you can create a .rhosts file using the same format in the oracle user's home directory on all nodes in the cluster.

   
   

2. To test user equivalence on all cluster nodes, including the first node, enter commands similar to the following from the node where you will run the Installer:

   # su oracle
   $ rsh nodename1 date
   $ rsh nodename2 date
       .
       .
       .
   $ exit
   
   

   Each command should return only the date setting on the specified node. If a node prompts for a password or displays any other text, then user equivalence is not set up correctly on that node. In this case, verify that the /etc/hosts.equiv (or ~oracle/.rhosts) file has the correct entries.

Configure Shell Limits

Verify that the shell limits shown in the following table are set to the values shown. The procedure following the table describes how to verify and set the values.

To view the current value specified for these shell limits, and to change them if necessary, follow these steps:

1. Enter the following command:

   # smit chuser
   
   

2. In the User NAME field, enter the user name of the Oracle software owner, for example oracle.

3. Scroll down the list and verify that the value shown for the soft limits listed in the previous table is -1.

   If necessary, edit the existing value.

4. When you have finished making changes, press F10 to exit.

Configure System Configuration Parameters

Verify that the maximum number of processes allowed per user is set to 2048 or greater. The procedure following the table describes how to verify and set the value.

1. Enter the following command:

   # smit chgsys
   
   

2. Verify that the value shown for Maximum number of PROCESSES allowed per user is greater than or equal to 2048.

   If necessary, edit the existing value.

3. When you have finished making changes, press F10 to exit.

Oracle Base Directory

The Oracle base directory acts as a top-level directory for Oracle software installations. It is analogous to the C:\Oracle directory used for Oracle software installations on Windows systems. On UNIX systems, the Optimal Flexible Architecture (OFA) guidelines recommend that you use a path similar to the following for the Oracle base directory:

/mount_point/app/oracle_sw_owner

In this example:

• mount_point is the mount point directory for the file system that will contain the Oracle software.

  The examples in this guide use /u01 for the mount point directory. However, you could choose another mount point directory, /oracle or /opt/oracle for example.

• oracle_sw_owner is the UNIX user name of the Oracle software owner, for example oracle.

You can use the same Oracle base directory for more than one installation or you can create separate Oracle base directories for different installations. If different UNIX users install Oracle software on the same system, each user must create a separate Oracle base directory. The following example Oracle base directories could all exist on the same system:

/u01/app/oracle
/u01/app/orauser
/opt/oracle/app/oracle

The following sections describe how to identify existing Oracle base directories that might be suitable for your installation and how to create a new Oracle base directory if necessary.

Regardless of whether you create a new Oracle base directory or decide to use an existing one, you must set the ORACLE_BASE environment variable to specify the full path to this directory.

Oracle Inventory Directory

The Oracle Inventory directory (oraInventory) stores an inventory of all software installed on the system. It is required by, and shared by, all Oracle software installations on a single system. The first time you install Oracle software on a system, the Installer prompts you to specify the path to this directory. If you are installing the software on a local file system, Oracle recommends that you choose the following path:

oracle_base/oraInventory

If the Oracle base directory is on a cluster file system, you must specify a path for the Oracle Inventory directory on a local file system, to enable all of the nodes to have separate inventories.

The Installer creates the directory that you specify and sets the correct owner, group, and permissions on it. You do not need to create it.

CRS Home Directory

The CRS home directory is the directory where you choose to install the software for Oracle Cluster Ready Services. You must install CRS in a separate home directory. When you run the Installer, it prompts you to specify the path to this directory, as well as a name that identifies it. The directory that you specify must be a subdirectory of the Oracle base directory. Oracle recommends that you specify a path similar to the following for the CRS home directory:

oracle_base/product/10.1.0/crs

The Installer creates the directory path that you specify under the Oracle base directory. It also sets the correct owner, group, and permissions on it. You do not need to create this directory.

Oracle Home Directory

The Oracle home directory is the directory where you choose to install the software for a particular Oracle product. You must install different Oracle products, or different releases of the same Oracle product, in separate Oracle home directories. When you run the Installer, it prompts you to specify the path to this directory, as well as a name that identifies it. The directory that you specify must be a subdirectory of the Oracle base directory. Oracle recommends that you specify a path similar to the following for the Oracle home directory:

oracle_base/product/10.1.0/db_1

The Installer creates the directory path that you specify under the Oracle base directory. It also sets the correct owner, group, and permissions on it. You do not need to create this directory.

Identifying an Existing Oracle Base Directory

Existing Oracle base directories might not have paths that comply with OFA guidelines. However, if you identify an existing Oracle Inventory directory or existing Oracle home directories, you can usually identify the Oracle base directories, as follows:

• Identify an existing Oracle Inventory directory

  Enter the following command on all cluster nodes to view the contents of the oraInst.loc file:

  # more /etc/oraInst.loc
  
  

  If the oraInst.loc file exists, the output from this command is similar to the following:

  inventory_loc=/u01/app/oracle/oraInventory
  inst_group=oinstall
  
  

  The inventory_loc parameter identifies the Oracle Inventory directory (oraInventory) on that system. The parent directory of the oraInventory directory is typically an Oracle base directory. In the previous example, /u01/app/oracle is an Oracle base directory.

• Identify existing Oracle home directories

  Enter the following command on all cluster nodes to view the contents of the oratab file:

  # more /etc/oratab
  
  

  If the oratab file exists, it contains lines similar to the following:

  *:/u03/app/oracle/product/10.1.0/db_1:N
  *:/opt/orauser/infra_904:N
  *:/oracle/9.2.0:N
  
  

  The directory paths specified on each line identify Oracle home directories. Directory paths that end with the user name of the Oracle software owner that you want to use are valid choices for an Oracle base directory. If you intend to use the oracle user to install the software, you could choose one of the following directories from the previous example:

  /u03/app/oracle
  /oracle
  

  
  

  Note: If possible, choose a directory path similar to the first (/u03/app/oracle). This path complies with the OFA guidelines.

  
  

Before deciding to use an existing Oracle base directory for this installation, make sure that it satisfies the following conditions:

• It should not be on the same file system as the operating system.

• It must have an identical path on all cluster nodes or it must be on a supported cluster file system.

  If you are not using a cluster file system, create identical Oracle base directories on the other nodes.

• It must have an identical path on all cluster nodes or it must be on a supported cluster file system.

  If you are not using a cluster file system, create identical Oracle base directories on the other nodes.

• It must have sufficient free disk space on all cluster nodes, as follows:

  Requirement	Free Disk Space
  The Oracle base directory will contain only software files.	Up to 4 GB
  The Oracle base directory will contain both software and database files (not recommended for production databases).	Up to 5 GB

  
  

  Use the following command to determine the free disk space on the file system where the Oracle base directory is located:

  # df -k oracle_base_path
  
  

To continue:

• If an Oracle base directory does not exist on the system or if you want to create a new Oracle base directory, see the "Creating a New Oracle Base Directory" section.

• If an Oracle base directory exists and you want to use it, see the "Configure Disk Storage for Oracle CRS and Database Files" section.

  When you are configuring the oracle user's environment later in this chapter, set the ORACLE_BASE environment variable to specify the directory you chose.

• If an Oracle base directory does not exist on the system or if you want to create a new Oracle base directory, see the following section.

Creating a New Oracle Base Directory

Before you create a new Oracle base directory, you must identify an appropriate file system with sufficient free disk space, as follows:

To identify an appropriate file system, follow these steps:

1. Use the df -k command to determine the free disk space on each mounted file system.

2. From the display, identify a file system that has appropriate free space.

   
   

   Note: he file system can be a local file system or a supported cluster file system.

   
   

   The path to the Oracle base directory must be the same on all nodes.

3. Note the name of the mount point directory for the file system that you identified.

To create the Oracle base directory and specify the correct owner, group, and permissions for it, follow these steps:

1. Enter commands similar to the following to create the recommended subdirectories in the mount point directory that you identified and set the appropriate owner, group, and permissions on them:

   # mkdir -p /mount_point/app/oracle_sw_owner
   # chown -R oracle:oinstall /mount_point/app
   # chmod -R 775 /mount_point/app
   
   

   If the mount point you identified is /u01 and oracle is the user name of the Oracle software owner, the recommended Oracle base directory path is as follows:

   /u01/app/oracle
   
   

2. If necessary, repeat the commands listed in the previous step to create the same directory on the other cluster nodes.

3. When you are configuring the oracle user's environment later in this chapter, set the ORACLE_BASE environment variable to specify this directory.

Choosing a Storage Option for Oracle CRS Files

Before you install Oracle Cluster Ready Services, you must choose the storage option that you want to use for the Oracle Cluster Registry (100 MB) and CRS voting disk (20 MB). You cannot use Automatic Storage Management to store these files, because they must be accessible before any Oracle instance starts.

Apart from ASM, you can use any of the storage options listed in the next section for the Oracle CRS files.

Choosing a Storage Option for Oracle Database Files

If you want to create a database during the installation, you must choose one of the following storage options for the database files:

• Cluster file system using GPFS for AIX

• Automatic Storage Management

• Raw logical volumes using HACMP

http://metalink.oracle.com

Choosing a Storage Option for Oracle Database Recovery Files

Additionally, if you want to enable automated backups during the installation, you must choose one of the following storage options for recovery files (the flash recovery area):

• Cluster file system using GPFS for AIX

• Automatic Storage Management

The storage option that you choose for recovery files can be the same as or different to the option you choose for the database files.

Configuring Disk Storage

For information about how to configure disk storage before you start the installation, see one of the following sections depending on your choice:

• To use a file system for Oracle CRS, database, or recovery file storage, see the "Create Directories for Oracle CRS, Database, or Recovery Files" section.

• To use ASM for database or recovery file storage, see the "Configure Disks for Automatic Storage Management" section.

• To use raw devices (partitions or logical volumes) for Oracle CRS or database file storage, see the "Configure Raw Logical Volumes" section.

Guidelines for Placing Oracle CRS Files on a File System

The Installer does not suggest a default location for the Oracle Cluster Registry (OCR) or the Oracle CRS voting disk. If you choose to create these files on a file system, use the following guidelines when deciding where to place them:

• You must choose a shared file system, for example, a cluster file system on a shared disk.

• It must have at least 100 MB of free disk space for the OCR and 20 MB of free disk space for the CRS voting disk.

• For improved reliability, you should choose a file system on a highly available storage device, for example, a RAID device that implements mirroring.

• If you are placing the Oracle Cluster Ready Services software on a shared file system, you can use the same file system for these files.

• The oracle user must have write permissions to create the files in the path you specify.

Guidelines for Placing Oracle Database Files on a File System

If you choose to place the Oracle database files on a file system, use the following guidelines when deciding where to place them:

• You must choose a shared file system, for example, a cluster file system on a shared disk.

• The default path suggested by the Installer for the database file directory is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• You can choose either a single file system or more than one file system to store the database files:

  • If you want to use a single file system, choose a file system on a physical device that is dedicated to the database.

    For best performance and reliability, choose a RAID device or a logical volume on more than one physical device and implement the stripe-and-mirror-everything (SAME) methodology.

  • If you want to use more than one file system, choose file systems on separate physical devices that are dedicated to the database.

    This method enables you to distribute physical I/O and create separate control files on different devices for increased reliability. It also enables you to fully implement the OFA guidelines. You must choose either the Advanced database creation option or the Custom installation type during the installation to implement this method.

• If you intend to create a preconfigured database during the installation, the file system (or file systems) that you choose must have at least 1.2 GB of free disk space.

  For production databases, you must estimate the disk space requirement depending on the use you want to make of the database.

• For optimum performance, the file systems that you choose should be on physical devices that are used only by the database.

• The oracle user must have write permissions to create the files in the path you specify.

Guidelines for Placing Oracle Recovery Files on a File System

If you choose to place the Oracle recovery files on a file system, use the following guidelines when deciding where to place them:

• To prevent disk failure from making both the database files and the recovery files unavailable, place the recovery files in a file system on a different physical disk from the database files.

  
  

  Note: Alternatively use an ASM disk group with a normal or high redundancy level for either or both file types.

  
  

• You must choose a shared file system, for example, a cluster file system on a shared disk.

• The file system that you choose should have at least 2 GB of free disk space.

  The disk space requirement is the default disk quota configured for the flash recovery area (specified by the DB_RECOVERY_FILE_DEST_SIZE initialization parameter).

  If you choose the Custom installation type or the Advanced database configuration option, you can specify a different disk quota value. After you create the database, you can also use Oracle Enterprise Manager Grid Control or Database Control to specify a different value.

  For more information about sizing the flash recovery area, see the Oracle Backup and Recovery Basics manual.

• The default path suggested by the Installer for the database file directory is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• The oracle user must have write permissions to create the files in the path you specify.

Creating Required Directories

To create directories for the Oracle CRS, database, or recovery files on separate file systems to the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   
   

   Note: Make sure that the file systems are configured to mount automatically when a node reboots.

   
   

2. Use the df -k command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems that you want to use:

   File Type	File System Requirements
   CRS files	Choose a file system with at least 120 MB of free disk space
   Datafiles	Choose either: A single file system with at least 1.2 GB of free disk space Two or more file systems with at least 1.2 GB of free disk space in total
   Recovery files	Choose a file system with at least 2 GB of free disk space.

   
   

   If you are using the same file system for more than one type of file, add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. Enter commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on them:

   • CRS file directory:

     # mkdir /mount_point/oracrs
     # chown oracle:oinstall /mount_point/oracrs
     # chmod 775 /mount_point/oracrs
     
     

   • Database file directory:

     # mkdir /mount_point/oradata
     # chown oracle:oinstall /mount_point/oradata
     # chmod 775 /mount_point/oradata
     
     

   • Recovery file directory (flash recovery area):

     # mkdir /mount_point/flash_recovery_area
     # chown oracle:oinstall /mount_point/flash_recovery_area
     # chmod 775 /mount_point/flash_recovery_area
     
     

6. If you also want to use ASM or raw devices for storage, see one of the following sections:

   • Configure Disks for Automatic Storage Management

   • Configure Raw Logical Volumes

   Otherwise see the "Verifying the Cluster Software Configuration" section.

Identifying Storage Requirements for ASM

To identify the storage requirements for using ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use ASM for Oracle database files (datafiles), recovery files, or both.

   
   

   Note: You do not have to use the same storage mechanism for database files and recovery files. One can use the file system, while the other uses ASM. If you choose to enable automated backups and you do not have a cluster file system available, you must choose ASM for recovery file storage.

   
   

   If you enable automated backups during the installation, you can choose ASM as the storage mechanism for recovery files by specifying an ASM disk group for the flash recovery area. Depending how you choose to create a database during the installation, you have the following options:

   • If you select an installation method that runs DBCA in interactive mode, by choosing the Advanced database configuration option for example, you can decide whether you want to use the same ASM disk group for datafiles and recovery files, or you can choose to use different disk groups for each file type.

     The same choice is available to you if you use DBCA after the installation to create a database.

   • If you select an installation type that runs DBCA in non-interactive mode, you must use the same ASM disk group for database files and recovery files.

2. Choose the ASM redundancy level that you want to use for the ASM disk group.

   The redundancy level that you choose for the ASM disk group determines how ASM mirrors files in the disk group and determines the number of disks and amount of disk space that you require, as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     Because ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use only RAID or similar devices that provide their own data protection mechanisms as disk devices in this type of disk group.

   • Normal redundancy

     In a normal redundancy disk group, ASM uses two-way mirroring by default, to increase performance and reliability. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For most installations, Oracle recommends that you use normal redundancy disk groups.

   • High redundancy

     In a high redundancy disk group, ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     While high redundancy disk groups do provide a high level of data protection, you must consider the higher cost of additional storage devices before deciding to use this redundancy level.

3. Determine the total amount of disk space that you require for the datafiles and recovery files.

   Use the following table to determine the minimum number of disks and the minimum disk space requirements for the installation:

   Redundancy Level	Minimum Number of Disks	Datafiles	Recovery FIles	Both File Types
   External	1	1.15 GB	2.3 GB	3.45 GB
   Normal	2	2.3 GB	4.6 GB	6.9 GB
   High	3	3.45 GB	6.9 GB	10.35 GB

   
   

   You must also add additional disk space for the ASM metadata. You can use the following formula to calculate the additional disk space requirements (in MB):

   15 + (2 * number_of_disks) + (126 * number_of_ASM_instances)

   For example, for a four-node RAC installation, using three disks in a high redundancy disk group, you require an additional 525 MB of disk space:

   (15 + (2 * 3) +(126 * 4)) = 525

   If an existing ASM instance exists on the system, you can use an existing disk group to meet these storage requirements. If necessary, you can add disks to an existing disk group during the installation.

   The following section describes how to identify existing disk groups and determine the free disk space that they contain.

4. Optionally identify failure groups for the ASM disk group devices.

   
   

   Note: You need to complete this step only if you intend to use an installation method that runs DBCA in interactive mode, for example, if you intend to choose the Custom installation type or the Advanced database configuration option. Other installation types do not enable you to specify failure groups.

   
   

   If you intend to use a normal or high redundancy disk group, you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To avoid failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   
   

   Note: If you define custom failure groups, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

   
   

5. If you are sure that a suitable disk group does not exist on the system, install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify more than one partition on a single physical disk as a disk group device. ASM expects each disk group device to be on a separate physical disk.

   • Although you can specify a logical volume as a device in an ASM disk group, Oracle does not recommend their use. Logical volume managers can hide the physical disk architecture, preventing ASM from optimizing I/O across the physical devices.

   For information about completing this task, see the "Configuring Disks for ASM" section.

Using an Existing ASM Disk Group

If you want to use ASM as the storage option for either database or recovery files, and an existing ASM disk group exists, you have the following choices, depending on the installation method that you select:

• If you select an installation method that runs DBCA in interactive mode, by choosing the Advanced database configuration option for example, you can decide whether you want to create a new disk group or use an existing one.

  The same choice is available to you if you use DBCA after the installation to create a database.

• If you select an installation type that runs DBCA in non-interactive mode, you must choose an existing disk group for the new database; you cannot create a new disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine whether an existing ASM disk group exists, or to determine whether there is sufficient disk space in a disk group, you can use Oracle Enterprise Manager Database Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine whether an ASM instance is configured on the system:

   # more /etc/oratab
   
   

   If an ASM instance is configured on the system, the oratab file should contain a line similar to the following:

   +ASM:oracle_home_path:N
   
   

   In this example, +ASM is the system identifier (SID) of the ASM instance and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an ASM instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the ASM instance that you want to use.

3. Connect to the ASM instance as the SYS user with SYSDBA privilege and start the instance if necessary:

   # $ORACLE_HOME/bin/sqlplus "SYS/SYS_password as SYSDBA"
   SQL> STARTUP
   
   

4. Enter the following command to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   SQL> SELECT NAME,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;
   
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   
   

   Note: If you are adding devices to an existing disk group, Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

   
   

Configuring Disks for ASM

To configure disks for use with ASM, follow these steps:

1. If necessary, install the shared disks that you intend to use for the ASM disk group and reboot the system.

2. To make sure that the disks are available, enter the following command:

   # /usr/sbin/lsdev -Cc disk
   
   

   The output from this command is similar to the following:

   hdisk0 Available 1A-09-00-8,0  16 Bit LVD SCSI Disk Drive
   hdisk1 Available 1A-09-00-9,0  16 Bit LVD SCSI Disk Drive
   hdisk2 Available 17-08-L       SSA Logical Disk Drive
   
   

3. If a disk is not listed as available, enter the following command to configure the new disks:

   # /usr/sbin/cfgmgr
   
   

4. To determine which disks are configured in a volume group, enter the following command:

   # /usr/sbin/lspv
   
   

   The output from this command is similar to the following:

   hdisk0     0000078752249812   rootvg
   hdisk1     none               none
   
   

   For each disk, this command shows:

   • The disk device name

   • Either the 16 character physical volume identifier (PVID) if the disk has one, or none

   • Either the volume group to which the disk belongs, or none

   The disks that you want to use might have a PVID, but they must not belong to a volume group.

5. If the disks that you want to use for the disk group do not have PVIDs, enter a command similar to the following for each disk that you want to use:

   # /usr/sbin/chdev -l hdiskn -a pv=yes
   
   

6. Enter commands similar to the following to change the owner, group, and permissions on the character raw device file for each disk that you want to add to the disk group:

   # chown oracle:dba /dev/rhdiskn
   # chmod 660 /dev/rhdiskn
   
   

Review Important Information

This section describes how to configure raw logical volumes for Oracle CRS and database file storage. The procedures in this section describe how to create a new volume group that contains the logical volumes required for both types of files.

Before you continue, review the following guidelines which contain important information about using volume groups with this release of Oracle Real Application Clusters:

• You must use concurrent-capable volume groups for Oracle CRS and database files.

• If you specify raw logical volumes for the Oracle Cluster Registry and Oracle CRS voting disk during the Oracle CRS installation, the installation process configures the Oracle CRS startup scripts to activate the volume group that contains these logical volumes when Oracle CRS starts.

  
  

  Note: For this reason, you must not configure the volume group that contains the Oracle CRS files in a HACMP concurrent resource group.

  
  

• The Oracle CRS files require less than 200 MB of disk space. To make efficient use of the disk space in a volume group, Oracle recommends that you use the same volume group for the logical volumes for both the CRS files and the database files.

• If you are upgrading an existing Oracle9i release 2 RAC installation that uses raw logical volumes, you can use the existing SRVM configuration repository logical volume for the OCR and create a new logical volume in the same volume group for the Oracle CRS voting disk. However, you must remove this volume group from the HACMP concurrent resource group that activates it before you install Oracle CRS.

  
  

  See Also: For information about removing a volume group from a concurrent resource group, see the HACMP documentation.

  
  
  
  

  Note: If you are upgrading a database, you must also create a new logical volume for the SYSAUX tablespace. See the "Create Raw Logical Volumes in the New Volume Group" section for more information about the requirements for the Oracle CRS voting disk and SYSAUX logical volumes.

  
  

• You must use a HACMP concurrent resource group to activate new or existing volume groups that contain only database files (not Oracle CRS files).

  
  

  See Also: For information about adding a volume group to a new or existing concurrent resource group, see the HACMP documentation.

  
  

• All volume groups that you intend to use for Oracle CRS or database files must be activated in concurrent mode before you start the installation.

• The procedures in this section describe how to create basic volumes groups and volumes. If you want to configure more complex volumes, using mirroring for example, use this section in conjunction with the HACMP documentation.

Create a Volume Group

To create a volume group for the Oracle database files and the Oracle CRS files, follow these steps:

1. If necessary, install the shared disks that you intend to use.

2. To make sure that the disks are available, enter the following command:

   # /usr/sbin/lsdev -Cc disk
   
   

   The output from this command is similar to the following:

   hdisk0 Available 1A-09-00-8,0  16 Bit LVD SCSI Disk Drive
   hdisk1 Available 1A-09-00-9,0  16 Bit LVD SCSI Disk Drive
   hdisk2 Available 17-08-L       SSA Logical Disk Drive
   
   

3. If a disk is not listed as available, enter the following command to configure it:

   # /usr/sbin/cfgmgr
   
   

4. To determine whether the disks are configured in a volume group, enter the following command:

   # /usr/sbin/lspv
   
   

   The output from this command is similar to the following:

   hdisk0     0000078752249812   rootvg
   hdisk1     none               none
   hdisk4     00034b6fd4ac1d71   ccvg1
   
   

   For each disk, this command shows:

   • The disk device name

   • Either the 16 character physical volume identifier (PVID) if the disk has one, or none

   • Either the volume group to which the disk belongs, or none

   The disks that you want to use might have a PVID, but they must not belong to existing volume groups.

5. If the disks that you want to use for the volume group do not have PVIDs, enter a command similar to the following for each disk that you want to use:

   # /usr/sbin/chdev -l hdiskn -a pv=yes
   
   

6. To identify used device major numbers, enter the following command on each node of the cluster:

   # ls -la /dev | more
   
   

   This command displays information about all configured devices, similar to the following:

   crw-rw----   1 root     system    45,  0 Jul 19 11:56 vg1
   
   

   In this example, 45 is the major number of the vg1 volume group device.

7. Identify an appropriate major number that is unused on all nodes in the cluster.

8. To create a volume group, enter a command similar to the following, or use SMIT (smit mkvg):

   # /usr/sbin/mkvg -y VGname -B -s PPsize -V majornum -n \
   -C PhysicalVolumes
   
   

   The following table describes the options and variables used in this example. See the mkvg man page for more information about these options.

   Command Option	SMIT Field	Sample Value and Description
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name you specify could be a generic name, as shown, or for a database volume group, it could specify the name of the database that you intend to create.
   -B	Create a big VG format Volume Group	Specify this option to create a big VG format volume group. Note: If you are using SMIT, choose yes for this field.
   -s PPsize	Physical partition SIZE in megabytes	32 Specify the size of the physical partitions for the database. The sample value shown enables you to include a disk up to 32 GB in size (32 MB * 1016).
   -V Majornum	Volume Group MAJOR NUMBER	46 Specify the device major number for the volume group that you identified in Step 7.
   -n	Activate volume group AUTOMATICALLY at system restart	Specify this option to prevent the volume group from being activated at system restart. Note: If you are using SMIT, choose no for this field.
   -C	Create VG Concurrent Capable	Specify this option to create a concurrent capable volume group. Note: If you are using SMIT, choose yes for this field.
   PhysicalVolumes	PHYSICAL VOLUME names	hdisk3 hdisk4 Specify the device names of the disks that you want to add to the volume group.

   
   

9. Enter a command similar to the following to vary on the volume group that you created:

   # /usr/sbin/varyonvg VGname
   

Create Raw Logical Volumes in the New Volume Group

To create the required raw logical volumes in the new volume group, follow these steps:

1. Choose a name for the database that you want to create.

   The name that you choose must start with a letter and have no more than four characters, for example, orcl.

2. From the following table, identify the logical volumes that you must create:

   Number	Partition Size (MB)	Purpose and Sample Logical Volume Name
   Raw Logical Volumes for Oracle Database
   1	500	SYSTEM tablespace: dbname_system_raw_500m
   1	300 + (Number of instances * 250)	SYSAUX tablespace: dbname_sysaux_raw_800m
   Number of instances	500	UNDOTBSn tablespace (One tablespace for each instance, where n is the number of the instance): dbname_undotbsn_raw_500m
   1	160	EXAMPLE tablespace: dbname_example_raw_160m
   1	120	USERS tablespace: dbname_users_raw_120m
   2 * number of instances	120	Two online redo log files for each instance (where n is the number of the instance and m is the log number, 1 or 2): dbname_redon_m_raw_120m
   2	110	First and second control files: dbname_control[1|2]_raw_110m
   1	250	TEMP tablespace: dbname_temp_raw_250m
   1	5	Server parameter file (SPFILE): dbname_spfile_raw_5m
   1	5	Password file: dbname_pwdfile_raw_5m
   Raw Logical Volumes for Oracle Cluster Ready Services (CRS)
   1	100	Oracle Cluster Registry: ora_ocr_raw_100m Note: You need to create this raw logical volume only once on the cluster. If you create more than one database on the cluster, they all share the same Oracle cluster registry. If you are upgrading from Oracle9i Release 2, you can continue to use the raw device that you used for the SRVM configuration repository instead of creating this new logical volume.
   1	20	Oracle CRS voting disk: ora_vote_raw_20m Note: You need to create this raw logical volume only once on the cluster. If you create more than one database on the cluster, they all share the same Oracle CRS voting disk.

   
   

3. To create each required logical volume, enter a command similar to the following (or use smit mklv):

   # /usr/sbin/mklv -y LVname -T O -w n -s n -r n VGname NumPPs
   
   

   In this example:

   • LVname is the name of the logical volume that you want to create

   • The -T O option specifies that the device subtype should be z, which causes Oracle to use a zero offset when accessing this raw logical volume.

   • VGname is the name of the volume group where you want to create the logical volume

   • NumPPs is the number of physical partitions to use

     To determine the value to use for NumPPs, divide the required size of the logical volume by the size of the physical partition and round the value up to an integer. For example, if the size of the physical partition is 32 MB and you want to create a 500 MB logical volume, you should specify 16 for the NumPPs (500/32 = 15.625).

   The following example shows the command used to create a logical volume for the SYSAUX tablespace of the test database in the oracle_vg1 volume group with a physical partition size of 32 MB (800/32 = 25):

   # /usr/sbin/mklv -y test_sysaux_800m -T O -w n -s n -r n oracle_vg1 25
   
   

4. Change the owner, group, and permissions on the character device files associated with the logical volumes that you created, as follows:

   
   

   Note: The device file associated with the Oracle Cluster Registry must be owned by root. All other device files must be owned by the Oracle software owner user (oracle).

   
   

   • Oracle Database files:

     # chown oracle:dba /dev/rdbname*
     # chmod 660 /dev/rdbname*
     
     

   • Oracle CRS files:

     # chown oracle:dba /dev/rora_vote_raw_20m
     # chmod 660 /dev/rora_vote_raw_20m
     # chown root:dba /dev/rora_ocr_raw_100m
     # chmod 640 /dev/rora_ocr_raw_100m
     
     

Import the Volume Group on the Other Cluster Nodes

To make the volume group available to all nodes in the cluster, you must import it on each node, as follows:

1. Because the physical volume names might be different on the other nodes, enter the following command to determine the PVID of the physical volumes used by the volume group:

   # /usr/sbin/lspv
   
   

2. Note the PVIDs of the physical devices used by the volume group.

3. To vary off the volume group that you want to use, enter commands similar to the following on the node where you created it:

   # /usr/sbin/varyoffvg VGname
   
   

4. On each cluster node, complete the following steps:

   1. Enter the following command to determine the physical volume names associated with the PVIDs you noted previously:

      # /usr/sbin/lspv
      
      

   2. On each node of the cluster, enter commands similar to the following to import the volume group definitions:

      # /usr/sbin/importvg -y VGname -V MajorNumber PhysicalVolume
      
      

      In this example, MajorNumber is the device major number for the volume group and PhysicalVolume is the name of one of the physical volumes in the volume group.

      For example, to import the definition of the oracle_vg1 volume group with device major number 45 on the hdisk3 and hdisk4 physical volumes, enter the following command:

      # /usr/sbin/importvg -y oracle_vg1 -V 45 hdisk3
      
      

   3. Change the owner, group, and permissions on the character device files associated with the logical volumes you created, as follows:

      • Oracle Database files:

        # chown oracle:dba /dev/rdbname*
        # chmod 660 /dev/rdbname*
        
        

      • Oracle CRS files:

        # chown oracle:dba /dev/rora_vote_raw_20m
        # chmod 660 /dev/rora_vote_raw_20m
        # chown root:dba /dev/rora_ocr_raw_100m
        # chmod 640 /dev/rora_ocr_raw_100m
        
        

   4. Enter the following command to ensure that the volume group will not be activated by the operating system when the node boots:

      # /usr/sbin/chvg -a n VGname
      

Activate the Volume Group in Concurrent Mode on All Cluster Nodes

To activate the volume group in concurrent mode on all cluster nodes, enter the following command on each node:

# /usr/sbin/varyonvg -c VGname

Create the DBCA Raw Device Mapping File

To enable Database Configuration Assistant (DBCA) to identify the appropriate raw device for each database file, you must create a raw device mapping file, as follows:

1. Set the ORACLE_BASE environment variable to specify the Oracle base directory that you identified or created previously:

   • Bourne, Bash or Korn shell:

     $ ORACLE_BASE=/u01/app/oracle ; export ORACLE_BASE
     
     

   • C shell:

     $ setenv ORACLE_BASE /u01/app/oracle
     
     

2. Create a database file subdirectory under the Oracle base directory and set the appropriate owner, group, and permissions on it:

   # mkdir -p $ORACLE_BASE/oradata/dbname
   # chown -R oracle:oinstall $ORACLE_BASE/oradata
   # chmod -R 775 $ORACLE_BASE/oradata
   
   

   In this example, dbname is the name of the database that you chose previously.

3. Change directory to the $ORACLE_BASE/oradata/dbname directory.

4. Enter the following command to create a text file that you can use to create the raw device mapping file:

   # find /dev -user oracle -name 'r*' -print > dbname_raw.conf
   
   

5. Edit the dbname_raw.conf file in any text editor to create a file similar to the following:

   
   

   Note: The following example shows a sample mapping file for a two-instance RAC cluster.

   
   

   system=/dev/rdbname_system_raw_500m
   sysaux=/dev/rdbname_sysaux_raw_800m
   example=/dev/rdbname_example_raw_160m
   users=/dev/rdbname_users_raw_120m
   temp=/dev/rdbname_temp_raw_250m
   undotbs1=/dev/rdbname_undotbs1_raw_500m
   undotbs2=/dev/rdbname_undotbs2_raw_500m
   redo1_1=/dev/rdbname_redo1_1_raw_120m
   redo1_2=/dev/rdbname_redo1_2_raw_120m
   redo2_1=/dev/rdbname_redo2_1_raw_120m
   redo2_2=/dev/rdbname_redo2_2_raw_120m
   control1=/dev/rdbname_control1_raw_110m
   control2=/dev/rdbname_control2_raw_110m
   spfile=/dev/rdbname_spfile_raw_5m
   pwdfile=/dev/rdbname_pwdfile_raw_5m
   
   

   In this example, dbname is the name of the database.

   Use the following guidelines when creating or editing this file:

   • Each line in the file must have the following format:

     database_object_identifier=logical_volume
     
     

     The logical volume names suggested in this manual include the database object identifier that you must use in this mapping file. For example, in the following logical volume name, redo1_1 is the database object identifier:

     /dev/rac_redo1_1_raw_120m
     
     

   • For a single instance database, the file must specify one automatic undo tablespace data file (undotbs1), and at least two redo log files (redo1_1, redo1_2).

   • For a RAC database, the file must specify one automatic undo tablespace datafile (undotbsn) and two redo log files (redon_1, redon_2) for each instance.

   • Specify at least two control files (control1, control2).

   • To use manual instead of automatic undo management, specify a single RBS tablespace datafile (rbs) instead of the automatic undo management tablespaces.

6. Save the file and note the file name that you specified.

7. When you are configuring the oracle user's environment later in this chapter, set the DBCA_RAW_CONFIG environment variable to specify the full path to this file.