5 Pre-Installation Tasks for Installing RAC on Linux-Based Systems

Network Hardware Requirements

Each node in the cluster must meet the following requirements:

• Each node must have at least two network adapters; one for the public network interface and one for the private network interface (the interconnect).

• The interface names associated with the network adapters for each network must be the same on all nodes.

• For increased reliability, you can configure redundant public and private network adapters for each node.

• For the public network, each network adapter must support TCP/IP.

• For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better recommended).

  
  

  Note: UDP is the default interconnect protocol for RAC and TCP is the interconnect protocol for Oracle CRS.

IP Address Requirements

Before starting the installation, you must identify or obtain the following IP addresses for each node:

• An IP address and an associated host name registered in the domain name service (DNS) for each public network interface

• One unused virtual IP address and an associated virtual host name registered in DNS that you will configure for the primary public network interface

  The virtual IP address must be in the same subnet as the associated public interface. After installation, you can configure clients to use the virtual host name or IP address. If a node fails, its virtual IP address fails over to another node.

• A private IP address and optional host name for each private interface

  Oracle recommends that you use private network IP addresses for these interfaces, for example: 10.*.*.* or 192.168.*.*. You can use the /etc/hosts file on each node to associate private host names with private IP addresses.

For example, if each node has two public and two private interfaces, you might have the following host names and IP addresses on one of the nodes (rac1) and similar host names and IP addresses on the other nodes:

Checking the Network Requirements

To verify that each node meets the requirements, follow these steps:

1. If necessary, install the network adapters for the public and private networks and configure them with either public or private IP addresses.

2. Register the host names and IP addresses for the public network interfaces in DNS.

3. For each node, register one virtual host name and IP address in DNS.

4. For each private interface on every node, add a line similar to the following to the /etc/hosts file on all nodes, specifying the private IP address and associated private host name:

   10.0.0.1     rac1-priv1
   
   

5. To identify the interface name and associated IP address for every network adapter, enter the following command:

   # /sbin/ifconfig
   
   

   From the output, identify the interface name and IP address for all network adapters that you want to specify as public or private network interfaces.

   
   

   Note: When you install Oracle CRS and RAC, you will require this information.

Check for Required Software

Depending on the products that you intend to install, verify that the following software is installed on the system. The procedure following the table describes how to check these requirements.

make-3.79
binutils-2.11.90.0.8-12

gcc-2.96.108.1
openmotif-2.1.30-11

gcc-3.2.3-2
compat-db-4.0.14.5
compat-gcc-7.3-2.96.122
compat-gcc-c++-7.3-2.96.122
compat-libstdc++-7.3-2.96.122
compat-libstdc++-devel-7.3-2.96.122
openmotif21-2.1.30-8
setarch-1.3-1

gcc-3.2.2-38
openmotif-2.2.2-124

MQSeriesClient
MQSeriesServer
MQSeriesRuntime

To ensure that the system meets these requirements, follow these steps:

1. To determine which distribution and version of Linux is installed, enter the following command:

   # cat /etc/issue
   
   

   
   

   Note: Only the distributions and versions listed in the previous table are supported. Do not install the software on other versions of Linux.

2. On SLES 8 systems only, enter the following command to determine whether Service Pack 3 is installed:

   # uname -r
   2.4.21-138-default
   
   

   This command shows the kernel version (2.4.21-138) and type (default). If the kernel version is less than 2.4.21-138, contact your SLES 8 vendor for information about obtaining and installing Service Pack 3.

3. To determine whether the required packages are installed, enter a command similar to the following:

   # rpm -q package_name
   
   

   If a package is not installed, install it from your Linux distribution media.

4. To determine whether OCFS is installed, enter the following command:

   # rpm -qa | grep ocfs
   
   

   If you want to install the Oracle database files on an OCFS file system and the packages are not installed, download them from the following Web site. Follow the instructions listed with the kit to install the packages and configure the file system:

   http://oss.oracle.com/projects/ocfs/
   
   

Check for Required Patches

Depending on the products that you intend to install, verify that the following patches are installed on the system. The procedure following the table describes how to check these requirements.

To ensure that the system meets these requirements, follow these steps:

1. On Red Hat Enterprise Linux 2.1 systems only, follow these steps:

   1. To determine whether kernel update RHSA-2003:195-06 or later is installed, enter the following command:

      # uname -r
      2.4.9-e.25
      
      

      This command shows the kernel version (2.4.9) and errata level (e.25). If the errata level is less than 25, install the latest kernel update. See the following Web site for information about kernel update RHSA-2003:195-06:

      https://rhn.redhat.com/errata/RHSA-2003-195.html
      
      

   2. To determine whether fix RHSA-2003:022-09 is installed, enter the following command:

      # rpm -q glibc
      glibc-2.2.4-31
      
      

      If the version shown is less than 2.2.4-31, see the following Web site for information about fix RHSA-2003:022-09:

      https://rhn.redhat.com/errata/RHSA-2003-022.html
      
      

2. If you require a CSD for WebSphere MQ, see the following Web site for download and installation information:

   http://www.ibm.com/software/integration/mqfamily/support/summary/lin.html
   
   

Creating the Oracle Inventory Group

You must create the Oracle Inventory group if it does not already exist. The following subsections describe how to determine the Oracle Inventory group name, if it exists, and how to create it if necessary.

# more /etc/oraInst.loc

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

# /usr/sbin/groupadd oinstall

Creating the OSDBA Group

You must create an OSDBA group in the following circumstances:

• An OSDBA group does not exist, for example, if this is the first installation of Oracle Database software on the system

• An OSBDA group exists, but you want to give a different group of UNIX users database administrative privileges in a new Oracle installation

If the OSDBA group does not exist or if you require a new OSDBA group, create it by entering the following command, using the group name dba unless a group with that name already exists:

# /usr/sbin/groupadd dba

Creating an OSOPER Group (Optional)

Create an OSOPER group only if you want to identify a group of UNIX users with a limited set of database administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group. If you want to use an OSOPER group, you must create it in the following circumstances:

• If an OSOPER group does not exist, for example, if this is the first installation of Oracle Database software on the system

• If an OSOPER group exists, but you want to give a different group of UNIX users database operator privileges in a new Oracle installation

If you require an new OSOPER group, create it by entering the following command, using the group name oper unless a group with that name already exists:

# /usr/sbin/groupadd oper

Creating the Oracle Software Owner User

You must create an Oracle software owner user in the following circumstances:

• If an Oracle software owner user does not exist, for example, if this is the first installation of Oracle software on the system

• If an Oracle software owner user exists, but you want to use a different UNIX user, with different group membership, to give database administrative privileges to those groups in a new Oracle Database installation

# id oracle

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

# /usr/sbin/usermod -g oinstall -G dba[,oper] oracle

Verifying That the UNIX User nobody Exists

Before installing the software, verify that the UNIX user nobody exists on the system:

1. To determine whether the user exists, enter the following command:

   # id nobody
   
   

   If this command displays information about the nobody user, you do not have to create that user.

2. If the nobody user does not exist, enter the following command to create it:

   # /usr/sbin/useradd nobody
   
   

3. Repeat this procedure on all of the other cluster nodes.

Create Identical Users and Groups on Other Cluster Nodes

The Oracle software owner user and the Oracle Inventory, OSDBA, and OSOPER groups must exist and be identical on all cluster nodes. To create these identical users and groups, you must identify the user ID and group IDs assigned them on the node where you created them, then create the user and groups with the same name and ID on the other cluster nodes.

Configure SSH on All Cluster Nodes

Before you install and use Oracle Real Application clusters, you must configure secure shell (SSH) for the oracle user on all cluster nodes. The Installer uses the ssh and scp commands during installation to run remote commands on and copy files to the other cluster nodes. You must configure SSH so that these commands do not prompt for a password.

To configure SSH, complete the following steps on each cluster node:

1. Log in as the oracle user.

2. If necessary, create the .ssh directory in the oracle user's home directory and set the correct permissions on it:

   $ mkdir ~/.ssh
   $ chmod 755 ~/.ssh
   
   

3. Enter the following commands to generate an RSA key for version 2 of the SSH protocol:

   $ /usr/bin/ssh-keygen -t rsa
   
   

   At the prompts:

   • Accept the default location for the key file

   • Enter and confirm a pass phrase that is different from the oracle user's password

   This command writes the public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file. Never distribute the private key to anyone.

4. Enter the following commands to generate a DSA key for version 2 of the SSH protocol:

   $ /usr/bin/ssh-keygen -t dsa
   
   

   At the prompts:

   • Accept the default location for the key file

   • Enter and confirm a pass phrase that is different from the oracle user's password

   This command writes the public key to the ~/.ssh/id_dsa.pub file and the private key to the ~/.ssh/id_dsa file. Never distribute the private key to anyone.

5. Copy the contents of the ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub files to the ~/.ssh/authorized_keys file on this node and to the same file on all other cluster nodes.

   
   

   Note: The ~/.ssh/authorized_keys file on every node must contain the contents from all of the ~/.ssh/id_rsa.pub and ~/.ssh/id_dsa.pub files that you generated on all cluster nodes.

6. Change the permissions on the ~/.ssh/authorized_keys file on all cluster nodes:

   $ chmod 644 ~/.ssh/authorized_keys
   
   

At this point, if you use ssh to log in to or run a command on another node, you are prompted for the pass phrase that you specified when you created the DSA key.

To enable the Installer to use the ssh and scp commands without being prompted for a pass phrase, follow these steps:

1. On the system where you want to install the software, log in as the oracle user.

2. Enter the following commands:

   $ exec /usr/bin/ssh-agent $SHELL
   $ /usr/bin/ssh-add
   
   

3. At the prompts, enter the pass phrase for each key that you generated.

   If you have configured SSH correctly, you can now use the ssh or scp commands without being prompted for a password or a pass phrase.

4. To test the SSH configuration, enter the following commands from the same terminal session, testing the configuration of each cluster node:

   $ ssh nodename1 date
   $ ssh nodename2 date
       .
       .
   
   

   These commands should display the date set on each node. If any node prompts for a password or pass phrase, verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys.

   
   

   Note: The first time you use SSH to connect to a node from a particular system, you might see a message stating that the authenticity of the host could not be established. Enter yes at the prompt to continue. You should not see this message again when you connect from this system to that node.

5. You must run the Installer from this session or remember to repeat steps 2 and 3 before you start the Installer from a different terminal session.

Configuring Kernel Parameters

Verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. The procedure following the table describes how to verify and set the values.

To view the current value specified for these kernel parameters, and to change them if necessary, follow these steps:

1. Enter the commands shown in the following to view the current values of the kernel parameters:

   
   

   Note: Make a note of the current values and identify any values that you must change.

   Parameter	Command
   semmsl, semmns, semopm, and semmni	# /sbin/sysctl -a | grep sem This command displays the value of the semaphore parameters in the order listed.
   shmall, shmmax, and shmmni	# /sbin/sysctl -a | grep shm
   file-max	# /sbin/sysctl -a | grep file-max
   ip_local_port_range	# /sbin/sysctl -a | grep ip_local_port_range This command displays a range of port numbers.

2. If the value of any kernel parameter is different to the recommended value, complete the following steps:

   1. Using any text editor, create or edit the /etc/sysctl.conf file and add or edit lines similar to the following:

      
      

      Note: Include lines only for the kernel parameter values that you want to change. For the semaphore parameters (kernel.sem), you must specify all four values. However, if any of the current values are larger than the recommended value, specify the larger value.

      kernel.shmall = 2097152
      kernel.shmmax = 2147483648
      kernel.shmmni = 4096
      kernel.sem = 250 32000 100 128
      fs.file-max = 65536
      net.ipv4.ip_local_port_range = 1024 65000
      
      

      By specifying the values in the /etc/sysctl.conf file, they persist when you reboot the system.

   2. Enter the following command to change the current values of the kernel parameters:

      # /sbin/sysctl -p
      
      

      Review the output from this command to verify that the values are correct. If the values are incorrect, edit the /etc/sysctl.conf file, then enter this command again.

   3. On SLES 8 only, enter the following command to cause the system to read the /etc/sysctl.conf file when it reboots:

      # /sbin/chkconfig boot.sysctl on
      
      

3. Repeat this procedure on all other cluster nodes.

Set Shell Limits for the oracle User

To improve the performance of the software, you must increase the following shell limits for the oracle user:

To increase the shell limits:

1. Add the following lines to /etc/security/limits.conf file:

   *               soft    nproc   2047
   *               hard    nproc   16384
   *               soft    nofile  1024
   *               hard    nofile  65536
   
   

2. Add the following line to the /etc/pam.d/login file, if it does not already exist:

   session    required     /lib/security/pam_limits.so
   
   

3. Depending on the oracle user's default shell, make the following changes to the default shell start-up file:

   • For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file (or the /etc/profile.local file on SLES 8 systems):

     if [ $USER = "oracle" ]; then
             if [ $SHELL = "/bin/ksh" ]; then
                   ulimit -p 16384
                   ulimit -n 65536
             else
                   ulimit -u 16384 -n 65536
             fi
     fi
     
     

   • For the C or tcsh shell, add the following lines to the /etc/csh.login file (or the /etc/csh.login.local file on SLES 8 systems):

     if ( $USER == "oracle" ) then
             limit maxproc 16384
             limit descriptors 65536
     endif
     
     

4. Repeat this procedure on all other cluster nodes.

Oracle Base Directory

The Oracle base directory acts as a top-level directory for Oracle software installations. It is analogous to the C:\Oracle directory used for Oracle software installations on Windows systems. On UNIX systems, the Optimal Flexible Architecture (OFA) guidelines recommend that you use a path similar to the following for the Oracle base directory:

/mount_point/app/oracle_sw_owner

In this example:

• mount_point is the mount point directory for the file system that will contain the Oracle software.

  The examples in this guide use /u01 for the mount point directory. However, you could choose another mount point directory, /oracle or /opt/oracle for example.

• oracle_sw_owner is the UNIX user name of the Oracle software owner, for example oracle.

You can use the same Oracle base directory for more than one installation or you can create separate Oracle base directories for different installations. If different UNIX users install Oracle software on the same system, each user must create a separate Oracle base directory. The following example Oracle base directories could all exist on the same system:

/u01/app/oracle
/u01/app/orauser
/opt/oracle/app/oracle

The following sections describe how to identify existing Oracle base directories that might be suitable for your installation and how to create a new Oracle base directory if necessary.

Regardless of whether you create a new Oracle base directory or decide to use an existing one, you must set the ORACLE_BASE environment variable to specify the full path to this directory.

Oracle Inventory Directory

The Oracle Inventory directory (oraInventory) stores an inventory of all software installed on the system. It is required by, and shared by, all Oracle software installations on a single system. The first time you install Oracle software on a system, the Installer prompts you to specify the path to this directory. If you are installing the software on a local file system, Oracle recommends that you choose the following path:

oracle_base/oraInventory

The Installer creates the directory that you specify and sets the correct owner, group, and permissions on it. You do not need to create it.

CRS Home Directory

The CRS home directory is the directory where you choose to install the software for Oracle Cluster Ready Services. You must install CRS in a separate home directory. When you run the Installer, it prompts you to specify the path to this directory, as well as a name that identifies it. The directory that you specify must be a subdirectory of the Oracle base directory. Oracle recommends that you specify a path similar to the following for the CRS home directory:

oracle_base/product/10.1.0/crs

The Installer creates the directory path that you specify under the Oracle base directory. It also sets the correct owner, group, and permissions on it. You do not need to create this directory.

Oracle Home Directory

The Oracle home directory is the directory where you choose to install the software for a particular Oracle product. You must install different Oracle products, or different releases of the same Oracle product, in separate Oracle home directories. When you run the Installer, it prompts you to specify the path to this directory, as well as a name that identifies it. The directory that you specify must be a subdirectory of the Oracle base directory. Oracle recommends that you specify a path similar to the following for the Oracle home directory:

oracle_base/product/10.1.0/db_1

The Installer creates the directory path that you specify under the Oracle base directory. It also sets the correct owner, group, and permissions on it. You do not need to create this directory.

Identifying an Existing Oracle Base Directory

Existing Oracle base directories might not have paths that comply with OFA guidelines. However, if you identify an existing Oracle Inventory directory or existing Oracle home directories, you can usually identify the Oracle base directories, as follows:

• Identify an existing Oracle Inventory directory

  Enter the following command on all cluster nodes to view the contents of the oraInst.loc file:

  # more /etc/oraInst.loc
  
  

  If the oraInst.loc file exists, the output from this command is similar to the following:

  inventory_loc=/u01/app/oracle/oraInventory
  inst_group=oinstall
  
  

  The inventory_loc parameter identifies the Oracle Inventory directory (oraInventory) on that system. The parent directory of the oraInventory directory is typically an Oracle base directory. In the previous example, /u01/app/oracle is an Oracle base directory.

• Identify existing Oracle home directories

  Enter the following command on all cluster nodes to view the contents of the oratab file:

  # more /etc/oratab
  
  

  If the oratab file exists, it contains lines similar to the following:

  *:/u03/app/oracle/product/10.1.0/db_1:N
  *:/opt/orauser/infra_904:N
  *:/oracle/9.2.0:N
  
  

  The directory paths specified on each line identify Oracle home directories. Directory paths that end with the user name of the Oracle software owner that you want to use are valid choices for an Oracle base directory. If you intend to use the oracle user to install the software, you could choose one of the following directories from the previous example:

  /u03/app/oracle
  /oracle
  

  
  

  Note: If possible, choose a directory path similar to the first (/u03/app/oracle). This path complies with the OFA guidelines.

Before deciding to use an existing Oracle base directory for this installation, make sure that it satisfies the following conditions:

• It should not be on the same file system as the operating system.

• It must have an identical path on all cluster nodes or it must be an NFS file system on a certified NAS device.

  If you are not using an NFS file system, create identical Oracle base directories on the other nodes.

• It must have sufficient free disk space on all cluster nodes, as follows:

  Requirement	Free Disk Space
  The Oracle base directory will contain only software files.	Up to 4 GB
  The Oracle base directory will contain both software and database files (not recommended for production databases).	Up to 5 GB

  Use the following command to determine the free disk space on the file system where the Oracle base directory is located:

  # df -k oracle_base_path
  
  

To continue:

• If an Oracle base directory does not exist on the system or if you want to create a new Oracle base directory, see the "Creating a New Oracle Base Directory" section.

• If an Oracle base directory exists and you want to use it, see the "Configure Disk Storage for Oracle CRS and Database Files" section.

  When you are configuring the oracle user's environment later in this chapter, set the ORACLE_BASE environment variable to specify the directory you chose.

• If an Oracle base directory does not exist on the system or if you want to create a new Oracle base directory, see the following section.

Creating a New Oracle Base Directory

Before you create a new Oracle base directory, you must identify an appropriate file system with sufficient free disk space, as follows:

To identify an appropriate file system, follow these steps:

1. Use the df -k command to determine the free disk space on each mounted file system.

2. From the display, identify a file system that has appropriate free space.

   
   

   Note: The file system can be a local file system or an NFS file system on a certified NAS device. Do not create the Oracle base directory on an OCFS file system.

   The path to the Oracle base directory must be the same on all nodes.

3. Note the name of the mount point directory for the file system that you identified.

To create the Oracle base directory and specify the correct owner, group, and permissions for it, follow these steps:

1. Enter commands similar to the following to create the recommended subdirectories in the mount point directory that you identified and set the appropriate owner, group, and permissions on them:

   # mkdir -p /mount_point/app/oracle_sw_owner
   # chown -R oracle:oinstall /mount_point/app
   # chmod -R 775 /mount_point/app
   
   

   If the mount point you identified is /u01 and oracle is the user name of the Oracle software owner, the recommended Oracle base directory path is as follows:

   /u01/app/oracle
   
   

2. If necessary, repeat the commands listed in the previous step to create the same directory on the other cluster nodes.

3. When you are configuring the oracle user's environment later in this chapter, set the ORACLE_BASE environment variable to specify this directory.

Choosing a Storage Option for Oracle CRS Files

Before you install Oracle Cluster Ready Services, you must choose the storage option that you want to use for the Oracle Cluster Registry (100 MB) and CRS voting disk (20 MB). You cannot use Automatic Storage Management to store these files, because they must be accessible before any Oracle instance starts.

Apart from ASM, you can use any of the storage options listed in the next section for the Oracle CRS files.

Choosing a Storage Option for Oracle Database Files

If you want to create a database during the installation, you must choose one of the following storage options for the database files:

• File system

• Automatic Storage Management

• Raw Devices

• Cluster file system using OCFS for Linux

• Automatic Storage Management

• NFS file system on a certified NAS device

• Raw partitions

http://metalink.oracle.com

Choosing a Storage Option for Oracle Database Recovery Files

Additionally, if you want to enable automated backups during the installation, you must choose one of the following storage options for recovery files (the flash recovery area):

• Cluster file system using OCFS for Linux

• Automatic Storage Management

• NFS file system on a certified NAS device

The storage option that you choose for recovery files can be the same as or different to the option you choose for the database files.

Configuring Disk Storage

For information about how to configure disk storage before you start the installation, see one of the following sections depending on your choice:

• To use a file system for Oracle CRS, database, or recovery file storage, see the "Create Directories for Oracle CRS, Database, or Recovery Files" section.

• To use ASM for database or recovery file storage, see the "Configure Disks for Automatic Storage Management" section.

• To use raw devices (partitions or logical volumes) for Oracle CRS or database file storage, see the "Configure Raw Partitions" section.

Guidelines for Placing Oracle CRS Files on a File System

The Installer does not suggest a default location for the Oracle Cluster Registry (OCR) or the Oracle CRS voting disk. If you choose to create these files on a file system, use the following guidelines when deciding where to place them:

• You must choose a shared file system, for example, a cluster file system on a shared disk or an NFS file system on a certified NAS device.

• It must have at least 100 MB of free disk space for the OCR and 20 MB of free disk space for the CRS voting disk.

• For improved reliability, you should choose a file system on a highly available storage device, for example, a RAID device that implements mirroring.

• If you are placing the Oracle Cluster Ready Services software on a shared file system, you can use the same file system for these files.

• The oracle user must have write permissions to create the files in the path you specify.

Guidelines for Placing Oracle Database Files on a File System

If you choose to place the Oracle database files on a file system, use the following guidelines when deciding where to place them:

• You must choose a shared file system, for example, a cluster file system on a shared disk or an NFS file system on a certified NAS device.

• The default path suggested by the Installer for the database file directory is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• You can choose either a single file system or more than one file system to store the database files:

  • If you want to use a single file system, choose a file system on a physical device that is dedicated to the database.

    For best performance and reliability, choose a RAID device or a logical volume on more than one physical device and implement the stripe-and-mirror-everything (SAME) methodology.

  • If you want to use more than one file system, choose file systems on separate physical devices that are dedicated to the database.

    This method enables you to distribute physical I/O and create separate control files on different devices for increased reliability. It also enables you to fully implement the OFA guidelines. You must choose either the Advanced database creation option or the Custom installation type during the installation to implement this method.

• If you intend to create a preconfigured database during the installation, the file system (or file systems) that you choose must have at least 1.2 GB of free disk space.

  For production databases, you must estimate the disk space requirement depending on the use you want to make of the database.

• For optimum performance, the file systems that you choose should be on physical devices that are used only by the database.

• The oracle user must have write permissions to create the files in the path you specify.

Guidelines for Placing Oracle Recovery Files on a File System

If you choose to place the Oracle recovery files on a file system, use the following guidelines when deciding where to place them:

• To prevent disk failure from making both the database files and the recovery files unavailable, place the recovery files in a file system on a different physical disk from the database files.

  
  

  Note: Alternatively use an ASM disk group with a normal or high redundancy level for either or both file types.

• You must choose a shared file system, for example, a cluster file system on a shared disk or an NFS file system on a certified NAS device.

• The file system that you choose should have at least 2 GB of free disk space.

  The disk space requirement is the default disk quota configured for the flash recovery area (specified by the DB_RECOVERY_FILE_DEST_SIZE initialization parameter).

  If you choose the Custom installation type or the Advanced database configuration option, you can specify a different disk quota value. After you create the database, you can also use Oracle Enterprise Manager Grid Control or Database Control to specify a different value.

  For more information about sizing the flash recovery area, see the Oracle Backup and Recovery Basics manual.

• The default path suggested by the Installer for the database file directory is a subdirectory of the Oracle base directory. You can choose this path only if you are using an Oracle base directory that is on a shared file system.

  This default location is not recommended for production databases.

• The oracle user must have write permissions to create the files in the path you specify.

Creating Required Directories

To create directories for the Oracle CRS, database, or recovery files on separate file systems to the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   
   

   Note: Make sure that the file systems are configured to mount automatically when a node reboots.

2. Use the df -k command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems that you want to use:

   File Type	File System Requirements
   CRS files	Choose a file system with at least 120 MB of free disk space
   Datafiles	Choose either: A single file system with at least 1.2 GB of free disk space Two or more file systems with at least 1.2 GB of free disk space in total
   Recovery files	Choose a file system with at least 2 GB of free disk space.

   If you are using the same file system for more than one type of file, add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. Enter commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on them:

   • CRS file directory:

     # mkdir /mount_point/oracrs
     # chown oracle:oinstall /mount_point/oracrs
     # chmod 775 /mount_point/oracrs
     
     

   • Database file directory:

     # mkdir /mount_point/oradata
     # chown oracle:oinstall /mount_point/oradata
     # chmod 775 /mount_point/oradata
     
     

   • Recovery file directory (flash recovery area):

     # mkdir /mount_point/flash_recovery_area
     # chown oracle:oinstall /mount_point/flash_recovery_area
     # chmod 775 /mount_point/flash_recovery_area
     
     

6. If you also want to use ASM or raw devices for storage, see one of the following sections:

   • Configure Disks for Automatic Storage Management

   • Configure Raw Partitions

   Otherwise see the "Verify that the Required Software is Running" section.

Identifying Storage Requirements for ASM

To identify the storage requirements for using ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use ASM for Oracle database files (datafiles), recovery files, or both.

   
   

   Note: You do not have to use the same storage mechanism for database files and recovery files. One can use the file system, while the other uses ASM. If you choose to enable automated backups and you do not have a cluster file system available, you must choose ASM for recovery file storage.

   If you enable automated backups during the installation, you can choose ASM as the storage mechanism for recovery files by specifying an ASM disk group for the flash recovery area. Depending how you choose to create a database during the installation, you have the following options:

   • If you select an installation method that runs DBCA in interactive mode, by choosing the Advanced database configuration option for example, you can decide whether you want to use the same ASM disk group for datafiles and recovery files, or you can choose to use different disk groups for each file type.

     The same choice is available to you if you use DBCA after the installation to create a database.

   • If you select an installation type that runs DBCA in non-interactive mode, you must use the same ASM disk group for database files and recovery files.

2. Choose the ASM redundancy level that you want to use for the ASM disk group.

   The redundancy level that you choose for the ASM disk group determines how ASM mirrors files in the disk group and determines the number of disks and amount of disk space that you require, as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     Because ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use only RAID or similar devices that provide their own data protection mechanisms as disk devices in this type of disk group.

   • Normal redundancy

     In a normal redundancy disk group, ASM uses two-way mirroring by default, to increase performance and reliability. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For most installations, Oracle recommends that you use normal redundancy disk groups.

   • High redundancy

     In a high redundancy disk group, ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     While high redundancy disk groups do provide a high level of data protection, you must consider the higher cost of additional storage devices before deciding to use this redundancy level.

3. Determine the total amount of disk space that you require for the datafiles and recovery files.

   Use the following table to determine the minimum number of disks and the minimum disk space requirements for the installation:

   Redundancy Level	Minimum Number of Disks	Datafiles	Recovery FIles	Both File Types
   External	1	1.15 GB	2.3 GB	3.45 GB
   Normal	2	2.3 GB	4.6 GB	6.9 GB
   High	3	3.45 GB	6.9 GB	10.35 GB

   You must also add additional disk space for the ASM metadata. You can use the following formula to calculate the additional disk space requirements (in MB):

   15 + (2 * number_of_disks) + (126 * number_of_ASM_instances)

   For example, for a four-node RAC installation, using three disks in a high redundancy disk group, you require an additional 525 MB of disk space:

   (15 + (2 * 3) +(126 * 4)) = 525

   If an existing ASM instance exists on the system, you can use an existing disk group to meet these storage requirements. If necessary, you can add disks to an existing disk group during the installation.

   The following section describes how to identify existing disk groups and determine the free disk space that they contain.

4. Optionally identify failure groups for the ASM disk group devices.

   
   

   Note: You need to complete this step only if you intend to use an installation method that runs DBCA in interactive mode, for example, if you intend to choose the Custom installation type or the Advanced database configuration option. Other installation types do not enable you to specify failure groups.

   If you intend to use a normal or high redundancy disk group, you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To avoid failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   
   

   Note: If you define custom failure groups, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

5. If you are sure that a suitable disk group does not exist on the system, install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify more than one partition on a single physical disk as a disk group device. ASM expects each disk group device to be on a separate physical disk.

   • Although you can specify a logical volume as a device in an ASM disk group, Oracle does not recommend their use. Logical volume managers can hide the physical disk architecture, preventing ASM from optimizing I/O across the physical devices.

   For information about completing this task, see the "Configure Disks for Automatic Storage Management" section.

Using an Existing ASM Disk Group

If you want to use ASM as the storage option for either database or recovery files, and an existing ASM disk group exists, you have the following choices, depending on the installation method that you select:

• If you select an installation method that runs DBCA in interactive mode, by choosing the Advanced database configuration option for example, you can decide whether you want to create a new disk group or use an existing one.

  The same choice is available to you if you use DBCA after the installation to create a database.

• If you select an installation type that runs DBCA in non-interactive mode, you must choose an existing disk group for the new database; you cannot create a new disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine whether an existing ASM disk group exists, or to determine whether there is sufficient disk space in a disk group, you can use Oracle Enterprise Manager Database Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine whether an ASM instance is configured on the system:

   # more /etc/oratab
   
   

   If an ASM instance is configured on the system, the oratab file should contain a line similar to the following:

   +ASM:oracle_home_path:N
   
   

   In this example, +ASM is the system identifier (SID) of the ASM instance and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an ASM instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the ASM instance that you want to use.

3. Connect to the ASM instance as the SYS user with SYSDBA privilege and start the instance if necessary:

   # $ORACLE_HOME/bin/sqlplus "SYS/SYS_password as SYSDBA"
   SQL> STARTUP
   
   

4. Enter the following command to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   SQL> SELECT NAME,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;
   
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   
   

   Note: If you are adding devices to an existing disk group, Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

Configuring Disks for ASM

Oracle provides an ASM library driver that you can use to improve the I/O performance of databases that use ASM for storage management. If you intend to use ASM for database storage, Oracle recommends that you install the ASM library driver and associated utilities and use them to configure the devices that you want to include in an ASM disk group. If you do not use the ASM library driver, you must bind each disk device that you want to use to a raw device. The following sections describe how to configure disks using both methods:

• Configuring Disks for ASM Using the ASM Library Driver

• Configuring Disks for ASM Using Raw Devices

configure

# /etc/init.d/oracleasm configure

enable
disable

# /etc/init.d/oracleasm enable

start
stop
restart

# /etc/init.d/oracleasm restart

createdisk

# /etc/init.d/oracleasm createdisk diskname devicename

deletedisk

# /etc/init.d/oracleasm deletedisk diskname

querydisk

# /etc/init.d/oracleasm querydisk {diskname | devicename}

listdisks

# /etc/init.d/oracleasm listsdisks

scandisks

# /etc/init.d/oracleasm scandisks

Review Important Information

The procedures contained in this section describe how to create raw partitions for Oracle CRS and database file storage. Although Red Hat Enterprise Linux 3 and SLES 8 provide a Logical Volume Manager (LVM), this LVM is not cluster aware. For this reason, Oracle does not support the use of logical volumes with RAC for either CRS or database files.

Create Raw Partitions

To create the required raw partitions, follow these steps:

1. If necessary, install the shared disks that you intend to use for the database and reboot the system.

   
   

   Note: Because the number of partitions that you can create on a single device is limited, you might need to create the required raw partitions on more than one device.

2. To identify the device name for the disks that you want to use for the database, enter the following command:

   # /sbin/fdisk -l
   
   

   Depending on the type of disk, the device name can vary:

   Disk Type	Device Name Format	Description
   IDE disk	/dev/hdxn	In this example, x is a letter that identifies the IDE disk and n is the partition number. For example, /dev/hda is the first disk on the first IDE bus.
   SCSI disk	/dev/sdxn	In this example, x is a letter that identifies the SCSI disk and n is the partition number. For example, /dev/sda is the first disk on the first SCSI bus.
   RAID disk	/dev/rd/cxdypz /dev/ida/cxdypz	Depending on the RAID controller, RAID devices can have different device names. In the examples shown, x is a number that identifies the controller, y is a number that identifies the disk and z is a number that identifies the partition. For example, /dev/ida/c0d1 is the second logical drive on the first controller.

   You can create the required raw partitions either on new devices that you added or on previously partitioned devices that have unpartitioned free space. To identify devices that have unpartitioned free space, examine the start and end cylinder numbers of the existing partitions and determine whether the device contains unused cylinders.

3. To create raw partitions on a device, enter a command similar to the following:

   # /sbin/fdisk devicename
   
   

   Use the following guidelines when creating partitions:

   • Use the p command to list the partition table of the device.

   • Use the n command to create a new partition.

   • After you have created all the required partitions on this device, use the w command to write the modified partition table to the device.

   • See the fdisk man page for more information about creating partitions.

   The following table lists the number, size, and purpose of the partitions that you must create:

   Number	Partition Size (MB)	Purpose
   Raw Partitions for Oracle Database
   1	500	SYSTEM tablespace
   1	300 + (Number of instances * 250)	SYSAUX tablespace
   Number of instances	500	UNDOTBSn tablespace (One tablespace for each instance)
   1	160	EXAMPLE tablespace
   1	120	USERS tablespace
   2 * number of instances	120	Two online redo log files for each instance
   2	110	First and second control files
   1	250	TEMP tablespace
   1	5	Server parameter file (SPFILE)
   1	5	Password file
   Raw Logical Volumes for Oracle Cluster Ready Services (CRS)
   1	100	Oracle Cluster Registry Note: You need to create this raw partition only once on the cluster. If you create more than one database on the cluster, they all share the same Oracle cluster registry. If you are upgrading from Oracle9i Release 2, you can continue to use the raw device that you used for the SRVM configuration repository instead of creating this new raw device.
   1	20	Oracle CRS voting disk Note: You need to create this raw partition only once on the cluster. If you create more than one database on the cluster, they all share the same Oracle CRS voting disk.

Bind the Partitions to Raw Devices

After you have created the required partitions, you must bind the partitions to raw devices. However, you must first determine what raw devices are already bound to other devices. The procedure that you must follow to complete this task varies, depending on the distribution that you are using:

• Red Hat:

  1. To determine what raw devices are already bound to other devices, enter the following command:

     # /usr/sbin/raw -qa
     
     

     Raw devices have device names in the form /dev/raw/rawn, where n is a number that identifies the raw device.

  2. Open the /etc/sysconfig/rawdevices file in any text editor and add a line similar to the following for each partition that you created:

     /dev/raw/raw1 /dev/sdb1
     
     

     Specify an unused raw device for each partition.

  3. For the raw device that you created for the Oracle Cluster Registry, enter commands similar to the following to set the owner, group, and permissions on the device file:

     # chown root:dba /dev/dev/rawn
     # chmod 640 /dev/raw/rawn
     
     

  4. For each other raw device that you specified in the rawdevices file, enter commands similar to the following to set the owner, group, and permissions on the device file:

     # chown oracle:oinstall /dev/dev/rawn
     # chmod 660 /dev/raw/rawn
     
     

  5. To bind the partitions to the raw devices, enter the following command:

     # /sbin/service rawdevices restart
     
     

     By editing the rawdevices file, the system binds the partitions to the raw devices when it reboots.

• SLES 8:

  1. To determine what raw devices are already bound to other devices, enter the following command:

     # /usr/sbin/raw -qa
     
     

     Raw devices have device names in the form /dev/raw/rawn, where n is a number that identifies the raw device.

  2. Open the /etc/raw file in any text editor and add a line similar to the following to associate each partition with an unused raw device:

     raw1:sdb1
     
     

  3. For the raw device that you created for the Oracle Cluster Registry, enter commands similar to the following to set the owner, group, and permissions on the device file:

     # chown root:dba /dev/dev/rawn
     # chmod 640 /dev/raw/rawn
     
     

  4. For each other raw device that you specified in the /etc/raw file, enter commands similar to the following to set the owner, group, and permissions on the device file:

     # chown oracle:dba /dev/raw/rawn
     # chmod 660 /dev/raw/rawn
     
     

  5. To bind the partitions to the raw devices, enter the following command:

     # /etc/init.d/raw
     
     

  6. To ensure that the raw devices are bound when the system reboots, enter the following command:

     # /sbin/chkconfig raw on
     
     

Create the DBCA Raw Device Mapping File

To enable Database Configuration Assistant (DBCA) to identify the appropriate raw partition for each database file, you must create a raw device mapping file, as follows:

1. Set the ORACLE_BASE environment variable to specify the Oracle base directory that you identified or created previously:

   • Bourne, Bash or Korn shell:

     $ ORACLE_BASE=/u01/app/oracle ; export ORACLE_BASE
     
     

   • C shell:

     $ setenv ORACLE_BASE /u01/app/oracle
     
     

2. Create a database file subdirectory under the Oracle base directory and set the appropriate owner, group, and permissions on it:

   # mkdir -p $ORACLE_BASE/oradata/dbname
   # chown -R oracle:oinstall $ORACLE_BASE/oradata
   # chmod -R 775 $ORACLE_BASE/oradata
   
   

   In this example, dbname is the name of the database that you chose previously.

3. Change directory to the $ORACLE_BASE/oradata/dbname directory.

4. Edit the dbname_raw.conf file in any text editor to create a file similar to the following:

   
   

   Note: The following example shows a sample mapping file for a two-instance RAC cluster.

   system=/dev/raw/raw1
   sysaux=/dev/raw/raw2
   example=/dev/raw/raw3
   users=/dev/raw/raw4
   temp=/dev/raw/raw5
   undotbs1=/dev/raw/raw6
   undotbs2=/dev/raw/raw7
   redo1_1=/dev/raw/raw8
   redo1_2=/dev/raw/raw9
   redo2_1=/dev/raw/raw10
   redo2_2=/dev/raw/raw11
   control1=/dev/raw/raw12
   control2=/dev/raw/raw13
   spfile=/dev/raw/raw14
   pwdfile=/dev/raw/raw15
   
   

   Use the following guidelines when creating or editing this file:

   • Each line in the file must have the following format:

     database_object_identifier=raw_device_path
     
     

   • For a single instance database, the file must specify one automatic undo tablespace datafile (undotbs1), and at least two redo log files (redo1_1, redo1_2).

   • For a RAC database, the file must specify one automatic undo tablespace datafile (undotbsn) and two redo log files (redon_1, redon_2) for each instance.

   • Specify at least two control files (control1, control2).

   • To use manual instead of automatic undo management, specify a single RBS tablespace datafile (rbs) instead of the automatic undo management tablespaces.

5. Save the file and note the file name that you specified.

6. When you are configuring the oracle user's environment later in this chapter, set the DBCA_RAW_CONFIG environment variable to specify the full path to this file.

Check the Configuration of the hangcheck-timer Module

The hangcheck-timer module monitors the Linux kernel for extended operating system hangs that could affect the reliability of a RAC node and cause a database corruption. If a hang occurs, the module reboots the node in seconds.

You can use the hangcheck_tick and hangcheck_margin parameters to control the behavior of the module, as follows:

• The hangcheck_tick parameter defines how often, in seconds, the hangcheck-timer checks the node for hangs. The default value is 60 seconds.

• The hangcheck_margin parameter defines how long the timer waits, in seconds, for a response from the kernel. The default value is 180 seconds.

If the kernel fails to respond within the sum of the hangcheck_tick and hangcheck_margin parameter values, the hangcheck-timer module reboots the system. Using the default values, the node would be rebooted if the kernel fails to respond within 240 seconds.

To verify that the hangcheck-timer module is running on every node:

1. Enter the following command on each node to determine which kernel modules are loaded:

   # /sbin/lsmod
   
   

2. If the hangcheck-timer module is not listed for any node, enter a command similar to the following to start the module on that node:

   # /sbin/insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
   
   

3. To ensure that the module is loaded every time the system reboots, verify that the local system startup file contains the command shown in the previous step, or add it if necessary:

   • Red Hat:

     On Red Hat Enterprise Linux systems, add the command to the /etc/rc.d/rc.local file.

   • SLES 8

     On SLES 8 systems, add the command to the /etc/init.d/boot.local file.

Check the Configuration of OCFS

If you intend to use Oracle Cluster File System for database files, verify that it is configured correctly, as follows:

http://oss.oracle.com/projects/ocfs

1. Enter the following command to verify that OCFS is configured to start at runlevels 2, 3, and 5:

   # /usr/sbin/chkconfig --list ocfs
   
   

2. Verify that the OCFS file systems you want to use are mounted and are specified in the /etc/fstab file.