3 Configuring Oracle Clusterware Storage

This chapter describes the Oracle Clusterware storage configuration tasks that you must complete before you start Oracle Universal Installer (OUI). This chapter includes information about the following tasks:

Preliminary Shared Disk Preparation
Reviewing Storage Options for Oracle Clusterware Files
Checking for Available Shared Storage with CVU
Configuring Storage for Oracle Clusterware Files on a Shared File System

3.1 Preliminary Shared Disk Preparation

Complete the following steps to prepare shared disks for storage:

Disabling Write Caching
Enabling Automounting for Windows 2003

3.1.1 Disabling Write Caching

You must disable write caching on all disks that will be used to share data between the nodes in your cluster. Perform these steps to disable write caching:

Click Start, then click Settings, then Control Panel, then Administrative Tools, then Computer Management, then Device Manager, and then Disk drives
Expand the Disk drives and double-click the first drive listed
Under the Disk Properties tab for the selected drive, uncheck the option that enables the write cache.

For Windows 2000, select the option Optimize for quick removal on the Policies tab.
Double-click each of the other drives that will be used by Oracle Clusterware and Oracle RAC and disable write caching as described in the previous step.

Caution:

Any disks that you use to store files, including database files, that will be shared between nodes, must have write caching disabled.

3.1.2 Enabling Automounting for Windows 2003

If you are using Windows 2003, then you must enable disk automounting, depending on the Oracle products that you are installing and on other conditions. You must enable automounting when using:

Raw partitions for Oracle Real Application Clusters (Oracle RAC)
Cluster file system for Oracle RAC
Oracle Clusterware
Raw partitions for single-node database installations
Logical drives for Automatic Storage Management (ASM)

To enable automounting:

Enter the following commands at a command prompt:

c:\> diskpart
DISKPART> automount enable
Automatic mounting of new volumes enabled.

Type exit to end the diskpart session
Repeat steps 1 and 2 for each node in the cluster.
When you have prepared all of the cluster nodes in your Windows 2003 system as described in the previous steps, restart all of the nodes.

Note:

All nodes in the cluster must have automatic mounting enabled in order to correctly install Oracle RAC and Oracle Clusterware. Oracle recommends that you enable automatic mounting before creating any logical partitions for use by the database, ASM, or the Oracle Cluster File System.

You must restart each node after enabling disk automounting. After it is enabled and the node is restarted, automatic mounting remains active until it is disabled.

3.2 Reviewing Storage Options for Oracle Clusterware Files

Use the following sections to help you select your storage option and take the first steps in configuring storage:

Overview of Oracle Clusterware Storage Options
General Storage Considerations
After You Have Selected Disk Storage Options

3.2.1 Overview of Oracle Clusterware Storage Options

There are two ways to store Oracle Clusterware files:

Oracle Cluster File System (OCFS): The cluster file system Oracle provides a storage solution for Windows users.
Raw storage: Raw storage is a logical partition created inside an extended partition that does not yet have a file system configured on it. These unformatted partitions are created and managed by Microsoft Windows disk management tools or by tools provided by third party vendors.

Note:

You cannot use Direct NFS to store Oracle Clusterware files. You can only use Direct NFS to store Oracle Database files. See Chapter 4 for more information on using Direct NFS.

3.2.2 General Storage Considerations

Oracle Clusterware files include voting disks, used to monitor cluster node status, and Oracle Cluster Registry (OCR) which contains configuration information about the cluster. The voting disks and OCR are shared files on a cluster file system. If you do not use a cluster file system, then you must place these files on shared raw devices.

The OCR is a file that contains the configuration information and status of the cluster. Oracle Universal Installer (OUI) automatically initializes the OCR during the Oracle Clusterware installation. Database Configuration Assistant uses the OCR for storing the configurations for the cluster databases that it creates.

For either storage option to meet high availability requirements, the files stored on the disk need to be protected by data redundancy, so that if one or more disks fail, the data stored on the failed disks can be recovered. This redundancy can be provided externally using Redundant Array of Independent Disks (RAID) devices, or logical volumes on more than one physical device and implement the stripe-and-mirror- everything methodology, also known as SAME. If you do not have a RAID devices or logical volumes, you can create additional copies, or mirrors, of the files on different file systems. If you choose to mirror the files, you must provide disk space for an additional Oracle Cluster Registry (OCR) file and at least two additional voting disk files.

The OCR mirror should be placed on a different disk than the primary OCR file. For voting disk file placement, ensure that each file is configured so that it does not share any hardware device or disk, or other single point of failure with the other voting disks. Any node that does not have available to it an absolute majority of voting disks configured (more than half) will be restarted.

Note:

For the most up-to-date information about supported storage options for Oracle Clusterware installations, refer to the Certify pages on the OracleMetaLink Web site:

https://metalink.oracle.com

The following table shows the storage options supported for storing Oracle Clusterware files. Oracle Clusterware files:

Storage Option	File Types Supported
Storage Option	OCR and Voting Disks	Oracle CRS Software
Automatic Storage Management	No	No
OCFS	Yes	Yes
Shared disk partitions	Yes	No
Direct NFS	No	No
Local storage	No	Yes

Use the following guidelines when choosing the storage options that you want to use for Oracle Clusterware:

You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.
You cannot use ASM to store Oracle Clusterware files, because these files must be accessible before any Oracle instance starts, which includes the ASM instances.
If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy.

Note:
Do not create an NTFS partition on a disk that you are using for OCFS.

3.2.3 After You Have Selected Disk Storage Options

When you have determined your disk storage options, you must perform the following tasks in the following order:

1: Check for available storage with CVU

Refer to "Checking for Available Shared Storage with CVU"

2: Configure shared storage for Oracle Clusterware files

To use OCFS for Oracle Clusterware files, refer to "Configuring Storage for Oracle Clusterware Files on a Shared File System".
To use raw devices (partitions) for Oracle Clusterware files, refer to "Configuring Storage for Oracle Clusterware Files on Raw Devices".

3.3 Checking for Available Shared Storage with CVU

To check for all shared file systems available across all nodes on the cluster, log in as the installation owner user (oracle or crs), and use the following syntax:

mountpoint\runcluvfy.bat comp ssa -n node_list

If you want to check the shared accessibility of a specific shared storage type to specific nodes in your cluster, then use the following command syntax:

mountpoint\runcluvfy.bat comp ssa -n node_list -s storageID_list

In the preceding syntax examples, the variable mountpoint is the mountpoint path of the installation media, the variable node_list is the list of nodes you want to check, separated by commas, and the variable storageID_list is the paths for the storage devices that you want to check.

For example, if you want to check the shared accessibility from node1 and node2 of storage devices \\host1\ocr_dir and \\host2\voting_disks, and your mountpoint is F:, then enter the following command:

F:\runcluvfy.bat comp ssa -n node1,node2 -s \\host1\ocr_dir,\\host2\voting_disks

If you do not specify storage device IDs in the command, then CVU searches for all available storage devices connected to the nodes on the list.

Note:

Prior to installing Oracle Clusterware, use the runcluvfy utility on the installation media. If Oracle Clusterware is installed, use the cluvfy utility instead.

3.4 Configuring Storage for Oracle Clusterware Files on a Shared File System

Requirements for Using a File System for Oracle Clusterware Files
Creating Required Partitions for Oracle Clusterware on Shared File Systems

Note:

The OCR is a shared file in a cluster file system environment. If you do not use a cluster file system, then you must place this file on a shared storage device.

3.4.1 Requirements for Using a File System for Oracle Clusterware Files

To use OCFS for Oracle Clusterware files, you must comply with the following requirements:

If you choose to place your Oracle Cluster Registry (OCR) files on a shared file system, then Oracle recommends that one of the following is true:
- The disks used for the file system are on a highly available storage device, (for example, a RAID device that implements file redundancy)
- At least three file systems are mounted, and use the features of Oracle Clusterware 11g release 1 (11.1) to provide redundancy for the OCR and voting disks
If you use a RAID device to store the Oracle Clusterware files, you must have a partition that has at least 560 MB of available space for the OCR and voting disk.
If you use the redundancy features of Oracle Clusterware to provide high availability for the OCR and voting disk files, then:
- One file system must have 560 MB of available space for the primary OCR and a voting disk.
- A second file system must have 560 MB of available space for the OCR mirror and a voting disk.
- A third file system must have at least 500 MB for a voting disk (500 MB is the smallest disk size that can be used by OCFS).
If you intend to store Oracle Clusterware binaries on OCFS, then you must have at least 650 MB of available space on one of the file systems.

Note:

Oracle does not recommend the use of a shared Oracle Clusterware home directory because it creates a single point-of-failure, and removes the ability to perform rolling upgrades of Oracle Clusterware.

The total required volume size listed in the previous paragraph is cumulative. For example, to store all OCR and voting disk files on a shared file system that does not provide redundancy at the hardware level (external redundancy), you should have at least 1.4 GB of storage available over a minimum of three volumes (two separate volume locations for the OCR and OCR mirror, and one voting disk on each volume). If you use a file system that provides data redundancy, then you need only one physical disk with 560 MB of available space to store the OCR and voting disk files.

Note:

If you are upgrading from a previous release of Oracle Clusterware, and the existing OCR and voting disk files are not 280 MB in size, you do not need to change the size of the OCR or voting disks before performing the upgrade.

3.4.2 Creating Required Partitions for Oracle Clusterware on Shared File Systems

The OCR and voting disk files are stored in the OCFS datafile directory (datafile_disk\cdata\clustername) where datafile_disk is the OCFS partition and clustername is the name of your cluster.

Oracle Universal Installer (OUI) does not suggest a default location for the Oracle Cluster Registry (OCR) or the Oracle Clusterware voting disk. If you choose to create these files on a file system, then perform the steps described in this section to set up the unformatted partitions on the shared disks for OCFS. Windows refers to unformatted partitions as logical drives. If you need more information about creating partitions, then refer to the Windows online help from within the disk administration tools.

Run Windows Disk Management from one node to create an extended partition. Use a basic disk: dynamic disks are not supported.
If you plan to store the Oracle Clusterware binaries on OCFS, create at least two logical partitions: one for the Oracle Clusterware home and one for the Oracle Clusterware files.

You do not need to create separate partitions for the OCR and voting disk if you plan to use OCFS. OCFS creates individual files for the OCR and voting disk.

The number of partitions used for OCFS affects performance. Therefore, you should create the minimum number of partitions needed for the OCFS option you choose.
If your file system is not based on a RAID system, then you must provide storage space for creating redundant Oracle Clusterware files. Create an extended partition and logical partition for each additional file system or disk drive that will be used by Oracle Clusterware. The additional storage used for Oracle Clusterware files should not share any hardware device or disk to avoid single points of failure.

To create the required partitions, perform the following steps:

From one of the existing nodes of the cluster, run the Windows disk administration tool as follows:
- Click Start, then select Settings, Control Panel, Administrative Tools, and then Computer Management
- Expand the Storage folder to Disk Management. Use a basic disk with a Master Boot Record (MBR) partition style as an extended partition for creating partitions.
Right click inside an unallocated part of an extended partition and choose Create Logical Drive. A wizard presents pages for configuring the logical drive. Select the select logical drive radio button and click Next.
Enter the size that you want for the partition and click Next.
Choose the option "Do not assign a drive letter or path", click Next, and then choose the option "Do not format this partition". Click Finish on the last page of the wizard.
Repeat steps 2 through 4 for the second partition and any additional partitions. An optimal configuration is one partition for the Oracle home and one partition for Oracle Clusterware files, or a single partition for both the OCR and voting disk.
If you are preparing drives on a Windows 2003 system, then you should restart all nodes in the cluster after you have created the logical drives.
Check all nodes in the cluster to ensure that the partitions are visible on all the nodes and to ensure that none of the Oracle partitions have drive letters assigned. If any partitions have drive letters assigned, then remove them by performing these steps:
- Right-click the partition in the Windows disk administration tool
- Select "Change Drive Letters and Paths..." from the menu
- Click Remove in the "Change Drive Letter and Paths" window

3.5 Configuring Storage for Oracle Clusterware Files on Raw Devices

To use raw devices, you must create a separate partition for each file. If you are using three raw devices to store the Oracle Clusterware files, then:

Create two partitions on the first raw device: one 280 MB partition for the voting disk and one 280 MB partition for the OCR.
Create two partitions on the second raw device: one 280 MB partition for the voting disk and one 280 MB partition for the OCR mirror.
Create a single partition on the third raw device for the voting file.

If you are not using OCFS or ASM for your datafiles, then you must also create raw partitions for your database files as described in "Configuring Logical Volumes or Unformatted Partitions".

To create and configure raw volumes or partitions, use the disk administration tools provided by the operating system or third party vendors. The following administration tools are provided by the operating system:

Disk Management snap-in to manage disks.

To access this tool, type diskmgmt.msc at the command prompt. Alternatively, from the Start menu, select Programs, then Administrative Tools, then Computer Management. Then select the Disk Management node in the Storage tree.
A command line tool to manage disks.

To access this tool, type diskpart.exe at the command prompt.
Note:
If you need to download the diskmgmt.msc tool, consult Microsoft documentation on the Microsoft Web site
```
http://www.microsoft.com/
```
See Also:
The online help or documentation for the administration tool you are using

You can use the diskpart tool command create partition to create primary or extended partitions, or create logical drives. The following example uses the diskpart tool to create a 280 MB extended partition on disk 100. In this syntax, diskpart.exe is the command line tool for managing disks.

c:\> diskpart.exe
DISKPART> select disk 100
DISKPART> create partition extended size=280

Optionally, mount the partition on an NTFS folder instead of assigning a drive letter:

DISKPART> assign mount=C:\mnt\raw_1

Note:

Be aware of the following restrictions for partitions:

You cannot use primary partitions for storing Oracle Clusterware files while running the OUI to install Oracle Clusterware as described in Chapter 5, "Installing Oracle Clusterware". You must create logical drives inside extended partitions for the disks to be used by Oracle Clusterware files and Oracle ASM.
With 32-bit Windows, you cannot create more than four primary disk partitions for each disk. One of the primary partitions can be an extend partition, which can then be subdivided into multiple logical partitions.
With 64-bit Windows, you can create up to 128 primary partitions for each disk.
You can assign mount points only to primary partitions and logical drives.
You must create logical drives inside extended partitions for the disks to be used by Oracle Clusterware files and Oracle ASM.
Oracle recommends that you limit the number of partitions you create on a single disk to prevent disk contention. Therefore, you may prefer to use extended partitions rather than primary partitions.

For these reasons, you might prefer to use extended partitions for storing Oracle software files and not primary partitions.