|
Oracle® Application Server 10g High Availability Guide
10g (9.0.4) Part No. B10495-02 |
|
|
|
|
Oracle® Application Server 10g High Availability Guide
10g (9.0.4) Part No. B10495-02 |
|
|
|
|
Disaster recovery refers to how a system recovers from catastrophic site failures caused by natural or unnatural disasters. Examples of catastrophic failures include earthquakes, tornadoes, floods, or fire. Additionally, disaster recovery can also refer to how a system is managed for planned outages. For most disaster recovery situations, the solution involves replicating an entire site, not just pieces of hardware or subcomponents. This also applies to the Oracle Application Server Disaster Recovery (OracleAS Disaster Recovery) solution.
This chapter describes the OracleAS Disaster Recovery solution, how to configure and set up its environment, and how to manage the solution for high availability. The discussion involves both OracleAS middle and Infrastructure tiers in two sites: production and standby. The standby site is configured identically to the production site. Under normal operation, the production site actively services requests. The standby site is maintained to mirror the applications and content hosted by the production site.
The sites are synchronized using the Oracle Application Server Backup and Recovery Tool (for configuration files in the file system) and Oracle Data Guard (for the Infrastructure database). The following table provides a summary of the OracleAS Disaster Recovery strategy:
Table 6-1 Overview of OracleAS Disaster Recovery strategy
In addition to the recovery strategies, configuration and installation of both sites are discussed. For these tasks, two different ways of naming the middle tier nodes are covered as well as two ways of resolving hostnames intra-site and inter-site.
With OracleAS Disaster Recovery, planned outages of the production site can be performed without interruption of service by switching over to the standby site. Unplanned outages are managed by failing over to the standby site. Procedures for switchover and failover are covered in this chapter.
This chapter is organized into the following main sections:
|
See Also: Oracle Application Server 10g Installation Guide for instructions on how to install the OracleAS Disaster Recovery solution. |
The Oracle Application Server Disaster Recovery solution consists of two identically configured sites - one primary (production/active) and one secondary (standby). Both sites have the same number of middle tier and Infrastructure nodes and the same number and types of components installed. In other words, the installations on both sites, middle tier and Infrastructure are identical. Both sites are usually dispersed geographically, and if so, they are connected via a wide area network.
This section describes the overall layout of the solution, the major components involved, and the configuration of these components. It has the following sections:
Before describing and detailing the OracleAS Disaster Recovery solution, several terms used in this chapter require clear definition in order for the concepts described in this chapter to be understood properly.
|
Note: The definitions below apply specifically to OracleAS Disaster Recovery. They may have a varying definition outside this context. |
For the purpose of discussion in this chapter, a differentiation is made between the terms physical hostname and logical hostname. Physical hostname is used to refer to the "internal name" of the current machine. In UNIX, this is the name returned by the command hostname.
Physical hostname is used by Oracle Application Server 10g components that are installed on the current machine to reference the local host. During the installation of these components, the installer retrieves the physical hostname from the current machine and stores it in Oracle Application Server 10g configuration metadata on disk.
Logical hostname is a name assigned to an IP address either through the /etc/hosts file (in UNIX), C:\WINDOWS\system32\drivers\etc\hosts file (in Windows), or through DNS resolution. This name is visible on the network that the host to which it refers to is connected. Often, the logical hostname and physical hostname are literally identical. However, their usage in the OracleAS Disaster Recovery solution necessitates them to be clearly distinct.
Virtual hostname is used to refer to the name for the Infrastructure host that is specified in the Specify High Availibility screen of the OracleAS installer. The virtual hostname is used by the middle tier and Infrastructure components to access the Infrastructure regardless of whether the Infrastructure is a single node installation or part of the OracleAS Cold Failover Cluster solution. Virtual hostname, as used in this chapter, applies only to the Infrastructure host(s).
To ensure that your implementation of the OracleAS Disaster Recovery solution performs as designed, the following requirements need to be adhered to:
On each host in the standby site, make sure the following is identical to its equivalent peer in the production site:
For the middle tier hosts, physical hostnames
Virtual hostname for the Infrastructure. The virtual hostname can be specified in the Specify High Availibility screen presented by the installer.
Hardware platform.
Operating system release and patch levels.
All installations conform to the requirements listed in the Oracle Application Server 10g Installation Guide to install Oracle Application Server.
Oracle Application Server software is installed in identical directory paths between each host in the production site and its equivalent peer in the standby site.
Username and password of the user who installed Oracle Application Server must be the same between a host in the production site and its peer in the standby site.
Numerical user ID of the user who installed Oracle Application Server on that particular node
Group name of the user who installed Oracle Application Server on that particular node
Numerical group ID of the group of the user who installed Oracle Application Server on that particular node
Environment profile
Shell (command line environment)
Directory structure and path of the Oracle home for each OracleAS installation on a node. Do not use symbolic links anywhere in the path.
OracleAS installation types:
Figure 6-1 depicts the topology of the OracleAS Disaster Recovery solution.
Figure 6-1 Oracle Application Server 10g site-to-site disaster recovery solution (load balancer appliance is optional if only one middle tier node is present)
The procedures and steps for configuring and operating the OracleAS Disaster Recovery solution support 1 to n number of middle tier installations in the production site. The same number of middle tier installations must exist in the standby site. The middle tiers must mirror each other in the production and standby sites.
For the Infrastructure, a uniform number of installations is not required between the production and standby sites. For example, the Oracle Application Server Cold Failover Clusters solution can be deployed in the production site, and a single node installation of the Infrastructure can be deployed in the standby site. This way, the production site’s Infrastructure has protection from host failure using an OracleAS Cold Failover Cluster. Refer to the section "Oracle Application Server Cold Failover Clusters" in Chapter 3 for more information on OracleAS Cold Failover Clusters.
The following are important characteristics of the OracleAS Disaster Recovery solution:
Middle tier installations are identical between the production and standby sites. In other words, each middle tier installation in the production site has an identically equivalent installation in the standby site. More than one middle tier node is recommended because this enables each set of middle tier installations on each site to be redundant. Being on multiple machines, problems and outages within a site of middle tier installations are transparent to clients.
The OracleAS Disaster Recovery solution is restricted to identical site configuration to ensure that processes and procedures are kept the same between sites, making operational tasks easier to maintain and execute. Identical site configuration also allows for a higher success rate for manually maintaining the synchronization of Oracle Application Server 10g component configuration files between sites.
When the production site becomes unavailable due to a disaster, the standby site can become operational within a reasonable time. Client requests are always routed to the site that is operating in the production role. After a failover or switchover operation occurs due to an outage, client requests are routed to another site that assumes the production role. The quality of service offered by the new production site should be the same as that offered by the original production site before the outage.
The sites are set up in active-passive configuration. An active-passive setup has one primary site used for production and one secondary site that is initially passive (on standby). The secondary site is made active only after a failover or switchover is made to it. Since the sites are symmetrical, after failover or switchover, the original standby site can be kept active as the new production site. After repairing or upgrading the original production site, it can be made into the new standby site. Either site should offer the same level of service to clients as the other.
The site playing the standby role contains a physical standby of the Oracle Application Server Infrastructure managed by Oracle Data Guard. Oracle Data Guard together with procedures for backing up and restoring Infrastructure configuration files provide configuration synchronization between the production and standby sites. Switchover and failover procedures allow the roles to be traded between the Infrastructures in the two sites. Refer to the section "Setting Up Oracle Data Guard" for instructions on how to set up Oracle Data Guard to work in the OracleAS Disaster Recovery solution.
Prior to the the installation of OracleAS software for the OracleAS Disaster Recovery solution, a number of system level configurations are required. The tasks that accomplish these configurations are:
This section covers the steps needed to perform these tasks.
Before performing the steps to set up the physical and logical hostnames, plan the physical and logical hostnames you wish to use with respect to the entire OracleAS Disaster Recovery solution. The overall approach to planning and assigning hostnames is to meet the following goals:
OracleAS components in the middle tier and Infrastructure can use the same physical hostnames in their configuration settings regardless of whether the components are in the production or standby site.
For example, if a middle tier component in the production site uses the name "asmid1" to reach a host in the same site, the same component in the standby site can use the same name to reach asmid1’s equivalent peer in the standby site.
No changes to hostnames (physical, logical, or virtual) are required when the standby site takes over the production role.
|
Note: Although the physical hostnames in the production and standby sites must remain uniform between the two sites, the resolution of these physical hostnames to the correct hosts can be different. The section "Configuring Hostname Resolution" explains more on hostname resolution. |
To illustrate what should be done to plan and assign hostnames, let us use an example as shown in Figure 6-2.
Figure 6-2 Name assignment example in the production and standby sites
In Figure 6-2, two middle tier nodes exist in the production site. The Infrastructure can be a single node or an OracleAS Cold Failover Cluster solution (represented by a single virtual hostname and a virtual IP, as for a single node Infrastructure). The common names in the two sites are the physical hostnames of the middle tier nodes and the virtual hostname of the Infrastructure. Table 6–2 below details what the physical, logical, and virtual hostnames are in the example:
Table 6-2 Physical, logical, and virtual hostnames in Figure 6-2
| Physical Hostnames | Virtual Hostname | Logical Hostnames |
|---|---|---|
asmid1 |
- |
prodmid1, standbymid1 |
asmid2 |
- |
prodmid2, standbymid2 |
- |
infra |
prodinfra, standbyinfra |
Co-hosting non OracleAS applications
If the hosts in the production site are running non OracleAS applications, and you wish to co-host OracleAS on the same hosts, changing the physical hostnames of these hosts may break these applications. In such a case, you can keep these hostnames in the production site and modify the physical hostnames in the standby site to the same as those in the production site. The non OracleAS applications can then also be installed on the standby hosts so that they can act in a standby role for these applications.
As explained in the section "Terminology", physical, logical, and virtual hostnames have differing purposes in the OracleAS Disaster Recovery solution. They are also set up differently. Information on how the three types of hostnames are set up follow.
The naming of middle tier hosts in both the production and standby sites require the changing of the physical hostname in each host.
In Solaris, to change the physical hostname of a host:
|
Note: For other UNIX variants, consult your system administrator for equivalent commands in each step. |
Check to see what the existing physical hostname is set to. Type:
prompt> hostname
Use a text editor, such as vi, to edit the name in /etc/nodename to your planned physical hostname.
For each middle tier host, reboot it for the change to take effect.
Repeat step 1 to verify the correct hostname has been set.
Repeat the above steps for each host in the production and standby sites.
In Windows, to change the physical hostname of a host:
|
Note: The user interface elements in your version of Windows may vary from those described in the following steps. |
In the Start menu, select Control Panel.
Double-click the System icon.
Select the Advance tab.
Select Environment variables.
Under the User Environment variables for the installer account, select New to create a new variable.
Enter the name of the variable as "_CLUSTER_NETWORK_NAME_".
For the value of this variable, enter the planned physical hostname.
The logical hostnames used in the OracleAS Disaster Recovery solution are defined in DNS. These hostnames are visible in the network that the solution uses and are resolved through DNS to the appropriate hosts via the assigned IP address in the DNS system. You need to add these logical hostnames and their corresponding IP addresses to the DNS system.
Using the example in Figure 6-2, the following should be the additions made to the DNS system serving the entire network that encompasses the production and standby sites:
prodmid1.oracle.com IN A 123.1.2.333 prodmid2.oracle.com IN A 123.1.2.334 prodinfra.oracle.com IN A 123.1.2.111 standbymid1.oracle.com IN A 213.2.2.443 standbymid2.oracle.com IN A 213.2.2.444 standbyinfra.oracle.com IN A 213.2.2.210
As defined in the Terminology section, virtual hostname applies to the Infrastructure only. It is specified during installation of the Infrastructure. When you run the Infrastructure installation type, a screen called "Specify High Availibility" appears to provide a textbox to enter the virtual hostname of the Infrastructure that is being installed. Refer to the Oracle Application Server 10g Installation Guide for more details.
For the example in Figure 6-2, when you install the production site’s Infrastructure, enter its virtual hostname, "infra", when you see the Specify High Availibility screen. Enter the same virtual hostname when you install the standby site’s Infrastructure.
|
Note: If the Infrastructure is installed in a OracleAS Cold Failover Cluster solution, the virtual hostname is the name that is associated with the virtual IP of the OracleAS Cold Failover Cluster. |
In the Oracle Application Server Disaster Recovery solution, one of two ways of hostname resolution can be configured to resolve the hostnames you planned and assigned in the previous section. These are:
In UNIX, the order of the method of name resolution can be specified using the "hosts" parameter in the file /etc/nsswitch.conf. The following is an example of the hosts entry:
hosts: files dns nis
In the above statement, local hostnaming file resolution is preferred over DNS and NIS (Network Information Service) resolutions. When a hostname is required to be resolved to an IP address, the /etc/hosts file (UNIX) or C:\WINDOWS\system32\drivers\etc\hosts file is consulted first. In the event that a hostname cannot be resolved using local hostnaming resolution, DNS is used. (NIS resolution is not used for the OracleAS Disaster Recovery solution.) Refer to your UNIX system’s documentation if you wish to find out more about /etc/nsswitch.conf.
This method of hostname resolution relies on a local hostnaming file to contain the requisite hostname-to-IP address mappings. In UNIX, this file is /etc/hosts. In Windows, this file is C:\WINDOWS\system32\drivers\etc\hosts.
To use the local hostnaming file to resolve hostnames for the OracleAS Disaster Recovery solution in UNIX, for each middle tier and Infrastructure host in both the production and standby sites, perform the following:
Use a text editor, such as vi, to edit the /etc/nsswitch.conf file. With the "hosts:" parameter, specify "files" as the first choice for hostname resolution.
Edit the /etc/hosts file to include the following:
The physical hostnames and their correct IP addresses of all middle tier nodes in the current site. Ensure that the first entry is the hostname and IP address of the current node.
For example, if you are editing the /etc/hosts file of a middle tier node in the production site, enter all the middle tier physical hostnames and their IP addresses in the production site beginning the list with the current host. (Note that you should also include fully qualified hostnames in addition to the abbreviated hostnames. See Table 6-3.)
The virtual hostname of the Infrastructure in the current site.
For example, if you are editing the /etc/hosts of a middle tier node in the standby site, enter the virtual hostname, fully qualified and abbreviated, and IP address of the Infrastructure host in the standby site.
Reboot each host after editing the above files.
From each host, ping each physical hostname that is valid in its particular site to ensure that the IP addresses have been assigned correctly.
For the example in Figure 6-2, on the asmid1 host, use the following commands in:
ping asmid1
The returned IP address should be 123.1.2.333.
ping asmid2
The returned IP address should be 123.1.2.334.
ping infra
The returned IP address should be 123.1.2.111.
|
Note: Some UNIX variants, such as Solaris, require the-s option to return an IP address.
|
In Windows, the method of ordering hostname resolution varies depending on the Windows version. Refer to the documentation for your verion of Windows for the appropriate steps.
Using the example in Figure 6-2, Table 6-3 contains the required entries in the /etc/hosts file of each UNIX host. The entries in the Windows C:\WINDOWS\system32\drivers\etc\hosts file should reflect similarly.
Table 6-3 Logical and virtual hostname entries in each /etc/hosts file of example in Figure 6-2
| Host | Entries in /etc/hosts |
|---|---|
asmid1 in production site
|
123.1.2.333 asmid1.oracle.com asmid1 123.1.2.334 asmid2.oracle.com asmid2 123.1.2.111 infra.oracle.com infra |
asmid2 in production site
|
123.1.2.334 asmid2.oracle.com asmid2 123.1.2.333 asmid1.oracle.com asmid1 123.1.2.111 infra.oracle.com infra |
infra in production site
|
123.1.2.111 infra.oracle.com infra 123.1.2.333 asmid1.oracle.com asmid1 123.1.2.334 asmid2.oracle.com asmid2 |
asmid1 in standby site
|
213.2.2.443 asmid1.oracle.com asmid1 213.2.2.444 asmid2.oracle.com asmid2 213.2.2.210 infra.oracle.com infra |
asmid2 in standby site
|
213.2.2.444 asmid2.oracle.com asmid2 213.2.2.443 asmid1.oracle.com asmid1 213.2.2.210 infra.oracle.com infra |
infra in standby site
|
213.2.2.210 infra.oracle.com infra 213.2.2.443 asmid1.oracle.com asmid1 213.2.2.444 asmid2.oracle.com asmid2 |
To set up the OracleAS Disaster Recovery solution to use DNS hostname resolution, site-specific DNS servers must be set up in the production and standby sites in addition to the overall corporate DNS servers (usually more than one DNS server exists in a corporate network for redundancy). Figure 6-3 provides an overview of this setup.
|
See Also: Appendix A, " Setting Up a DNS Server " for instructions on how to set up a DNS server in a UNIX environment. |
For the above topology to work, the following requirements and assumptions are made:
The production and standby sites’ DNS servers are not aware of each other. They make non authoritative lookup requests to the overall corporate DNS servers if they fail to resolve any hostnames within their specific sites.
The production site and standby site DNS servers contain entries for middle tier physical hostnames and Infrastructure virtual hostnames. Each DNS server contain entries of hostnames within their own site only. The sites have a common domain name that is different from that of the overall corporate domain name.
The overall corporate DNS servers contain logical hostname entries for the middle tier hosts and Infrastructure hosts of both production and standby sites.
In UNIX, the /etc/hosts file in each host does not contain any entries for the physical, logical, or virtual hostnames of any host in either site. In Windows, this applies to the file C:\WINDOWS\system32\drivers\etc\hosts.
To set up the OracleAS Disaster Recovery solution for DNS resolution:
Configure each of the overall corporate DNS servers with the logical hostnames of all the hosts in the production and standby sites. Using the example in Figure 6-2, the following entries are made:
prodmid1.oracle.com IN A 123.1.2.333 prodmid2.oracle.com IN A 123.1.2.334 prodinfra.oracle.com IN A 123.1.2.111 standbymid1.oracle.com IN A 213.2.2.443 standbymid2.oracle.com IN A 213.2.2.444 standbyinfra.oracle.com IN A 213.2.2.210
For each site, production and standby, create a unique DNS zone by configuring a DNS server as follows:
Select a unique domain name to use for the two sites that is different from the corporate domain name. As an example, let’s use the name "oracleas" for the domain name for the two sites in Figure 6-2. The high level corporate domain name is oracle.com.
Configure the DNS server in each site to point to the overall corporate DNS servers for unresolved requests.
Populate the DNS servers in each site with the physical hostnames of each middle tier host and the virtual hostname of each Infrastructure host. Include the domain name selected in the previous step.
For the example in Figure 6-2, the entries are as follows:
For the production site’s DNS:
asmid1.oracleas IN A 123.1.2.333 asmid2.oracleas IN A 123.1.2.334 infra.oraclas IN A 123.1.2.111
For the standby site’s DNS:
asmid1.oracleas IN A 213.2.2.443 asmid2.oracleas IN A 213.2.2.444 infra.oracleas IN A 213.2.2.210
|
Note: If you are using the OracleAS Cold Failover Cluster solution for the Infrastructure in either site, enter the cluster’s virtual hostname and virtual IP address. For example, in the previous step above,infra is the virtual hostname and 123.1.2.111 is the virtual IP of the cluster in the production site. For more information on the OracleAS Cold Failover Cluster solution, see "Oracle Application Server Cold Failover Clusters".
|
Because Oracle Data Guard technology is used to synchronize the production and standby Infrastructure databases, the production Infrastructure must be able to reference the standby Infrastructure and vice versa.
For this to work, the IP address of the standby Infrastructure host must be entered in the production site’s DNS server with a unique hostname with respect to the production site. Similarly, the IP address of the production Infrastructure host must be entered in the standby site’s DNS server with the same hostname. The reason for these DNS entries is that Oracle Data Guard uses TNS Names to direct requests to the production and standby Infrastructures. Hence, the appropriate entries must be made to the tnsnames.ora file as well.
Using the example in Figure 6-2 and assuming that the selected name for the remote Infrastructure is "remoteinfra", the entries in the DNS server in the production site are:
asmid1.oracleas IN A 123.1.2.333 asmid2.oracleas IN A 123.1.2.334 infra.oracleas IN A 123.1.2.111 remoteinfra.oracleas IN A 213.2.2.210
And, for the standby site, its DNS server should have the following entries:
asmid1.oracleas IN A 213.2.2.443 asmid2.oracleas IN A 213.2.2.444 infra.oracleas IN A 213.2.2.210 remoteinfra.oracleas IN A 123.1.2.111
Oracle Data Guard sends redo data across the network to the standby system using OracleNet. SSH tunneling should be used with Oracle Data Guard as an integrated way to encrypt and compress the redo data before it is transmitted by the production system and subsequently decrypt and uncompress the redo data when it is received by the standby system.
|
See Also:
|
This section provides an overview of the steps for installing the OracleAS Disaster Recovery solution. After following the instructions in the previous section to set up the environment for the solution, go through this section for an overview of the installation process. Thereafter, follow the detailed instructions in the Oracle Application Server 10g Installation Guide to install the solution.
The following is the overall sequence for installing the OracleAS Disaster Recovery solution:
Install OracleAS Infrastructure in the production site (refer to Oracle Application Server 10g Installation Guide).
Install OracleAS Infrastructure in the standby site (refer to Oracle Application Server 10g Installation Guide).
Start the Infrastructure in each site before installing the middle tiers for that site.
Install the middle tiers in the production site (refer to Oracle Application Server 10g Installation Guide).
Install the middle tiers in the standby site (refer to Oracle Application Server 10g Installation Guide).
Note the following important points when you perform the installation:
The Infrastructure Identity Management and OracleAS Metadata Repository components must be installed on the same host. These components cannot be distributed over multiple hosts. (This requirement also applies to the OracleAS Cold Failover Cluster and OracleAS Active Failover Cluster solutions.)
Ensure that the same ports are used by equivalent peer hosts in both sites. For example, the asmid1 host in the standby site must use the same ports as the asmid1 host in the production site. Utilize a static ports definition file for this purpose (see note above and the next point).
Start the installer from the command line to use a static ports definition file. The command syntax is different for the middle tier and Infrastructure hosts.
For each middle tier host, use the following syntax:
In UNIX:
runInstaller oracle.iappserver.iapptop:s_staticPorts=staticports.ini
In Windows:
setup oracle.iappserver.iapptop:s_staticPorts=staticports.ini
For each Infrastructure host, use the following syntax:
In UNIX:
runInstaller oracle.iappserver.infrastructure:s_staticPorts=staticports.ini
In Windows:
setup oracle.iappserver.infrastructure:s_staticPorts=staticports.ini
In the installer’s Select Configuration Options screen, ensure that you select the High Availability Addressing option.
During Infrastructure installation, specify the virtual address assigned to the Infrastructure in the Specify High Availibility screen.
For the middle tier hosts, any of the available middle tier installation types can be installed. (Ensure that the Infrastructure services have been started for a site before installing any middle tiers in that site.)
During each middle tier installation, specify the Infrastructure’s virtual hostname as the Infrastructure database.
Start the OracleAS services on the hosts in each site starting with the Infrastructure.
For OracleAS Disaster Recovery purposes, the metadata information maintained within the the Infrastructure database is kept in synchronization by utilizing Oracle Data Guard technology. This technology propagates all database changes at the production site to the standby site for disaster tolerance.
Note that for OracleAS Disaster Recovery, archive logs are shipped from the production Infrastructure database to the standby Infrastructure database but are not applied. The application of these logs have to be done with the synchronization of file system configuration information, which is discussed in the section "Backing Up Configuration Files (Infrastructure and Middle Tier)".
The setup of Oracle Data Guard for OracleAS Disaster Recovery involves the following steps:
Prepare the Initialization Parameter File to be Copied to the Standby Database
Set Initialization Parameters for the Physical Standby Database
Configure Listeners for the Production and Standby Databases
By default, the production database does not have ARCHIVELOG mode enabled. However, it needs to be in ARCHIVELOG mode in order to ship archive logs to the standby database. The default destination directory for archive logs is:
UNIX:
<INFRA_ORACLE_HOME>/dbs/arch/
Windows:
<INFRA_ORACLE_HOME>\database\archive
To enable ARCHIVELOG mode:
Make sure the ORACLE_HOME and ORACLE_SID (the default is asdb) environment variables are properly set.
Ensure that the database is not being used by stopping all usage of the Infrastructure database. Execute the following commands on the Infrastructure database host:
UNIX:
<ORACLE_HOME>/bin/emctl stop iasconsole <ORACLE_HOME>/opmn/bin/opmnctl stopall
Windows:
<ORACLE_HOME>\bin\emctl stop iasconsole <ORACLE_HOME>\opmn\bin\opmnctl stopall
Ensure that Enterprise Manager has been stopped using the following command:
UNIX:
<ORACLE_HOME>/bin/emctl status iasconsole
Windows:
<ORACLE_HOME>\bin\emctl status iasconsole
Execute the following commands to connect and confirm that ARCHIVELOG mode is not enabled:
<ORACLE_HOME>/bin/sqlplus /nolog SQL> connect sys/<password> as sysdba SQL> archive log list Database log mode No Archive Mode Automatic archival Disabled Archive destination /private/oracle/oracleas/dbs/arch Oldest online log sequence 4 Current log sequence 6
In Windows, the sqlplus command can be executed as:
<ORACLE_HOME>\bin\sqlplus /nolog
In Windows, the archive destination should be <ORACLE_HOME>\database\archive.
Shutdown the database instance. Execute:
SQL> shutdown immediate
For Windows only, create an spfile using the following commands:
SQL> connect sys/<password> as sysdba SQL> create spfile from pfile;
Start up the instance, mount it, but do not open the database.
SQL> startup mount;
Enable database ARCHIVELOG mode.
SQL> alter database archivelog; SQL> alter system set log_archive_start=true scope=spfile; SQL> alter system set LOG_ARCHIVE_DEST_1= ’LOCATION=/private/oracle/oracleas/oradata MANDATORY’ SCOPE=BOTH;
In Windows, substitute the archive log path shown above appropriately (<ORACLE_HOME>\oradata).
Shut down and restart the database instance.
SQL> shutdown SQL> connect sys/<password> as sysdba SQL> startup
Verify that the database is now in ARCHIVELOG mode.
Execute the following command and verify that Database log mode is in Archive Mode and Automatic archival is Enabled.
SQL> archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination /private/oracle/oracleas/oradata Oldest online log sequence 4 Next log sequence to archive 6 Current log sequence 6
In Windows, substitute the archive destination path shown above appropriately.
On the production database, query the V$DATAFILE view to list the files that will be used to create the physical standby database as follows (UNIX paths are shown):
SQL> SELECT NAME FROM V$DATAFILE; NAME -------------------------------------------------------------------------------- /private/oracle/oracleas/oradata/asdb/system01.dbf /private/oracle/oracleas/oradata/asdb/undotbs01.dbf /private/oracle/oracleas/oradata/asdb/drsys01.dbf /private/oracle/oracleas/oradata/asdb/dcm.dbf /private/oracle/oracleas/oradata/asdb/portal.dbf . . . 24 rows selected.
On the production database, perform the following steps to make a closed backup copy of the production database:
Shut down the production database. Issue the following SQLPLUS statement to shut down the production database:
SQL> SHUTDOWN IMMEDIATE;
Copy the datafiles to a temporary location. Copy the datafiles that you identified in the previous section, "Identifying the Production Database Datafiles", to a temporary location using an operating system utility copy command. The following example uses the UNIX cp command (<ORACLE_HOME> is /private/oracle/oracleas):
mkdir /private/standby cp /private/oracle/oracleas/oradata/asdb/system01.dbf /private/standby/system01.dbf cp /private/oracle/oracleas/oradata/asdb/undotbs01.dbf /private/standby/undotbs01.dbf cp /private/oracle/oracleas/oradata/asdb/drsys01.dbf /private/standby/drsys01.dbf cp /private/oracle/oracleas/oradata/asdb/dcm.dbf /private/standby/dcm.dbf cp /private/oracle/oracleas/oradata/asdb/portal.dbf /private/standby/portal.dbf
In Windows, use the md command to create a new directory and the copy command to copy the files.
Restart the production database. Issue the following SQLPLUS statement to restart the production database:
SQL> STARTUP;
On the production database, create the control file for the standby database, as shown in the following example in UNIX (in Windows, substitute with your appropriate path):
SQL> alter database create standby controlfile as ’/private/standby/asdb.ctl’;
The filename for the newly created standby control file must be different from the filename of the current control file of the production database. In the example above, it is created in the new temporary directory for the standby. The control file must also be created after the last time stamp for the backup datafiles.
Create a traditional text initialization parameter file from the server parameter file used by the production database; a traditional text initialization parameter file can be copied to the standby location and modified. For example in UNIX (in Windows, substitute with your appropriate path):
SQL> create pfile=’/private/standby/initasdb.ora’ from spfile
Later, in the section "Create a Server Parameter File for the Standby Database" and thereafter, you will convert this file back to a server parameter file after it is modified to contain the parameter values appropriate for use with the physical standby database.
|
Note: You cannot use a single control file for both the production and standby databases. |
On the production system, use an operating system copy utility to copy the binary files mentioned in the following steps from the production system to the standby system. Before copying, make sure the following tasks have been performed:
Stop any processing against the infrastructure by using opmnctl and emctl as specified in the section "Enable ARCHIVELOG Mode for Production Database".
Backup datafiles created in the section "Make a Copy of the Production Database".
Standby control file created in the section "Create a Control File for the Standby Database".
Initialization parameter file created in the section "Prepare the Initialization Parameter File to be Copied to the Standby Database".
Clean up standby database files:
|
Note: The commands in this step must be run on the standby database. |
In UNIX:
standby> rm /private/oracle/oracleas/dbs/spfileasdb.ora standby> rm /private/oracle/oracleas/dbs/orapwasdb standby> rm /private/oracle/oracleas/oradata/asdb/*.*
In Windows:
del <ORACLE_HOME>\database\spfileasdb.ora del <ORACLE_HOME>\database\PWDasdb.ora del <ORACLE_HOME>\oradata\asdb\*
Identify the location of the init paramter file and clean this up (this is performed on the standby system).
In UNIX:
standby> ls -l initasdb.ora lrwxrwxrwx 1 nedcias svrtech 54 Nov 10 09:25 initasdb.ora -> /private/oracle/oracleas/admin/asdb/pfile/initasdb.ora standby> rm /private/oracle/oracleas/admin/asdb/pfile/initasdb.ora
In Windows:
del <ORACLE_HOME>\database\initasdb.ora
|
Note: These copy steps are examples and depending on the network configuration other utilities may need to be used. |
Copy the parameter initialization file in the previous step from the production machine to the standby machine.
In UNIX:
production> cd /private/standby production> cp initasdb.ora /net/standby/private/oracle/oracleas/admin/asdb/pfile/initasdb.ora
In Windows, use Windows Explorer or FTP to perform the copy operation.
Copy the control files to the standby machine:
For example, in UNIX:
production> cp asdb.ctl /net/standby/private/oracle/oracleas/oradata/asdb/control01.ctl production> cp asdb.ctl /net/standby/private/oracle/oracleas/oradata/asdb/control02.ctl production> cp asdb.ctl /net/standby/private/oracle/oracleas/oradata/asdb/control03.ctl
In Windows, you can use Windows Explorer to copy and paste the files or use FTP to copy them to the following location: <STANDBY_ORACLE_HOME>\oradata\asdb\
Copy the data files to the standby machine:
In UNIX:
production> cp <ORACLE_HOME>/oradata/asdb/*.dbf /net/standby/private/oracle/oracleas/oradata/asdb/.
In Windows, use Windows Explorer to copy and paste the files or use FTP to copy the *.dbf files to the following location: <STANDBY_ORACLE_HOME>\oradata\asdb\
Although most of the initialization parameter settings in the text initialization parameter file that you copied from the production system are also appropriate for the physical standby database, some modifications need to be made.
The following steps detail the parameters to modify or add to the standby initialization parameter file:
Edit the following parameters in the initasdb.ora file that was copied over from the production system:
*.standby_archive_dest - Specify the location of the archived redo logs that will be received from the production database.
*.standby_file_management - Set to AUTO.
*.remote_archive_enable - Set to TRUE.
For example, in UNIX:
*.standby_archive_dest=’/private/oracle/oracleas/standby/’ *.standby_file_management=AUTO *.remote_archive_enable=TRUE
In Windows, substitute the appropriate path for the standby_archive_dest parameter.
Create the directory that is specified for the standby_archive_dest parameter. For example, in UNIX:
Standby> mkdir /private/oracle/oracleas/standby
In Windows, use Windows Explorer or the md command to create a new directory.
At the standby site , make sure the ORACLE_HOME and ORACLE_SID (the default is asdb) environment variables are properly set.
For example, in UNIX:
Standby> setenv ORACLE_HOME /private/oracle/oracleas Standby> setenv ORACLE_SID asdb
In Windows, these variables should be set correctly in the registry.
If the standby system is running on a Windows system, use the ORADIM utility to create a Windows Service. For example:
<ORACLE_HOME>\bin\oradim -NEW -SID payroll2 -STARTMODE manual
A new password file has to be created on the standby system. Use the commands in the following example:
In UNIX:
standby> cd $ORACLE_HOME/dbs standby> $ORACLE_HOME/bin/orapwd file=orapwasdb password=<passwd>
In Windows:
cd %ORACLE_HOME%\database %ORACLE_HOME%\bin\orapwd file=PWDasdb.ora password=<passwd>
On both the production and standby sites, the Oracle Net Manager can be used to configure a listener for the respective databases. This is completed during the installation of the Infrastructure. During the installation process, the listeners were configured and started. The following is the configuration file that maintains the listener configuration information.
In UNIX:
<ORACLE_HOME>/network/admin/listener.ora
In Windows:
<ORACLE_HOME>\network\admin\listener.ora
Any modifications to this file requires the listeners to be restarted using the following commands:
In UNIX:
<ORACLE_HOME>/bin/lsnrctl stop <ORACLE_HOME>/bin/lsnrctl start
In Windows:
<ORACLE_HOME>\bin\lsnrctl stop <ORACLE_HOME>\bin\lsnrctl start
Enable dead connection detection by setting the SQLNET.EXPIRE_TIME parameter to 2 in the SQLNET.ORA parameter file on the standby system. For example:
SQLNET.EXPIRE_TIME=2
On both the production and standby systems, use Oracle Net Manager to create a network service name for the production and standby databases that is to be used by log transport services.
The Oracle Net service name must resolve to a connect descriptor that uses the same protocol, host address, port, and SID that are specified in the listener configuration file listener.ora. The connect descriptor must also specify that a dedicated server be used.
The following steps illustrate how the above should be set up:
On both the production and standby hosts, there needs to be an entry to point at the local database as well as the remote copy. Execute the TNSPING command on both nodes to confirm that the entries in the TNSNAMES.ORA file are correct.
Add the following to the TNSNAMES.ORA file on the production database host that points to the standby database (the standby hostname in this example is standby.oracle.com):
ASDB_REMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = standby.oracle.com)(PORT=1521)) ) (CONNECT_DATA = (SERVICE_NAME = asdb.oracle.com) ) )
TNSPING the remote host to verify that it can be reached:
In UNIX:
production> /private/oracle/oracleas/bin/tnsping asdb_remote
In Windows:
<ORACLE_HOME>\bin\tnsping asdb_remote
|
Note: Thetnsping command may require a fully qualified hostname. The NAMES.DEFAULT_DOMAIN setting in sqlnet.ora determines whether a fully qualified hostname is required or not.
|
Add the following entry to the TNSNAMES.ORA file on the standby host that points to the production database (the production host in this example is assumed to be production.oracle.com):
ASDB_REMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = production.oracle.com)(PORT=1521)) ) (CONNECT_DATA = (SERVICE_NAME = asdb.oracle.com) ) )
TNSPING the production host to verify that it can be reached:
In UNIX:
standby> /private/oracle/oracleas/bin/tnsping asdb_remote
In Windows:
<ORACLE_HOME>\bin\tnsping asdb_remote
On an idle standby database, use the SQLPLUS CREATE statement to create a server parameter file from the text initialization parameter file that was edited in the section "Set Initialization Parameters for the Physical Standby Database". For example:
Make sure the ORACLE_HOME and ORACLE_SID (the default is asdb) environment variables are properly set.
Connect and create an spfile.
In UNIX:
$ORACLE_HOME/bin/sqlplus /nolog SQL> connect sys/password as sysdba SQL> create spfile=’/private/oracle/oracleas/dbs/spfileasdb.ora’ from pfile=’/private/oracle/oracleas/dbs/initasdb.ora’;
In Windows:
%ORACLE_HOME%\bin\sqlplus /nolog SQL> connect sys/password as sysdba SQL> create spfile=’<ORACLE_HOME>\database\spfileasdb.ora’ from pfile=’<ORACLE_HOME>\database\initasdb.ora’;
On the standby database,issue the following SQLPLUS statements to start and mount the database in standby mode:
SQL> STARTUP NOMOUNT; SQL> ALTER DATABASE MOUNT STANDBY DATABASE;
This section describes the minimum amount of work you must do on the production database to set up and enable archiving to the physical standby database.
To configure archive logging from the production database to the standby site, the LOG_ARCHIVE_DEST_n and LOG_ARCHIVE_DEST_STATE_n parameters must be defined. The service name used must be the same as that set up in the "Create Oracle Net Service Names" section.
The following statements executed on the production database set the initialization parameters needed to enable archive logging to the standby site:
SQL> alter system set log_archive_dest_2=’SERVICE=asdb_remote’ scope=both; SQL> alter system set log_archive_dest_state_2=enable scope=both;
Archiving of redo logs to the remote standby location does not occur until after a log switch. A log switch occurs, by default, when an online redo log becomes full.
To force the current redo logs to be archived immediately, use the SQLPLUS ALTER SYSTEM statement on the production database:
SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
Once you create the physical standby database and set up log transport services, you should verify that database modifications are being successfully shipped from the production database to the standby database.
To see the new archived redo logs that were received on the standby database, you should first identify the existing archived redo logs on the standby database, archive a few logs on the production database, and then check the standby database again. The following steps illustrate how to perform these tasks:
Identify the existing archived redo logs.
On the standby database, query the V$ARCHIVED_LOG view to identify existing archived redo logs:
SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#; SEQUENCE# FIRST_TIME NEXT_TIME ---------- ------------------ ------------------ 8 11-JUL-02 17:50:45 11-JUL-02 17:50:53 9 11-JUL-02 17:50:53 11-JUL-02 17:50:58 10 11-JUL-02 17:50:58 11-JUL-02 17:51:03 3 rows selected.
Archive the current log.
On the production database, archive the current log using the following SQLPLUS statement:
SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
Verify that the new archived redo log has been received.
On the standby database, query the V$ARCHIVED_LOG view to verify that the redo log has been received using the following statement:
SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#; SEQUENCE# FIRST_TIME NEXT_TIME ---------- ------------------ ------------------ 8 11-JUL-02 17:50:45 11-JUL-02 17:50:53 9 11-JUL-02 17:50:53 11-JUL-02 17:50:58 10 11-JUL-02 17:50:58 11-JUL-02 17:51:03 11 11-JUL-02 17:51:03 11-JUL-02 18:34:11 4 rows selected.
The logs have now been shipped and are available on the standby database. To confirm that they are there, list out the contents of the directory <ORACLE_HOME>/standby.
Verify that the new archived redo log has NOT been applied.
On the standby database, query the V$ARCHIVED_LOG view to verify that the archived redo log has not been applied.
SQL> SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#; SEQUENCE# APP --------- --- 8 NO 9 NO 10 NO 11 NO 4 rows selected.
|
See Also: Oracle Data Guard Concepts and Administration Release 2 (9.2) (part number A96653-02) for more information on Oracle Data Guard. |
Once Oracle Data Guard has been set up between the production and standby sites, the procedure for synchronizing the two sites can be carried out. An initial synchronization should be done, before the production site is used, in order to obtain a baseline snapshot of the post-installation production site onto the standby site. This baseline can then be used to recover the production site configuration on the standby site if needed later.
In order to obtain a consistent point-in-time snapshot of the production site, the information stored in the Infrastructure database and the Oracle Application Server-related configuration files in the middle tier and Infrastructure hosts must be synchronized at the same time. Synchronization of the configuration files can be done by backing up the files and restoring them on the standby hosts using the Oracle Application Server Backup and Recovery Tool. For the Infrastructure database, synchronization is done using Oracle Data Guard by shipping the archive logs to the standby Infrastructure and applying these logs in coordination with the restoration of the configuration files.
The sequence of steps for the baseline synchronization (which can also be used for future synchronizations) are:
Backing Up Configuration Files (Infrastructure and Middle Tier)
Restoring Configuration Files (Infrastructure and Middle Tier)
These steps are detailed in the following two main sections.
The main strategy and approach to synchronizing configuration information between the production and standby sites is to synchronize the backup of Infrastructure and middle tier configuration files with the application of log information on the standby Infrastructure database.
For Oracle Application Server, not all the configuration information is in the Infrastructure database. The backup of the database files needs to be kept synchronized with the backup of the middle tier and Infrastructure configuration files. Due to this, log-apply services should not be enabled on the standby database. The log files from the production Infrastructure are shipped to the standby Infrastructure but are not applied.
The backup process of the production site involves backing up the configuration files in the middle tier and Infrastructure nodes. Additionally, the archive logs for the Infrastructure database are shipped to the standby site.
The procedures to perform the backups and the log ship are discussed in the following sections:
Backing Up Configuration Files (Infrastructure and Middle Tier)
|
IMPORTANT: Ensure that no configuration changes are going to be made to the Oracle Application Server system (underlying configuration files and Infrastructure database) as you perform the steps in this section. |
|
Note: At the minimum, the backup and restoration steps discussed in this section and the "Restoring to Standby Site" section should be performed whenever there is any administration change in the production site (inclusive of changes to the Infrastructure database and configuration files on the middle tier and Infrastructure nodes). On top of that, scheduled regular backups and restorations should also be done (for example, on a daily or twice weekly basis). See the Oracle Application Server 10g Administrator's Guide for more backup and restore procedures. |
After installing the OracleAS Disaster Recovery solution, Oracle Data Guard should have been installed in both the production and standby databases. The steps for shipping the archive logs from the production Infrastructure database to the standby Infrastructure database involve configuring Oracle Data Guard and executing several commands for both the production and standby databases. Execute the following steps to ship the logs for the Infrastructure database:
If not disabled already, disable log-apply services by running the following SQLPLUS statement on the standby host:
SQL> alter database recover managed standby database cancel;
Run the following command to perform a log switch on the production Infrastructure database. This ensures that the latest log file is shipped to the standby Infrastructure database
SQL> alter system switch logfile;
In normal operation of the production site, the production database frequently ships log files to the standby database but are not applied. At the standby site, you want to apply the logs that are consistent up to the same time that the production site’s configuration files are backed up. The following SQL statement encapsulates all Infrastructure database changes into the latest log and allows the Oracle Data Guard transport services to transport this log to the Infrastructure in the standby site:
SQL> select first_change# from v$log where status=’CURRENT’;
A SCN or sequence number is returned, which essentially represents the timestamp of the transported log.
Note down the SCN number as you will need this for the restoration of the production database changes on the standby site.
Continue to the next section to back up the configuration files on the middle tier host(s) and Infrastructure host.
Use the instructions in this section to back up the configuration files. The instructions require the use of the Oracle Application Server Backup and Recovery Tool. They assume you have installed and configured the tool on each OracleAS installation (middle tier and Infrastructure) as it needs to be customized for each installation. Refer to Oracle Application Server 10g Administrator's Guide for more details about that tool, including installation and configuration instructions.
For each middle tier and Infrastructure installation, perform the following steps (the same instructions can be used for the middle tier and Infrastructure configuration files):
After performing the installation and configuration steps detailed in the Oracle Application Server 10g Administrator's Guide, for the Oracle Application Server Backup and Recovery Tool, the variables oracle_home, log_path, and config_backup_path in the tool’s configuration file, config.inp, should have the appropriate values. Also, the following command for the tool should have been run to complete the configuration:
perl bkp_restore.pl -m configure_nodb
In Windows, the Perl executable can be found in <ORACLE_HOME>\perl\<perl_version>\bin\MSWin32-x86.
If you have not completed these tasks, do so before continuing with the ensuing steps.
Execute the following command to back up the configuration files from the current installation:
perl bkp_restore.pl -v -m backup_config
This command creates a directory in the location specified by the config_backup_path variable specified in the config.inp file. The directory name includes the time of the backup. For example: config_bkp_2003-09-10_13-21.
A log of the backup is also generated in the location specified by the log_path variable in the config.inp file. Check the log files for any errors that may have occurred during the backup process.
Copy the Backup and Recovery Tool’s directory structure and contents from the current node to its equivalent in the standby site. Ensure that the path structure on the standby node is identical to that on the current node.
Copy the backup directory (as defined by config_backup_path) from the current node to its equivalent in the standby site. Ensure that the path structure on the standby node is identical to that on the current node.
Repeat the steps above for each Oracle Application Server installation in the production site (middle tier and Infrastructure).
|
Note: There are two important items that should be maintained consistently between the production and standby sites. The directory names should be the same and the correlation of SCN to a given backup directory should be noted at both sites in administration procedures. |
After backing up the configuration files from the middle tier Oracle Application Server instances and Infrastructure together with the Infrastructure database, restore the files and database in the standby site using the instructions in this section, which consists of the following sub-sections:
Restoring the backed up files from the production site requires the Oracle Application Server Backup and Recovery Tool that was used for the backup. The instructions in this section assume you have installed and configured the tool on each OracleAS installation in the standby site, both in the middle tier and Infrastructure nodes. Refer to Oracle Application Server 10g Administrator's Guide for instructions on how to install the tool.
For each middle tier and Infrastructure installation in the standby site, perform the following steps (the same instructions can be used for the middle tier and Infrastructure configuration files):
Check that the Backup and Recovery Tool’s directory structure and the backup directory from the equivalent installation in the production site are present in the current node.
Stop the Oracle Application Server instances and their processes so that no modification of configuration files can occur during the restoration process. Use the following OPMN command:
In UNIX:
<ORACLE_HOME>/opmn/bin/opmnctl stopall
In Windows:
<ORACLE_HOME>\opmn\bin\opmnctl stopall
Check that all relevant processes are no longer running. In UNIX, use the following command:
ps -ef | grep <ORACLE_HOME>
In Windows, press <ctrl><alt><del> to bring up the Task Manager and verify that the processes have stopped.
Configure the backup utility for the Oracle home.
This can be accomplished either by configuring the Backup and Recovery Tool for the Oracle home or copying the backup configuration file, config.inp, from the production site peer. Below is an example of running the Backup and Recovery Tool configuration option:
perl bkp_restore.pl -v -m configure_nodb
In Windows, the Perl executable can be found in <ORACLE_HOME>\perl\<perl_version>\bin\MSWin32-x86.
Execute the following command to view a listing of the valid configuration backup locations:
perl bkp_restore.pl -v -m restore_config
Restore the configuration files using the following command:
perl bkp_restore.pl -v -m restore_config -t <backup_directory>
where <backup_directory> is the name of the directory with the backup files that was copied from the production site. For example, this could be config_bkp_2003-09-10_13-21.
Check the log file specified in config.inp for any errors that may have occurred during the restoration process.
Repeat the steps above for each Oracle Application Server installation in the production site (middle tier and Infrastructure).
During the backup phase, you executed several instructions to ship the database log files from the production site to the standby site up to the SCN number that you recorded as per instructed. To restore the standby database to that SCN number, apply the log files to the standby Infrastructure database using the following SQLPLUS statement:
SQL> alter database recover automatic from ’/private/oracle/oracleas/standby/’ standby database until change <SCN>;
(In Windows, substitute the path shown above appropriately.)
With this command executed and the instructions to restore the configuration files completed on each middle tier and Infrastructure installation, the standby site is now synchronized with the production site. However, there are two common problems that can occur during the application of the log files: errors caused by the incorrect specification of the path and gaps in the log files that have been transported to the standby site.
The following are methods of resolving these problems:
Find the correct log path.
On the standby Infrastructure database, try to determine location and number of received archive logs using the following SQLPLUS statement:
SQL> show parameter standby_archive_dest NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ standby_archive_dest string /private/oracle/oracleas/standby/
(The above example shows the UNIX path. The Windows equivalent path is shown in Windows systems.)
Use the log path obtained from the previous step to ensure that all log files have been transported.
At the standby Infrastructure database, perform the following:
standby> cd /private/oracle/oracleas/standby standby> ls 1_13.dbf 1_14.dbf 1_15.dbf 1_16.dbf 1_17.dbf 1_18.dbf 1_19.dbf
(In Windows, use the command cd to change to the appropriate directory and dir to view the directory contents.)
At the production Infrastructure database, execute the following SQLPLUS statement:
SQL> show parameter log_archive_dest_1 NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ log_archive_dest_1 string LOCATION=/private/oracle/oracleas/oradata MANDATORY log_archive_dest_10 string
(The above example shows the UNIX path. The Windows equivalent path is shown in Windows systems.)
Using the path specified in step 1, note the number and sequence of the log files. For example:
production> cd /private/oracle/oracleas/oradata production> ls 1_10.dbf 1_12.dbf 1_14.dbf 1_16.dbf 1_18.dbf asdb 1_11.dbf 1_13.dbf 1_15.dbf 1_17.dbf 1_19.dbf
(In Windows, use the command cd to change to the appropriate directory and dir to view the directory contents.)
In the above example, note the discrepency where the standby Infrastructure is missing files 1_10.dbf through 1_12.dbf. Since this gap in the log files happened in the past, it could be due to a problem with the historic setup involving the network used for the log transport. This problem has obviously been corrected and subsequent logs have been shipped. To correct the problem, copy (FTP) the log files to the corresponding directory on the standby Infrastructure database host and re-attempt the SQLPLUS recovery statement shown earlier in this section.
Scheduled outages are planned outages. They are required for regular maintenance of the technology infrastructure supporting the business applications and include tasks such as hardware maintenance, repair and upgrades, software upgrades and patching, application changes and patching, and changes to improve performance and manageability of systems. Scheduled outages can occur either for the production or standby site. Descriptions of scheduled outages that impact the production or standby site are:
Site-wide maintenance
The entire site where the current production resides is unavailable. Examples of site-wide maintenance are scheduled power outages, site maintenance, and regular planned switchovers.
OracleAS Cold Failover Cluster cluster-wide maintenance
This is scheduled downtime of the OracleAS Cold Failover Cluster for hardware maintenance. The scope of this downtime is the whole hardware cluster. Examples of cluster-wide maintenance are repair of the cluster interconnect and upgrade of the cluster management software.
Testing and validating the standby site as a means to test disaster recovery readiness.
For scheduled outages, a site switchover has to be performed, which is explained in the following section.
A site switchover is performed for planned outages of the production site. Both the production and standby sites have to be available during the switchover. The application of the database redo logs is synchronized to match the backup and restoration of the configuration files for the middle tier and Infrastructure installations.
During site switchover, considerations to avoid long periods of cached DNS information have to be made. Modifications to the site’s DNS information, specifically time-to-live (TTL), have to performed. Refer to "Manually Changing DNS Names" for instructions.
|
Note: All sessions to the production and standby databases need to be closed in order to perfom the switchover operation. This requires that all middle tier and Infrastructure instances need to be shut down. Also, theCJQO and QMNO database processes have sessions that need to be stopped.
|
To switchover from the production to standby site, perform the following:
Reduce the wide area DNS TTL value for the site.
On the production site, backup the middle tier and Infrastructure configuration files as described in the section "Backing Up Configuration Files (Infrastructure and Middle Tier)".
On the standby site, restore the backed up configuration files as described in the section "Restoring Configuration Files (Infrastructure and Middle Tier)".
Execute the following SQLPLUS statement to enable log apply services on the standby Infrastructure database so that all of the archive redo logs are applied:
SQL> alter database recover managed standby database disconnect from session;
Shut down all Oracle Application Server instances to close all sessions to the databases. Use the following command on all hosts:
In UNIX:
<ORACLE_HOME>/opmn/bin/opmnctl stopall
In Windows:
<ORACLE_HOME>\opmn\bin\opmnctl stopall
Stop the CJQ0 and QMN0 database processes as these also have open sessions to the database that need to be closed.
To stop the CJQ0 process, run the following query for the production and standby databases:
SQL> ALTER SYSTEM SET JOB_QUEUE_PROCESSES=0;
To stop the QMN0 process, run the following query for the production and standby databases:
SQL> ALTER SYSTEM SET AQ_TM_PROCESSES=0;
(The changes effected by the above statements need not require a database restart.)
Perform the switchover steps for the Infrastructure database in each site.
On the production database, perform the following:
Verify that it is possible to perform a switchover operation.
On the current production database, query the SWITCHOVER_STATUS column of the V$DATABASE fixed view on the production database to verify that it is possible to perform a switchover operation. For example:
SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE; SWITCHOVER_STATUS ----------------- TO STANDBY 1 row selected
The TO STANDBY value in the SWITCHOVER_STATUS column indicates that it is possible to switch the production database to the standby role. If the TO STANDBY value is not displayed, then verify that the Data Guard configuration is functioning correctly (for example, verify that all LOG_ARCHIVE_DEST_n parameter values are specified correctly).
Initiate the switchover operation on the production database.
To transition the current production database to a physical standby database role, use the following SQLPLUS statement on the production database:
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY;
After this statement completes, the production database is converted into a standby database. The current control file is backed up to the current SQLPLUS session trace file before the switchover operation. This makes it possible to reconstruct a current control file, if necessary.
Shut down and restart the former production instance.
Shut down the former production instance and restart it without mounting the database:
SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP NOMOUNT; SQL> ALTER SYSTEM SET STANDBY_ARCHIVE_DEST=’/private/oracle/oracleas/standby/’ SCOPE=BOTH; SQL> ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=’auto’ SCOPE=BOTH;
(In Windows, substitute the path above appropriately.)
Mount the database as a physical standby database:
SQL> ALTER DATABASE MOUNT STANDBY DATABASE;
Create the standby archive destination directory. For example:
In UNIX:
mkdir /private/oracle/oracleas/standby/
In Windows, use Windows Explorer or the following command:
md <ORACLE_HOME>\standby\
At this point in the switchover process, both databases are configured as standby databases.
On the original standby database, perform the following:
Verify the switchover status in the V$DATABASE view.
After you transition the production database to the physical standby role and the switchover notification is received by the standby databases in the configuration, you should verify if the switchover notification was processed by the original standby database by querying the SWITCHOVER_STATUS column of the V$DATABASE fixed view on the original standby database.
For example:
SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE; SWITCHOVER_STATUS ----------------- SWITCHOVER PENDING 1 row selected
The SWITCHOVER PENDING value of the SWITCHOVER_STATUS column indicates the standby database is about to switch from the standby role to the production role. If the SWITCHOVER PENDING value is not displayed and the TO PRIMARY value is displayed, this indicates all redo has been received and applied and the standby is now a candidate for switchover to a production role. Verify that the Data Guard configuration is functioning correctly (for example, verify that all LOG_ARCHIVE_DEST_n parameter values are specified correctly).
Switch the original standby database to the production role.
You can switch a physical standby database from the standby role to the production role when the standby database instance is either mounted in managed recovery mode or open for read-only access. It must be mounted in one of these modes so that the production database switchover operation request can be coordinated.
The SQL ALTER DATABASE statement used to perform the switchover automatically creates online redo logs if they do not already exist. Use the following SQLPLUS statements on the physical standby database that you want to transition to the production role:
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY; SQL> SHUTDOWN IMMEDIATE; SQL> CONNECT sys/<PASSWORD> AS SYSDBA SQL> STARTUP MOUNT; SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2=’SERVICE=asdb_remote’ SCOPE=BOTH; SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=enable SCOPE=BOTH; SQL> ALTER DATABASE OPEN;
Shut down and restart the new production database.
Shut down the original standby instance and restart it using the appropriate initialization parameters for the production role:
SQL> SHUTDOWN; SQL> STARTUP;
The original physical standby database is now transitioned to the production database role.
Begin sending redo data to the standby databases.
Issue the following statement on the new production database:
SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
On the new production database, start the Oracle Application Server instances using the following command:
In UNIX:
<ORACLE_HOME>/opmn/bin/opmnctl startall
In Windows:
<ORACLE_HOME>\opmn\bin\opmnctl startall
Perform a wide area DNS switchover to direct requests to the new production site based on one of the options presented in the section "Wide Area DNS Operations".
Adjust the wide area DNS TTL to an appropriate value.
An unplanned outage that impacts either or both the production and standby sites can be one of the following:
Site-wide failure The entire site where the current production resides is unavailable. Examples of site-wide outages are disasters at the production or standby site such as fire, flood, earthquake, or power outages.
Complete failure of the middle tier The entire middle tier is not available. Either all nodes are down or the application server instances on all nodes are down. The last surviving node of the Oracle Application Server 10g Farm for a service is no longer available.
OracleAS Cold Failover Cluster cluster-wide failure. The entire hardware cluster hosting the OracleAS Infrastructure is unavailable or crashes. This includes failure of nodes in the cluster as well as any other components that results in the hardware cluster and the Infrastructure on this site not being available.
Unplanned outages warrant the failover of the production site to the standby site. Configuration files restoration and an Oracle Data Guard failover operation are required. Failover restores the Oracle Application Server environment to the point of the last successful backup.
A site failover is performed for unplanned outages for the production site. Failover operations require the restoration on the standby site of the last backup of the configuration files of all hosts and the synchronized application of equivalent point-in-time redo logs (using the correct SCN number) to the standby database.
To failover the production site to the standby site:
On the standby site, restore the most recently backed up configuration files as described in the section "Restoring Configuration Files (Infrastructure and Middle Tier)".
Initiate the failover operation on the target physical standby database. Execute the following SQLPLUS statement:
SQL> alter database recover automatic from ’/private/oracle/oracleas/standby/’ standby database until change <SCN>;
(In Windows, substitute the path above appropriately.)
Convert the physical standby database to the production role.
Once the statement in the previous step completes successfully, the standby database is recovered to a consistent level with the configuration files that were restored in step 1.
Execute the following statements to transform the standby database to the production role:
SQL> connect sys/internal as sysdba SQL> select OPEN_MODE, STANDBY_MODE, DATABASE_ROLE from v$database; OPEN_MODE STANDBY_MOD DATABASE_ROLE ---------- ----------- ---------------- MOUNTED UNPROTECTED PHYSICAL STANDBY SQL> alter database activate standby database; Database altered. SQL> alter database mount; Database altered. SQL> select OPEN_MODE, STANDBY_MODE, DATABASE_ROLE from v$database; OPEN_MODE STANDBY_MOD DATABASE_ROLE ---------- ----------- --------------- MOUNTED UNPROTECTED PRIMARY SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-01139: RESETLOGS option only valid after an incomplete database recovery
|
Note: The last statement, "alter database open resetlogs;", may generate an ORA-01139 message (as shown) depending on the completeness of the recovery command in step 2. The message appears if the database recovery is complete and can be ignored.
Also, after issuing the last statement, you can no longer use this database as a standby database and subsequent redo logs from the original production database cannot be applied. |
Shut down and restart the new production database.
To complete the failover operation, you need to shut down the new production database and restart it in read/write mode using the proper traditional initialization parameter file (or server parameter file) for the production role:
SQL> SHUTDOWN IMMEDIATE; SQL> STARTUP; ORACLE instance started. Total System Global Area 143427356 bytes Fixed Size 280348 bytes Variable Size 92274688 bytes Database Buffers 50331648 bytes Redo Buffers 540672 bytes Database mounted. Database opened. SQL> select OPEN_MODE, STANDBY_MODE, DATABASE_ROLE from v$database; OPEN_MODE STANDBY_MOD DATABASE_ROLE ---------- ----------- ---------------- READ WRITE UNPROTECTED PRIMARY
Perform a wide area DNS switchover to direct requests to the new production site based on one of the options presented in the section "Wide Area DNS Operations".
After starting the new production database, a new standby site needs to be created. The steps for performing this are documented in this chapter starting from the section "Setting Up Oracle Data Guard" to the section "Backing Up Configuration Files (Infrastructure and Middle Tier)".
Once a new standby site has been established, a planned switchover may be performed to migrate production quality processing to the correct geographical site. Perform the steps in the section "Site Switchover Operations".
In order for client requests to be directed to the entry point of a production site, DNS resolution is used. When a site switchover or failover is performed, client requests have to be redirected transparently to the new site playing the production role. To accomplish this redirection, the wide area DNS that resolves requests to the production site has to be switched over to the standby site. The DNS switchover can be accomplished in one of the following two ways:
|
Note: A hardware load balancer is assumed to be front-ending each site. Check http://metalink.oracle.com for supported load balancers. |
When a wide area load balancer (global traffic manager) is deployed in front of the production and standby sites, it provides fault detection services and performance-based routing redirection for the two sites. Additionally, the load balancer can provide authoritative DNS name server equivalent capabilities.
During normal operations, the wide area load balancer can be configured with the production site’s load balancer name-to-IP mapping. When a DNS switchover is required, this mapping in the wide area load balancer is changed to map to the standby site’s load balancer IP. This allows requests to be directed to the standby site, which should have been brought up and now has the production role.
This method of DNS switchover works for both site switchover and failover. One advantage of using a wide area load balancer is that the time for a new name-to-IP mapping to take effect can be almost immediate. The downside is that an additional investment needs to be made for the wide area load balancer.
This method of DNS switchover involves the manual change of the name-to-IP mapping that is originally mapped to the IP address of the production site’s load balancer. The mapping is changed to map to the IP address of the standby site’s load balancer. Follow these instructions to perform the task:
Note the current time-to-live (TTL) value of the production site’s load balancer mapping. This mapping is in the DNS cache and will be there until the TTL expires. For the purposes of discussion and providing an example, let’s assume this TTL to be 3600 seconds.
Modify the TTL value to a short interval. For example, 60 seconds.
Wait one interval of the original TTL. This is the original TTL of 3600 seconds that we noted in step 1.
Ensure that the standby site is switched over to receive requests.
Modify the DNS mapping to resolve to the standby site’s load balancer giving it the appropriate TTL value for normal operation (for example, 3600 seconds).
This method of DNS switchover works for planned site switchovers only. The TTL value set in step 2 should be a reasonable time period where client requests cannot be fulfilled. The modification of the TTL is effectively modifying the caching semantics of the address resolution from a long period of time to a short period. Due to the shortened caching period, an increase in DNS requests can be observed.
