11 Performing Postupgrade Tasks

Performing General Postupgrade Steps

Perform the following general postupgrade tasks:

1. Follow the postupgrade steps outlined in the chapter that describes how to install a new Enterprise Manager system, in the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

2. As a sanity check, ensure that all the database parameters are reset to their original values that existed before you started the upgrade operation, particularly for the job_queue_process parameter.

3. Restart the Management Agent that monitors the Management Services and Repository target.

Upgrading Oracle Database Plug-Ins

Ensure that you upgrade the existing database plug-in revisions to the latest revision by applying the required patches on the database plug-in homes of the OMS. Table 11-1 lists the patches required for various database plug-in revisions.

Make sure you apply this patch to the DB plug-in home of the OMS. For instructions to apply the patch, refer to the ReadMe that is released with the patch.

Conflicts/Supersets for each patch are:
Patch : 14336159
     Bug Conflict with 12361589
     Conflicting bugs are:
     14171309
Patch : 14340329
     Bug Superset of 14089634
     Super set bugs are:
     13713877,14089634
Following patches have conflicts: [12361589 14336159]

Upgrading Oracle Exalogic System Targets

Ensure that you upgrade your Oracle Exalogic System targets.

1. To do so, from the Enterprise menu, select Job, then select Library.

2. On the Job Library page, select the job name UPGRADE EXALOGIC SYSTEMS TO FUSION MIDDLEWARE 12.1.0.3.0 MODEL, and click Submit.

For the job to run successfully and upgrade the Oracle Exalogic System targets, you should have already upgraded the Management Agents, which are monitoring these targets, to 12c Release 2 (12.1.0.2) so that they contain the 12.1.0.3 Oracle Fusion Middleware plugin.

If one or more such Management Agents are not upgraded yet to 12c Release 2 (12.1.0.2), then the job fails and does not upgrade the Oracle Exalogic System targets. Under such circumstances, to resolve the issue, first upgrade those Management Agents, and then try submitting this job to upgrade the Oracle Exalogic System targets.

Restarting Central Agent When Additional OMS Is Upgraded

After you upgrade an additional OMS of 10g Release 5 (10.2.0.5) to 12c Release 2 (12.1.0.2) using 1-system upgrade approach, the targets that begin with /EMGC_GCDomain/GCDomain might appear as if they are down (see Example 11-1).

To resolve this issue, restart the upgraded central agent (that is, the Management Agent installed with the OMS).

/EMGC_GCDomain/GCDomain/EMGC_OMS2 
/EMGC_GCDomain/GCDomain/EMGC_OMS2/emgc 
/EMGC_GCDomain/GCDomain/EMGC_OMS2/empbs 
/EMGC_GCDomain/GCDomain/EMGC_OMS2/OCMRepeater 
/EMGC_GCDomain/GCDomain/EMGC_OMS2/oracle.security.apm(11.1.1.3.0)

Stopping OCM Scheduler

If OCM was installed in the Enterprise Manager system that you upgraded, then follow these steps:

• If you upgraded from Enterprise Manager 10g Grid Control Release 5 (10.2.0.5), then follow these steps:

  1. Set the environment variable ORACLE_HOME to the Oracle Management Service (OMS) home.

     • In bash terminal, run the following command:

       export ORACLE_HOME=<absolute_path_to_oms_home>

     • In other terminals, run the following command:

       setenv ORACLE_HOME <absolute_path_to_oms_home>

  2. Stop the OCM scheduler:

     $ORACLE_HOME/ccr/bin/emCCR stop

  Note:

  If you have the OCM scheduler installed on Oracle Management Agent, then repeat these steps on the Management Agent home as well. In this case, set the environment variable ORACLE_HOME to the Management Agent home as described in Step (1), and perform Step (2) from its home.

• If you upgraded from Enterprise Manager 11g Grid Control Release 1 (11.1.0.1), then

  1. Set the environment variable ORACLE_CONFIG_HOME to the OMS instance home:

     • In bash terminal, run the following command:

       export ORACLE_CONFIG_HOME=<absolute_path_to_gc_inst>

     • In other terminals, run the following command:

       setenv ORACLE_CONFIG_HOME <absolute_path_to_gc_inst>

  2. Set the environment variable ORACLE_HOME to the OMS home:

     setenv ORACLE_HOME <absolute_path_to_oms_home>

  3. Stop the OCM scheduler:

     $ORACLE_HOME/ccr/bin/emCCR stop

  Note:

  If you have the OCM scheduler installed on Oracle Management Agent, then repeat these steps on the Management Agent home as well. In this case, skip Step (1), set the environment variable ORACLE_HOME to the Management Agent home as described in Step (2), and perform Step (3) from its home.

Deleting Obsolete Targets

The following targets from the earlier release of Enterprise Manager are not deleted automatically. This is because you might have created notification rules, metric threshold settings, compliance standard settings, jobs, and so on for these targets, and you might want to copy them to the new Oracle WebLogic Server targets of Enterprise Manager Cloud Control. After copying them, manually delete these targets.

• While upgrading from Enterprise Manager 10g Grid Control Release 5 (10.2.0.5), the following targets are not deleted automatically. You must delete them manually.

  • EM Website

  • EM Website System

  • The following targets of each old OMS. You can delete these by deleting the top-level Oracle Application Server target of each old OMS.

    • Oracle Application Server

    • OC4J

    • Oracle HTTP Server

    • Web Cache

• While upgrading from Enterprise Manager 11g Grid Control Release 1 (11.1.0.1), the following targets are not deleted automatically. You must delete them manually.

  • EM Website

  • EM Website System

  • The following targets of each old OMS. You can manually delete these by deleting the top-level farm target secFarm_GCDomain.

    • Oracle Fusion Middleware Farm

    • Oracle Weblogic Domain

    • Application Deployment

    • Metadata Repository

  Note:

  After you upgrade your Enterprise Manager using the 1-System upgrade approach, these targets might appear as if they are up and running. This status is false. In any case, do not investigate to resolve the status issue. Manually delete the targets as they are no longer needed in the Enterprise Manager Cloud Control.

Disabling Incident Rule Sets

If you upgraded using the 2-System upgrade approach, then disable the default incident rule sets. If you do not disable them, the incidents are automatically created for all critical metric alerts.

To disable the default incident rule sets, follow the steps:

1. In Cloud Control, from the Setup menu, select Incidents, and then select Incident Rules.

2. On the Incident Rules - All Enterprise Rules page, scroll down the table, and expand Incident management Ruleset for all targets.

3. Select the row for the rule set you want to disable.

4. From the Actions menu, click Disable.

5. On the pop-up window, click OK.

6. Repeat Step (3) to Step (5) for every rule set you want to disable.

7. Once all the rule sets are disabled, ensure that the Enabled column of the table shows No with a warning icon next to it.

Resolving Metric Collection Errors for SOA Target

If you were monitoring a SOA target in your earlier release of Enterprise Manager, then in the health report you generated as described in Generating Health Report of Deployed Oracle Management Agents, you might have seen metric collection errors for the metrics Top SOA SQL Queries and Dehydration Store Tables.

To resolve that issue, access the Monitoring Configuration page of the SOA target, and provide the database credentials.

Enabling Linux Patching After Enterprise Manager Upgrade

After upgrading Enterprise Manager, you must stop all active jobs for Linux patching that were submitted prior to the upgrade, and resubmit them on the same targets. This section describes how you can achieve this.

In particular, this section covers the following:

• Configuring DownloadLatestPackages Job

• Configuring UpdateHostPackages Job

(Optional) Deleting the Old OMS Home

After upgrading your Enterprise Manager system completely, you can choose to delete your old OMS home. To do so, follow these steps:

• If you upgraded from 12c Release 1 (12.1.0.1) to 12c Release 2 (12.1.0.2), then:

  • If you upgraded a full release of 12c Release 1 (12.1.0.1) to 12c Release 2 (12.1.0.2), then follow the instructions outlined in Appendix L.

  • If you upgraded an additional OMS of 12c Release 1 (12.1.0.1) to 12c Release 2 (12.1.0.2), then follow the instructions outlined in the Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide.

• If you upgraded from 11g Release 1 (11.1.0.1) to 12c Release 2 (12.1.0.2), either full release or an additional OMS, then follow the instructions outlined in the Oracle Enterprise Manager Grid Control Advanced Installation and Configuration Guide.

• If you upgraded from 10g Release 5 (10.2.0.5) to 12c Release 2 (12.1.0.2), either full release or an additional OMS, then follow the instructions outlined in the Oracle Enterprise Manager Grid Control Installation and Configuration Guide.

Overview of Deferred Data Migration

Deferred Data Migration is a post-upgrade activity to migrate the format of the data stored in an earlier release of Enterprise Manager to the format compatible with the upgraded Enterprise Manager system. The migration activity is essentially a job in Enterprise Manager that is submitted when the Oracle Management Repository gets upgraded and is scheduled to run in the background when the upgraded Enterprise Manager system starts functioning.

The format of the data stored in Enterprise Manager Cloud Control is different from the format in which the data was stored in any earlier release of Enterprise Manager.

When you upgrade from an earlier release of Enterprise Manager to Enterprise Manager Cloud Control, the data format gets automatically converted or migrated to the format that is compatible with Enterprise Manager Cloud Control.

However, the time taken to migrate the data format depends on the volume of data available in your earlier release of Enterprise Manager. Therefore, if you have a large amount of data, then it takes longer to migrate, and as a result, the upgrade process takes more time to complete. Unfortunately, until the upgrade process completes, your existing Enterprise Manager system might be unavailable, particularly when you use a 1-System upgrade approach (either on the local host or on a different, remote host).

Considering this, Oracle has fine-tuned its upgrade process to migrate the data format in two distinct phases.

In the first phase, when the Enterprise Manager system is shut down and upgraded, the most critical data relevant to the functioning of Enterprise Manager Cloud Control is migrated within a short time so that the new system can start functioning without much downtime. At this point, only some historical data is unavailable, but you can start monitoring the targets in the upgraded Enterprise Manager system, and see new alerts generated by the upgraded Oracle Management Agent.

In the second phase, after the upgraded Enterprise Manager system starts functioning, the remaining data is migrated.

The data whose format is migrated in the second phase, after Enterprise Manager Cloud Control starts functioning, is called the Deferred Data, and this process of migrating from old to new format is called the Deferred Data Migration.

Tracking the Status of Deferred Data Migration Jobs

To track the status of deferred data migration jobs, follow these steps:

1. In Cloud Control, from the Setup menu, select Manage Cloud Control and then, click Post Upgrade Tasks.

2. On the Post Upgrade Tasks page, click the Deferred Data Migration tab.

   The tab provides a list of data migration jobs with their status, start date and time, and end date and time.

3. In this tab, you can do the following:

   • To start a data migration job for a component, select that component and click Start. By default, the job starts immediately. You cannot change this behavior.

   • To rerun a failed job, click the status icon to reach the Job Run page. On the Job Run page, click Edit. On the Edit <JobName> Job page, click Submit to rerun the job.

   • To hide or unhide the table columns, from the View list, select an appropriate option.

   • To detach the table from the screen, click Detach.

Overview of Accrued Data Migration

Accrued Data Migration is a post-upgrade activity to migrate the accrued data stored in an earlier release of Oracle Management Repository (Management Repository) to the upgraded Management Repository. The accrued data relates to functional areas such as blackouts, alerts, events, metrics, and so on.

The migration activity is essentially a job in Enterprise Manager that runs in the background immediately after the earlier releases of Oracle Management Agents (Management Agents) are switched over to Oracle Management Agent 12c, particularly in the case of 2-System upgrade approach.

As part of the upgrade process, when you switch over the earlier releases of Oracle Management Agents to Oracle Management Agent 12c, the new Management Agents start communicating with Enterprise Manager Cloud Control, and start uploading data to the upgraded Oracle Management Repository (Management Repository).

In the 1-System upgrade approach, the entire Enterprise Manager system is upgraded, and therefore, all the Management Agents are switched over at a given time. In this approach, earlier releases of Management Agents cannot coexist with the upgraded ones, and once they are switched over, the upgraded Management Agents upload data only to one Management Repository at any given point, which is the upgraded repository. Therefore, there is no scope for accrued data.

However, in the 2-System upgrade approach, you can choose to switch over your Management Agents in phases. When one set of Management Agents are switched over, they start uploading data about their hosts and targets to the upgraded Management Repository, whereas the ones that are not switched over yet, continue to upload data about their hosts and targets to the old Management Repository. Therefore, the earlier releases of Management Agents coexist with the upgraded ones when only some of them are switched over, and under such circumstances, the data is uploaded to the old Management Repository as well as the new one at a given point. However, note that only one Management Agent represents a host at any given time, so the coexistence does not indicate two Management Agents on the same host. It only indicates an earlier release of Management Agent monitoring a host that coexists with an upgraded Management Agent monitoring another host.

When you swtich over the next set of Management Agents in the next phase, although they start uploading data to the upgraded Management Repository, the data uploaded to the old Management Repository before being switched over continues to exist in the old Management Repository. This data is called the Accrued Data, and the process of migrating this accrued data from the old Management Repository to the new one is called the Accrued Data Migration Process.

Tracking the Status of Accrued Data Migration Jobs

To track the status of accrued data migration jobs, follow these steps:

1. In Cloud Control, from the Setup menu, select Manage Cloud Control and then, click Post Upgrade Tasks.

2. On the Post Upgrade Tasks page, click the Accrued Data Migration tab.

   The Accrued Data Migration page displays information about the accrued data migration jobs related to all the targets available in your Enterprise Manager system. By default, for every target, the accrued data migration job migrates the ECM history details as well as the metric details.

3. In this tab, you can do the following:

   • To control the data being migrated for all the targets, from the Run accrued data migration for list, select one of the following options:

     • ECM HISTORY MIGRATION, to migrate only the ECM history details.

     • METRIC DATA MIGRATION, to migrate only the metric details.

     • All, to migrate the ECM history details as well as the metric details.

   • To view different types of data, from the View Targets list, select one of the following options:

     • All, to view all the accrued data migration jobs for all the targets in your Enterprise Manager system.

     • Active, to view the accrued data migration jobs that either failed or are currently running.

     • History, to view the accrued data migration jobs that succeeded.

     • Not Started, to view the accrued data migration jobs that have not started yet.

   • To retry a failed job, select the target for which the job failed and click Retry.

   • To stop a running job, select the target for which you want to stop the job and click Stop.

Overview of Diff Reports

Diff Reports provide information about the configuration or setup-related changes that were manually made to the earlier release of Enterprise Manager while it was being upgraded to Enterprise Manager Cloud Control using the 2-System upgrade approach.

In the 2-System upgrade approach, the earlier release of Enterprise Manager coexists with Enterprise Manager Cloud Control until the upgrade operation ends. In some production environments, the upgrade operation might take a long time to complete due to a large volume of data, and during this period, you might make some changes to the functional areas in the earlier release. For example, you might configure an additional location for the software library.

Such changes must be carried over to Enterprise Manager Cloud Control so that the upgraded system is identical to the earlier release. The Diff Reports show a summary of such changes so that you can manually redo those changes in Enterprise Manager Cloud Control.

Generating and Viewing Diff Reports

To generate and view diff reports, follow these steps:

1. In Cloud Control, from the Setup menu, select Manage Cloud Control and then, click Post Upgrade Tasks.

2. On the Post Upgrade Tasks page, click the Diff Report tab.

   The Diff Report page lists the reports that were generated in the past for various components. However, when you visit this page the first time, you will not see any reports.

3. On this page, you can do the following:

   • To generate the reports for various components, click Regenerate Report.

     Note:

     On clicking Regenerate Report, the status in the Status column changes to In Progress. Keep refreshing the page manually until the status changes to Completed.

   • To view a report of a particular component, select a component (row) from the table, and click Show Report.

   • To refresh a report and view the latest changes related to a component, select a component (row) from the table, and click Regenerate Report.

   • To hide or unhide the table columns, or to reorder the table columns, from the View list, select an appropriate option.

   • To detach the table from the screen, click Detach.

Why Update Incident Rules

In the earlier releases of Enterprise Manager, OMS and Repository was a common target type defined for all the OMS instances in your environment. And the metrics collected for the different OMS instances were shown within this common target type.

However, in Enterprise Manager Cloud Control, in addition to the target type OMS and Repository, a new target type Oracle Management Service has been introduced to represent each of the OMS instances in your environment. Therefore, if you have five OMS instances in your environment, you will see one target type OMS and Repository and five instances of the target type Oracle Management Service - one for each OMS.

While the target type OMS and Repository captures metrics that are common to all the OMS instances in your environment, the target type Oracle Management Service captures metrics specific to an OMS.

Table 11-2 lists the metrics and describes the changes done to them due to the introduction of a new target type Oracle Management Service, and the actions you must take for each change.

Updating Incident Rules for Moved Metrics

To update the incident rules set to the metrics that have been moved to the individual target type Oracle Management Service in Enterprise Manager Cloud Control, follow these steps:

1. In Cloud Control, from the Setup menu, select Incidents, and then, click Incident Rules.

2. On the Incident Rules page, from the table, select an incident rule created for Oracle Management Service (OMS), click Edit.

3. On the Edit Rule Set page, in the Rules tab, select the event rule that operates on the moved metrics, and click Edit.

   Enterprise Manager displays the Edit Rule Wizard where you can edit the incident rule for the selected incident rule set.

4. In the Edit Rule Wizard, do the following:

   1. On the Select Events page, select the metrics listed in the second row of Table 11-2, and click Remove.

   2. Click Next.

   3. On the Add Actions page and the Specify Name and Description page, click Next without making any change.

   4. On the Review page, click Continue.

5. Click Save.

   Enterprise Manager displays the Incident Rules page.

6. On the Incident Rules page, click Create Rule Set.

7. On the Create Rule Set page, enter a unique name for the rule set.

8. In the Targets tab, select All targets of types, and select Oracle Management Service.

9. In the Rules tab, click Create.

10. In the Select Type of Rule to Create dialog, select Event Rule, and click Continue.

    Enterprise Manger displays the Create New Rule Wizard where you can create a new incident rule set.

11. In the Create New Rule Wizard, do the following:

    1. On the Select Events page, from the Event Type list, select Metric Alert.

       Note:

       For the metric Management Service Status, select Target Availability from the Event Type list.

    2. Select Specific metric alerts, and click Add.

    3. In the Select Specific Metric Alert dialog, do the following:

       1. In the Search region, from the Target Type list, select Oracle Management Service, and then, click Search.

       2. From the table, select the moved metrics listed in the second row of Table 11-2.

       3. In the Severity and Corrective Action Status region, from the Severity list, select an appropriate severity level.

       4. Click OK.

    4. Click Next.

    5. On the Add Actions page and the Specify Name and Description page, click Next without making any change.

    6. On the Review page, click Continue.

12. Click Save.

    Enterprise Manager displays the Incident Rules page.

Updating Incident Rules for Renamed Metrics

To update the incident rules set to the metrics that have been renamed in Enterprise Manager Cloud Control, follow these steps:

1. In Cloud Control, from the Setup menu, select Incidents, and then, click Incident Rules.

2. On the Incident Rules page, from the table, select an incident rule created for Oracle Management Service (OMS), click Edit.

3. On the Edit Rule Set page, in the Rules tab, select the event rule that operates on the renamed metrics, and click Edit.

   Enterprise Manager displays the Edit Rule Wizard where you can edit the incident rule for the selected incident rule set.

4. In the Edit Rule Wizard, do the following:

   1. On the Select Events page, select the metrics listed in the third row of Table 11-2, and click Remove.

   2. Click Add.

   3. In the Select Specific Metric Alert dialog, do the following:

      1. In the Search region, from the Target Type list, select OMS and Repository, and then, click Search.

      2. From the table, select the renamed metrics listed in the third row of Table 11-2.

      3. In the Severity and Corrective Action Status region, from the Severity list, select an appropriate severity level.

      4. Click OK.

   4. Click Next.

   5. On the Add Actions page and the Specify Name and Description page, click Next without making any change.

   6. On the Review page, click Continue.

Deleting Incident Rules

To remove the decommissioned metrics from the the incident rules, follow these steps:

1. In Cloud Control, from the Setup menu, select Incidents, and then, click Incident Rules.

2. On the Incident Rules page, from the table, select an incident rule created for Oracle Management Service (OMS), click Edit.

3. On the Edit Rule Set page, in the Rules tab, select the rule Metric Alerts Event Rule, and click Edit.

   Enterprise Manager displays the Edit Rule Wizard where you can edit the incident rule for the selected incident rule set.

4. In the Edit Rule Wizard, do the following:

   1. On the Select Events page, select the metrics listed in the last row of Table 11-2, and click Remove.

   2. Click Next.

   3. On the Add Actions page and the Specify Name and Description page, click Next without making any change.

   4. On the Review page, click Continue.