Oracle® Enterprise Manager

System Monitoring Plug-In Installation Guide for Oracle Exadata Healthchecks

Release 12.1.0.2.0

E27420-01

March 2012

The Oracle Exadata Healthchecks plug-in processes the XML output from the Exachk tool, which is included as part of Oracle Enterprise Manager system monitoring. The Exachk tool provides functionality for system administrators to automate the assessment of Exadata V2, X2-2, and X2-8 systems for known configuration problems and best practices.

Oracle Exadata Healthchecks Plug-in Contents

The Oracle Exadata Healthchecks plug-in bundle consists of the following files:

Healthchecks Plug-in XML parser utility: A Perl script that parses the resulting XML file generated from the Exachk tool and returns the output in em_result format. This format can be processed by the Enterprise Manager Metric Engine.
A metadata XML file: defines the new Target Type and Metrics.
A Default Collections XML file: defines the schedule (that is, the time interval for the data collection of metrics). It also contains the logic for raising the alerts depending on results from the Exachk tool, and the messages to be shown for alerts.
A Report SQL file: defines a new Report, which will be created during the jar deployment.
A Messages dlf file: contains the risk, recommendation, benefit, and fail message to be displayed in case of a healthcheck failure.
The mpcui files: used for home page customization to define the content of the home page and the menus provided on this page.

Audience

The Oracle Exadata Healthchecks plug-in is for users who perform administrative and problem-resolution tasks on Oracle Exadata V2, X2-2, and X2-8 systems.

Prerequisites

The following prerequisites must be met before you can deploy the plug-in:

Review Oracle Exadata Best Practices (MOS Note 757552.1) and Oracle Database Machine Monitoring Best Practices (MOS Note 1110675.1).

These documents provide a collection of articles related to best practices for the deployment of Oracle Database Machine and Exadata Storage Server.
Verify and enable the Exachk tool. For more information on enabling the Exachk tool, refer to "Configuring the Exachk Tool to Run Automatically".
Oracle Linux release 5.3 or later.

Downloading the Plug-in

Enterprise Manager Cloud Control 12c provides a self-update feature to download and apply software updates. Updates must be downloaded to the Software Library (the local store) before they can be applied. To download the Oracle Exadata Healthchecks plug-in:

Log in to Enterprise Manager Cloud Control as root.
Click Setup, then Extensibility, and finally Self Update.
On the Self Update page, select the Plug-in type.
On the Plug-in Updates page, click Oracle Exadata Healthchecks Plug-in and then Download.
Once the plug-in is downloaded, click Oracle Exadata Healthchecks Plug-in and then Apply.

Deploying the Oracle Exadata Healthchecks Plug-In

After you ensure that the prerequisites are met, follow these steps to deploy the plug-in to the appropriate Management Agents.

Log in to Enterprise Manager Cloud Control as root.
Click Setup from the upper-right-corner of the Cloud Control home page, then click Extensibility, and finally Plug-in.
Select Oracle Exadata Healthchecks. Click Deploy-on and select Management Servers.

The Deploy Management Plug-in wizard appears.
Enter the system password for the OMS database and click Next. Deployment prerequisites checks are done, and errors are reported if any. Click Deploy to start deployment on server.
After successful deployment on server, on the Plug-in page, select Oracle Exadata Healthchecks and click arrow next to Deploy-on and select Management Agent.

The Deploy Management Plug-in wizard appears with the option to add the agent on which to deploy the plug-in.
Click the Add button in the Selected Management Agent section.
Select the Agent(s) you want to deploy the plug-in to, then click Select.
Click Continue. Deployment prerequisites checks are done and errors are reported if any. Click Deploy to start deployment on agent.
Add the Oracle Exadata Healthchecks target:
1. Select the Add Targets Manually menu from Setup->Add targets.
2. Select Add Non-Host Targets by Specifying Target Monitoring Properties.
3. Select Oracle Exadata Healthchecks target type.
4. Select a management agent to host the new target.
5. Specify the name for the target instance, and set the following values for the instance properties:
  
  Exachk Results Directory = <specify the path of the exachk output directory>
  
  Max interval allowed (in days) between consecutive Exachk runs = 33 (This property will be defaulted to 30 days if unset.)
  
  Note:
  You can install the Oracle Exadata Healthchecks plug-in on one compute node.
By default, the Oracle Exadata Healthchecks plug-in is set to run hourly. You can change the schedule of the plug-in execution by changing the Default Collections XML file.

Prerequisite for Executing the Exachk Tool

The following environment variables should be set up before executing exachk (all three modes: -a, -s, -S).

To enable copying of results files on all the nodes in the cluster:
```
export RAT_COPY_EM_XML_FILES=1
```
Specify the exachk output path. The result files will be copied to this location on all the nodes in the cluster:
```
export RAT_OUTPUT = [exachk output directory]
```
Note:
The exachk output directory should exist on all nodes of the cluster.

Configuring the Exachk Tool to Run Automatically

The Exachk tool bundle can be downloaded from My Oracle Support (See Doc 1070954.1).

The Exachk tool should be run in "silent" mode using the -S option. Refer to "Frequency of Running the Exachk Tool" to schedule the Exachk execution.

Using the "-S" Command Line Option

Configure the SSH user equivalence for the RDBMS software owner (for example, oracle) from execution database server to oracle user all other database servers (mandatory). To validate a proper SSH user equivalence configuration, log in as the RDBMS software owner (for example, oracle) and execute the following commands:

$ ssh -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=no -l oracle dbServerName "echo \"oracle user equivalence is setup correctly\""

Where oracle is the RDBMS software owner and dbServerName is the database server hostname. Repeat for each database server.
If "Permission denied (publickey,gssapi-with-mic,password)" is returned, then the SSH user equivalence is not properly configured.

Note:

In the current version of the Exachk tool regardless of option (-s or -S), Storage Server Checks and InfiniBand Switch checks will not be performed.

A future release of the Exachk tool may provide silent support for InfiniBand switches.

Frequency of Running the Exachk Tool

The Exachk tool should be run in the following scenarios:

Once a month on a regular basis.
After taking corrective actions for the failures reported by the Exachk tool.

Metrics Collected

The checks executed by the Exachk tool are placed under the metric ExadataResults.

You can also evaluate the metrics any time by running the following command:

./emctl control agent runCollection <targetName>:oracle_exadata_hc ExadataResults

Reports

A HealthCheck Results report, which will be visible in the target's home page, lists all metrics along with the failed checks, irrespective of whether the alert is raised or not. A link to the reports page Full View will be available in this pane to have a pop-up view of the report.

The user can also view the report in a printable view using the menu from the target home page.

On the target home page click Targets->Information Publisher Reports, select the Exachk Execution Results link, and click Continue. The user is directed to the reports screen where they can filter the results on the metric type. Figure 1 shows an example of the report results.

This Report is bundled along with the plug-in and will be created in the OMS during deployment.

Figure 1 Oracle Exadata Healthchecks Reports Results

Oracle Exadata Healthchecks Report Results Example

Alerts

By default the plug-in is designed to raise alerts for the following checks only:

Verify Disk Cache Policy on database server.
Verify database server disk controllers use writeback cache.
Verify RAID Controller Battery Condition (database server).
Verify RAID Controller Battery Temperature (database server).
Verify Database Server Virtual Drive Configuration.
Verify Database Server Physical Drive Configuration.
Exachk not running.
Results and Exception file(s) missing.

Disable Alerts Using the Monitoring Template

To change the behavior of alerts, you can apply the Oracle-provided Oracle Exadata Healthchecks No Alerts template to disable alerts for all checks executed by the Exachk tool.

Follow the steps below to apply the Oracle-provided templates:

Log in to Enterprise Manager Cloud Control as SYSMAN.
Click Enterprise from the upper-left-corner of the Cloud Control home page. Click Monitoring from the drop-down menu, then Monitoring Templates.
Click the check box to enable search to "Display Oracle-provided templates and Oracle Certified templates." Search for the Oracle-provided template named Oracle Exadata Healthchecks No Alerts.
Select the radio button next to the template name and click Apply.
In the Apply Monitoring Template screen, click Add and select the targets on which you wish to apply this template. Click OK.

Note:
You can enable the alerts by applying the Oracle-provided Oracle Exadata Healthchecks template, following the same steps as above. Applying this template will enable alerts for the check names specified in Alerts section

Define a Monitoring Template

To define your own monitoring template and enable alerts for checks, follow the steps below:

Identify the Check ID of the check you want to be alerted of. The Check ID is used to uniquely identify each check. For the check name you want to set up an alert, get the corresponding Check ID from the Exachk Execution Results report.
Click Enterprise from the upper-left-corner of the Cloud Control home page, then click Monitoring from the drop-down menu, then select Monitoring Templates.
Click the Create button. In the next screen, you will be prompted to select a target instance for which to create the template for. Select the Oracle Exadata Healthcheck target instance that you have set up.
In the General tab of the Create Monitoring Template screen, enter a template name and description.
In the Metric Threshold tab of the Create Monitoring Template screen, click edit (pencil icon) next to the status column of Exadata Healthchecks metric in the tree structure displayed. This will redirect to the Edit Advance Settings: Status page.
Click the Add button on the top right corner of the Create Monitoring Template screen. In the text box that appears, enter the Check ID that you want to be alerted for Check ID. Enter FAIL as the critical threshold.

Repeat for all Check IDs that you want alerts to be enabled.
Click Continue and then OK to close the Edit Advance settings: Status page.
In the Monitoring Templates screen, select the radio button next to the template you created and click Apply.
In the Apply Monitoring Template screen, click Add and select the targets on which you wish to apply this template. Click OK.

Disable a Specific Alert

To disable a specific alert, follow the steps below:

Select the Metric and Policy Settings under the Monitoring menu in the target home page.
In this page, update the value for the Critical Threshold column to blank for the alert that you wish to disable and click OK.

Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

System Monitoring Plug-In Installation Guide for Oracle Exadata Exachk, Release 12.1.0.2.0

E27420-01

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.