Oracle® Enterprise Manager Exadata Management Getting Started Guide Release 12.1 (12.1.0.1, 12.1.0.2, and 12.1.0.3) Part Number E27442-04 |
|
|
PDF · Mobi · ePub |
Once the Oracle Exadata Database Machine has been discovered through Enterprise Manager Cloud Control 12c, you must complete post-discovery configuration on the following targets:
Configure Storage Cell SNMP for Enterprise Manager Monitoring
Configure the Compute Node ILOM SNMP for Enterprise Manager Monitoring
Remember:
You must remove the SNMP notification on the cell, InfiniBand switch, ILOM, Cisco switch, PDU, and KVM manually if you remove these targets from Enterprise Manager.Starting with Exadata plug-in Release 12.1.0.3, when the Exadata Database Machine target is deleted through Enterprise Manager Cloud Control, there is an option to remove the SNMP notification from the cells and InfiniBand switches.
You must configure SNMP forwarding on each of the compute nodes being used as monitoring agents for other targets.
For the Enterprise Manager Agent to receive traps, an SNMP trap forwarder must be set up on the host where the Enterprise Manager Agent is running so that the forwarder utility can receive traps using port 162 and forward the same to the agent receivelet's listening port.
Note:
The Enterprise Manager Agent receivelet listening port is specified as the value of theEMD_URL
property in the <AGENT_INST>/sysman/config/emd.properties
file, where <AGENT_INST>
is the location of the Agent's agent_inst
directory.Follow the steps below on the compute nodes being used to monitor other targets. Ideally, you should perform the steps on all compute nodes in preparation for a scenario when monitoring is moved to a compute node that was not previously being used to monitor other targets.
The following commands must be run as root
user:
Get the port value from EMD_URL
property in the <AGENT_INST>/sysman/config/emd.properties
file. This is the port at which receivelet will listen over UDP for traps (for this example, the port value is 3872).
Confirm that the receivelet is listening on this port over UDP. An entry should be seen in the below command's output:
netstat -an | grep 3872 | grep udp
Run the following command to stop the forwarder utility:
service snmptrapd stop
Typically, the service
command will be located in the /sbin
directory.
Add the following entry to the /etc/snmp/snmptrapd.conf
configuration file:
authcommunity log,execute,net <community string> forward default udp:localhost:3872
Run the following command:
chkconfig snmptrapd on
Run the following command to start the forwarder utility:
service snmptrapd start
Exadata Storage Cell SNMP configuration is performed using the cellcli
command and can be run in batch using dcli
from a compute node.
Notes:
For Enterprise Manager Cloud Control 12c Bundle Pack 1 (BP1):
The SNMP trap setup for Exadata Storage Cells is done automatically after finishing the guided discovery process.
For Exadata plug-in Release 12.1.0.2 and later:
During the discovery process, you can optionally provide the necessary root
credentials to set up SNMP trap for Exadata Storage Cells. If you have done so, then you can skip the remaining steps of this section and proceed with Configure and Verify SNMP for InfiniBand Switch Targets.
While using the ALTER CELL
command, all existing subscribers should be specified along with the new subscriber being added. Similarly, you can also modify the notificationPolicy
or notificationMethod
attributes.
While using the ALTER CELL
command, the host=
and community=
attribute values should be quoted, and type=
is NOT quoted.
If you are using the DCLI utility to set up SNMP alerting, then any command containing punctuation, which will be interpreted by the local shell, must be enclosed with double quotation marks. If the command includes the following characters, then outer quotation marks and escape characters are required:
The backslash (\) is the escape character that allows the characters to be passed to the CellCLI utility without being interpreted by the remote shell.
Check the current SNMP configuration using the following cellcli
commands:
To list the current subscribers for SNMP:
cellcli -e list cell attributes snmpSubscriber
When correctly configured, this command should list the Master and non-Master agents for the cell target, for example:
# cellcli -e list cell attributes snmpSubscriber ((host=[hostname.mycompany.com],port=3872,community=public), (host=[hostname2.mycompany.com],port=3872,community=public))
To list the currently configured notification methods:
cellcli -e list cell attributes notificationMethod
Possible values are snmp, mail and snmp,mail. When correctly configured, this command should return either snmp
or snmp,mail
, for example:
# cellcli -e list cell attributes notificationMethod snmp,mail
To list the currently configured notification policy for SNMP:
cellcli -e list cell attributes notificationPolicy
Possible values are any or all of Critical, Warning, and Clear. When correctly configured, this command should return Critical
, Warning
, Clear
, for example:
# cellcli -e list cell attributes notificationPolicy Critical,Warning,Clear
cellcli
To set the values of snmpSubscriber
, notificationMethod
and notificationPolicy
:
Set the snmpSubscriber
value:
cellcli -e "alter cell snmpSubscriber=((host='[host name]',port=[port}),(host='[host name]',port=[port]))"
Where [host name]
and [port]
values are Agents monitoring your cell targets.
Note:
Take special care not to overwrite existing settings forsnmpSubsriber
. If there are existing subscribers, then append the agent subscriptions. For example, if the cellcli -e list cell attributes snmpSubscriber
command returned:
# cellcli -e list cell attributes snmpSubscriber ((host=ilm-asr1.example.com,port=162,community=public,type=asr))
Then you must append the Agent subscriptions:
#cellcli -e "alter cell snmpSubscriber=((host=ilm-asr1.example.com,port=162, community=public,type=asr),(host='[host name]',port=[port]),(host='[host name]',port=[port]))"
Set the notificationMethod
value:
cellcli -e "alter cell notificationMethod='snmp,mail'"
Set the notificationPolicy
value:
cellcli -e "alter cell notificationPolicy='Critical,Warning,Clear'"
dcli
(optional)The SNMP configuration commands can be run using dcli
to perform the configuration in batch on all storage cells:
Set the snmpSubscriber
value:
dcli -g cell_group -l root "cellcli -e \"alter cell snmpSubscriber=((host='[host name]',port=[port]),(host='[host name]',port=[port]))\""
Where [host name]
and [port]
values are Agents monitoring your cell targets.
Set the notificationMethod
value:
dcli -g cell_group -l root "cellcli -e \"alter cell notificationMethod='snmp,mail'\""
Set the notificationPolicy
value:
dcli -g cell_group -l root "cellcli -e \"alter cell notificationPolicy='Critical,Warning,Clear'\""
Open a new terminal and verify whether the SSH connectivity was successfully established:
ssh -l cellmonitor <cell_ipaddress> cellcli -e 'list cell detail'
If you are not prompted for any password, then you can assume that the connectivity is established.
If you are asked to confirm whether you want to continue connecting, specify Yes.
To remove the subscription, use the ALTER CELL
command again by excluding the host name that you want to remove from the snmpsubscriber
list.
Notes:
The SNMP receivelet listens on a single address and port for all monitored targets. The port is the UDP port with the same number as the TCP port used in theEMD_URL
.
By default, the SNMP receivelet listens on all addresses; if the property SnmpRecvletListenNIC
is set in the emd.properties
file, the receivelet will attempt to resolve the value as either a name or IP address, and listen on only that address.
This parameter is independent of AgentListenOnAllNICs
and EMD_URL
because in some installations, the Agent may need to communicate with the OMS and with managed targets on different networks.
The SNMP configuration for Enterprise Manager monitoring of InfiniBand Switches is done automatically as part of the Enterprise Manager guided discovery process. It is good practice, however, to verify that SNMP configuration has been successful.
Notes:
For Enterprise Manager Cloud Control 12c Bundle Pack 1 (BP1):
The SNMP trap setup for Exadata Storage Cells is done automatically after finishing the guided discovery process.
For Exadata plug-in Release 12.1.0.2 and later:
During the discovery process, you can optionally provide the necessary root
credentials to set up SNMP trap for Exadata Storage Cells. If you have done so, then you can skip the remaining steps of this section and proceed with Configure the Compute Node ILOM SNMP for Enterprise Manager Monitoring.
To configure (if necessary) and verify the SNMP configuration for an InfiniBand Switch:
Log in to the InfiniBand Switch ILOM web interface using the URL https://<ib_switch_hostname>
as root.
Note:
Try using Internet Explorer if the console does not display all fields/values in your browser of choice.Click Configuration, then System Management Access, and finally SNMP.
Ensure the following values are set:
State=Enabled Port=161 Protocols=v1,v2c,v3
If you need to make changes, make sure you click Save.
Click Alert Management.
If not already listed, for each Agent that monitors the InfiniBand Switch target, select an empty alert (one that has the Destination Summary 0.0.0.0, snmp v1, community 'public'
) and click Edit. Provide the following values:
Level = Minor Type = SNMP Trap Address = [agent compute node hostname] Destination Port = [agent port] SNMP Version = v1 Community Name = public
Click Save.
Verify the InfiniBand Switch SNMP configuration for Enterprise Manager monitoring:
snmpget -v 1 -c <community_string> <hostname_of_IB_switch> 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5
For example:
$ snmpget -v 1 -c public my_IB_switch.my_company.com 1.3.6.1.4.1.42.2.70.101.1.1.9.1.1.5 SNMPv2-SMI::enterprises.42.2.70.101.1.1.9.1.1.5 = INTEGER: 1
Note:
If the Timeout message is displayed as a output for the above command, then it means that the InfiniBand Switch is not yet configured for SNMP.Note:
To remove the subscription:echo "set /SP/alertmgmt/rules/12 destination='0.0.0.0' destination_port=0" | spsh
You can set up SNMP for InfiniBand Switch targets using the Enterprise Manager Cloud Control console:
Navigate to the IB Network target (not the individual switches) and select Administration.
Select the IB Switch target type, then one of the IB Switch targets.
Select the Setup SNMP Subscription command, then select the management agent URL that monitors the InfiniBand switch target from the Agent URL list. Click Next.
Provide credentials for the InfiniBand switch. Click Next.
Review the details you provided. If there are no further changes, then click Submit.
Perform steps 1-5 for both the Monitoring Agent and Backup Monitoring Agent of the InfiniBand switch target.
The compute node ILOM targets are responsible for displaying a number of disk failure alerts for their respective compute node that are received as SNMP traps. For Enterprise Manager to receive those traps, the /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl
script must be run to configure SNMP subscriptions for the agents that have been configured to monitor the compute node ILOM targets. This step is applicable to Exadata Plug-in Release 12.1.0.2 and later.
The exadata_mon_hw_asr.pl
script is run as the root user with the -set_snmp_subscribers
parameter to add SNMP subscribers. For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -set_snmp_subscribers "(host=hostname1.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444),(host=hostname2.mycompany.com,port=3872,community=public,type=asr,fromip=12.345.67.890)" Try to add ASR destination Host - hostname1.mycompany.com IP - 11.222.33.44 Port - 3872 Community - public From IP - 22.333.44.555 Try to add ASR destination Host - hostname2.com IP - 11.111.11.111 Port - 3872 Community - public From IP - 22.333.44.555
The script needs to be run on each compute node:
The host
values should be the hostnames of the agents configured to monitor the compute node ILOM target associated with the compute node.
The fromip
values should be the IP address of the compute node that the compute node ILOM target is associated with.
For example, if you have an X2-2 machine with compute node targets edbm01db01
through edbm01db08
and associated compute node ILOM targets edbm01db01-c
through edbm01db08-c
, then you would need to run the script once on each compute node - therefore, the script would be run eight times in total.
On compute node edbm01db01
, the host
and port
values would be the hostnames and ports of the agents monitoring compute node ILOM target edbm01db01-c
and the fromip
value would be the IP address of the compute node itself, edbm01db01
.
On compute node edbm01db02
, the host
and port
values would be the hostnames and ports of the agents monitoring compute node ILOM target edbm01db02-c
and the 'fromip' value would be the IP address of the compute node itself, edbm01db02 ... and so on.
This is a good example of where Manual selection of Management Agents for targets is useful. If the first two compute nodes are always the Monitoring Agent and Backup Monitoring Agent, then it is easy to work out the values needed for -set_snmp_subscribers
parameters, the host
and port
values would be the same for all compute nodes.
Important Note:
Theexadata_mon_hw_asr.pl
script, overwrites any existing SNMP subscriptions. While setting the SNMP subscribers, make sure that current subscribers are included in the new list of subscribers.
It is possible to use the exadata_mon_hw_asr.pl
script to get the current set of subscribers using the -get_snmp_subscribers
parameter.
For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -get_snmp_subscribers -type=asr
Suppose the current list is:
(host=hostname1.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=hostname2.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444)
Then new subscriptions can be added using the following command:
/opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -set_snmp_subscribers "(host=asrhostname1.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=asrhostname2.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=hostname1.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444), (host=hostname2.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444)"
After adding the new subscribers, run the command exadata_mon_hw_asr.pl
script with the -get_snmp_subscribers
parameter to get the list of SNMP subscribers and verify the new SNMP subscriptions were added successfully. For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -get_snmp_subscribers -type=asr (host=host1.mycompany.com,port=162,community=public,type=asr,fromip=10.10.10.226), (host=host2.mycompany.com,port=162,community=public,type=asr,fromip=10.10.10.226), (host=host3.mycompany.com,port=3872,community=public,type=asr,fromip=10.10.10.226) ,(host=host4.mycompany.com,port=3872,community=public,type=asr,fromip=10.10.10.226)
To verify that Alerts can be successfully raised and cleared for the Oracle ILOM Server targets, perform the following steps:
Log in to the Enterprise Manager Cloud Control console as an administrator.
Click Targets and select Exadata. Select one of the Oracle ILOM Server targets using the Target Navigation pane.
The ILOM target page will be displayed, showing the current status of that target as well as any incidents that have been raised for it.
Raise an alert manually from the ILOM Server being validated. Run following command as root
on the first database server in the cluster:
ipmitool -I lan -H sclczdb01-c -U oemuser -P oempasswd -L OPERATOR event PS0/VINOK fail assert
The output will be similar to:
Finding sensor PS0/VINOK... ok 0 | Pre-Init Time-stamp | Power Supply #0x65 | State Asserted
After running the above command, wait a few minutes then refresh the ILOM target page. An incident should appear in the Incidents section.
Clear the alert raised in step 3. Run the following command as root
on the first database server in the cluster:
ipmitool -I lan -H sclczdb01-c -U oemuser -P oempasswd -L OPERATOR event PS0/VINOK nofail deassert
The output will be similar to:
Finding sensor PS0/VINOK... ok 0 | Pre-Init Time-stamp | Power Supply #0x65 | State Deasserted
After running the above command, wait a few minutes then refresh the ILOM target page. The incident that was raised in step 3 should show as cleared in the Incidents section.
Note:
Do not forget to clear the alert raised in step 3, as it was raised for testing only and did not reflect a true fault condition!Repeat for the remaining configured ILOM service processors in your Oracle Database Machine.
The Cisco Ethernet Switch must be configured to allow the Agents that monitor it to be able to both poll the switch and to receive SNMP alerts from the switch. To allow this, perform the following steps (swapping the example switch name dm01sw-ip
with the name of the Cisco Ethernet Switch target being configured):
Login to the Cisco switch and enter Configure mode:
# telnet dm01sw-ip User Access Verification Password: dm01sw-ip> enable Password: dm01sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. dm01sw-ip(config)#
Enable access to allow the Agents monitoring Cisco Switch target to poll the switch.
In the command, [EMagentIPaddr]
is the IP address of the server where the Enterprise Manager Agent is running. The SNMP community specified must match the value provided when configuring the Cisco Switch target:
dm01sw-ip(config)# access-list 1 permit [EMagentIPaddr] dm01sw-ip(config)# snmp-server community <community_string> ro 1
Set the monitoring Agent as the location where SNMP traps are delivered. The SNMP community specified must match the value provided during Enterprise Manager Cisco Switch Management Plug-In setup:
dm01sw-ip(config)# snmp-server host <EMagentIPaddr> version 1 <community string> udp-port [EMagentRecvltListenPort]
Where [EMagentRecvltListenPort]
is the EMD_URL
port of the emagent
or SnmpRecvletListenNIC
property value if it is enabled.
Configure the Cisco Switch to send only environmental monitor SNMP traps:
dm01sw-ip(config)# snmp-server enable traps envmon
Verify settings and save the configuration:
dm01sw-ip(config)# end dm01sw-ip# show running-config dm01sw-ip# copy running-config startup-config
Run the snmpwalk
command line utility or equivalent tool to verify the Cisco Switch configuration.
Run the following command to fetch and display the data from the Cisco switch:
snmpget –v 1 –c <community_string> <hostname_of_cisco_switch> 1.3.6.1.4.1.9.2.1.56.0
Note:
If a timeout message is displayed as an output for the above command, then it means that the Cisco Switch is not yet configured correctly.To enable Enterprise Manager to collect metric data and raise events for the PDU target, you must configure the PDU to accept SNMP queries from the Agents that monitor the PDU target. Also, appropriate threshold values for different phase values needs to be set on the PDU.
This section assumes that this is a first time configuration of the PDU. SNMP must be enabled and the trap section completed. Granting SNMP access to a different monitoring Agent IP address is an example where only the "Trap Host Setup" section needs to be changed.
Log in to the PDU network interface through a browser at http://<pdu-name>
, for example: http://edbm01-pdu1.example.com
Click Net Configuration, then log in again.
Scroll down until you reach the SNMP section of the frame.
Note:
The network interface for the PDU is a frame within a window. In order to scroll down on this page, you must see the scroll bar for the PDU frame as well as the outside scroll bar for the browser in which you accessed the PDU.If your PDU is not SNMP-enabled, select the SNMP Enable check box, then click Submit.
Scroll to the NMS region of the frame.
Enter the following in Row 1 under NMS:
IP: Enter the IP address of the first monitoring Agent
Community: Enter "public"
Click Submit.
For details on configuring the PDU thresholds settings, see PDU Threshold Settings for Oracle Exadata Database Machine (See Doc ID 1299851.1) in My Oracle Support.
Use the snmpwalk
command line utility or equivalent tool to verify the PDU configuration.
Run the following command to fetch and display the data from PDU:
snmpget –v 1 –c <community_string> <hostname_of_pdu> 1.3.6.1.4.1.2769.1.2.3.1.1.1.0
Note:
If a timeout message is displayed as an output for the above command, then it means that the PDU is not yet configured correctly.The KVM needs to be configured to send SMNP traps to the Agents that monitor it.
Configure the KVM to send traps to monitoring agent host and receivelet listening port, the port value of EMD_URL
property from the $ORACLE_HOME/sysman/config/emd.properties
file (for example, 3872):
Log in to the KVM management console through a browser at https://<kvm-name>
, for example: https://edbm01sw-kvm.example.com
Click SNMP under Target Devices in the left-hand navigation bar.
Select the Enable SNMP check box and enter the appropriate community name in the Read, Write, and Trap fields (for example, enter "public").
Click Save.
Click Destinations under SNMP in the left-hand navigation bar.
Enter the IP address of the Agent monitoring the KVM target.
Click Save.
Run the snmptrapd
command line utility or equivalent tool to verify the KVM configuration.
Follow the steps below on the monitoring Agent host:
Log in as root.
Run the following command:
service snmptrapd stop
Rename the /etc/snmp/snmptrapd.conf
file to be /etc/snmp/snmptrapd.conf_bk
to create a backup.
Run the following command:
snmptrapd -p 162
Reboot the KVM (to generate SNMP traps).
The terminal where the snmptrapd
command was run should display the received trap which is generated by the KVM due to the reboot.
If step 6 is confirmed, the KVM has been configured correctly. Rename the backup file and restart the snmptrapd
service:
Use Ctrl+C to stop the snmptrapd
command that was started in step 4 above.
Rename the /etc/snmp/snmptrapd.conf_bk
backup file to be /etc/snmp/snmptrapd.conf
Run the following command:
service snmptrapd start
Follow the steps below to validate that the monitoring Agent can receive the SNMP traps generated by the KVM and convert the received traps to Enterprise Manager events.
Log in to the KVM.
Click Overview on the left side of the screen under Unit View, Appliance.
Click Reboot.
Click OK to confirm KVM reboot in the window that asks if you want to continue.
In the Enterprise Manager Cloud Control console, go to the KVM target page. An Incident should be displayed for the KVM reboot event in the Incidents and Problems section of the page.
You can create a Database Machine dashboard to monitor the performance and usage metrics of the Database Machine system, its sub-components, as well as all the database system components residing on the Database Machine.
Dashboard Creation for Exadata Database Machine Plug-in Release 12.1.0.3 and Later
For Exadata Database Machine plug-in Release 12.1.0.3 and later, create the Database Machine Dashboard:
Log in to Enterprise Manager.
From the Enterprise Manager home page, click the Enterprise menu. Select the Reports submenu, then Jobs, and finally Activity.
Select Database Machine Services Dashboard from the drop-down menu next to the Create Job option.
Click Go.
Enter a name of the job (for example, CREATE_DBM_JOB_1
).
Click Add and select the DBMachine target. After adding, make sure the target is selected by clicking on the check box next to it.
Click the Parameters tab.
Three options are provided through the drop-down:
Select Create if it is a new report.
Select Update for updating an existing report with new components.
Select Cleanup to remove services created by the Create job executed earlier.
Finally, click Submit to perform the operation.
A message CREATE_DBM_JOB_1 submitted successfully
should display.
You can monitor the job by clicking on the link corresponding to the job.
Dashboard Creation for Exadata Database Machine Plug-in Release 12.1.0.2 and Earlier
For Exadata Database Machine plug-in Release 12.1.0.2 and earlier, create the Database Machine Dashboard:
Go to the following directory:
/opt/oracle.SupportTools/emkit/exadata/dashboard/
Run the following script:
perl DashboardOperations.pl
Running this script will generate one Dashboard report per Database Machine system.
Note:
For details of script usage, please see the Dashboard Report Script README.Creation of the dashboard assumes the use of the OMS emkit
and can only be run from the OMS server that the kit was deployed on.
How to Make the Report "Public"
The generated report is accessible only by the Enterprise Manager user who creates it. To make the report public:
Select the dashboard report from the list of reports shown after following the steps mentioned above.
Click Edit.
Select the Run report using target privileges of the report owner option under Privileges section in General tab.
Click the Access tab.
Select the Allow viewing without logging in to Enterprise Manager option.
Click OK.
Find All Database Machine Reports
To find all Database Machine reports:
Log in to Enterprise Manager.
From the Enterprise Manager home page, click the Enterprise menu. Select Reports, then Information Publisher Reports.
Search for the report name. Dashboard report names, one for each Database Machine, are displayed in the following format:
[DBMachine Name]_DASHBOARD_REPORT
Click on the report to view the dashboard report.