7 Troubleshooting the Connector

Diagnosing the Problem

Follow these steps to diagnose a problem.

1. Verify that the SCOM Web Service has been successfully started and the WSDL for the SCOM Web Service can be accessed from the machine where it is installed. Performing the following steps to do this:

   1. Perform the steps in Using the Correct URL for SCOM Web Service Operations to determine the URL used by the SCOM Web Service.

   2. Open a browser and paste the URL from the previous step in the address window. Append ?WSDL to the end of the URL and attempt to load the URL.

   3. The WSDL should be loaded if the web service is operational.

      Note:

      A WSDL is an XML file that describes the web service.

   4. If the WSDL cannot be loaded, this indicates the web service had startup issues. See Troubleshooting Web Service Startup Errors to diagnose the problem.

2. Verify that the WSDL for the SCOM Web Service can be accessed from the machine where the Enterprise Manager server is installed.

   1. Open a browser at the Enterprise Manager server and copy the URL from step 1-b above to the address window. The hostname for the URL will be localhost. Change localhost to the actual hostname or IP address of the machine where the SCOM Web Service is installed. If you specify a hostname, you must make sure that the hostname is recognized at the Enterprise Manager server machine. This can be done using the ping command.

      For example, if the Web Service is installed on the server with a hostname of SCOMServer01 and the URL listed in framework.log is:

      http://localhost:8080/services/SCOM2007/EventService

      The URL used at the Enterprise Manager server machine would be:

      http://SCOMServer01: 8080/services/SCOM2007/EventService

   2. Attempt to load the WSDL by appending ?WSDL at the end of the URL. If the WSDL cannot be loaded, either the hostname is not recognized at the Enterprise Manager machine or there is a connectivity issue between the two machines. If you specified a hostname, try using the IP address instead of the hostname in the URL. If it still will not load, you have a connectivity problem. You will need to consult with your IT department to resolve this issue.

3. Verify that the SCOM Connector specifies the correct URL for the createEvent and updateEvent operations.

   1. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.

   2. From the Enterprise Manager console, click Setup.

   3. Select Extensibility in the pane.

   4. Select Management Connectors in the left pane.

   5. The Management Connectors page appears, which shows the installed connectors.

   6. Click the Configure icon associated with the SCOM Connector. This invokes edit mode, enabling you to configure the connector.

   7. Verify that the URL identified in step 2-b is specified for the createEvent and updateEvent operations. Also verify the URLs listed for the setup, initialize, and uninitialize operations. The URL is basically the same as the URL listed above except it is for the SCOMService instead of the EventService. Listed below is the URL that would be used based on the previous example.

      http://SCOMServer01: 8080/services/SCOM2007/SCOMService

   8. If any of the operations are incorrect, change to the correct URL.

      Note:

      Do not append the WSDL to the end of the URL that is specified here.

   9. Click OK.

4. If there were no errors found in the previous steps, this rules out connectivity issues between the connector and the SCOM Web Service. The problem must lie in the SCOM Web Service or the Oracle SCOM Agent. See Troubleshooting Web Service Operations Errors for information on diagnosing errors.

Troubleshooting Web Service Startup Errors

To identify the cause of a startup failure, navigate to the adapters/log directory in the SCOM Web Service install directory and open the framework.log file in a text editor. Search for Exception to find any errors in the file. If the file does not exist, it indicates that there is a problem locating or executing the JVM. See JVM Errors for information about resolving JVM issues.

Listed below are some possible Exceptions, an explanation of the root cause, and a description of the solution.

java.net.BindException: Address already in use: bind

This error indicates that the web service could not start because of a port conflict. There are two possible causes for this error.

1. Another application is using a port that the Web service is configured to use. If the web service is configured to use SSL, the port number is 8443. If it is not configured to use SSL, the port number is 8080.

   There are two possible solutions to this. You can change the other application to use a different port or you can change the SCOM Web Service to use a different port. To change the SCOM Web Service to use a different port, see Changing Default Port Numbers.

2. There is an instance of the Web service already running. If this is the case then there is no change required. You should only run one instance of the Web service at a time.

org.springframework.beans.factory.BeanInitializationException: Could not load properties; nested exception is java.io.FileNotFoundException: … framework.properties (Permission denied)

This error indicates that the web service could not start because the permissions on the framework.properties file in the conf directory were not set correctly.

To solve the problem, change the permissions to give the account or group under which the SCOM Web Service runs read and execute permissions.

For any other startup errors, consult Oracle Support.

JVM Errors

The SCOM Web Service requires version 1.6 of the JVM. If there are multiple versions of the JVM installed on the machine, it is possible that an older version of the JVM is being executed whenever the web service starts.

On UNIX systems, the JAVA_HOME environment variable must be set to the directory where Java 1.6 is installed in the shell where the web service is started. To properly start the web service on a UNIX platform, perform the following:

1. Set the JAVA_HOME environment variable to the Java 1.6 install directory.

2. Navigate to the adapters/bin subdirectory in the Web service install directory.

3. Execute the ./service.sh start command.

On Windows systems, perform the following to insure that Java 1.6 is used when starting the web service.

1. Navigate to the adapters/bin subdirectory in the Web service install directory.

2. Run the iWaveAdaptersw.exe executable.

3. Click the Java tab.

4. Be sure the Use Default check box is not checked.

5. In the Java Virtual Machine box, specify the path to the jvm.dll file in the Java 1.6 install directory.

6. Click OK.

Using the Correct URL for SCOM Web Service Operations

Perform the following steps to identify and configure the connector to use the correct URL for SCOM Web Service operations.

1. Open a command terminal on the system where the SCOM web service is installed.

2. Change the working directory to the adapters/log directory in the SCOM web service installation directory.

3. Open the framework.log file in a text editor.

4. Go to the bottom of the file and search backwards for the string Setting the server's publish address to be. Continue searching backwards until you find the URL that contains EventService.

   You should specify the URL listed here for the createEvent and updateEvent operations, with one exception. You must replace the host name of localhost in the URL with the actual host name or IP address of the system where the SCOM Web Service is installed.

5. Go to the bottom of the file and search backwards for the string Setting the server's publish address to be. Continue searching backwards until you find the URL that contains SCOMService.

   You should specify the URL listed here for the setup, initialize and uninitialize and operations, with one exception.You must replace the host name of localhost in the URL with the actual host name or IP address of the system where the SCOM Web Service is installed.

6. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.

7. From the Enterprise Manager console, click Setup.

8. Select Extensibility in the pane.

9. Select Management Connectors in the left pane.

10. Click the name of the SCOM Connector.

    The General tab of the Configure Management Connector page appears.

11. Verify that the URL identified in step 4 is specified for the creatEvent and updateEvent operations.

12. Verify that the URL identified in step 5 is specified for the setup, initialize and uninitialize operations.

13. If any of the operations are incorrect, change to the correct URL and click OK.

Troubleshooting Web Service Operations Errors

Perform the following diagnostic steps if messages are not being generated or updated as expected in SCOM.

1. Verify that the event that was triggered is referenced in a rule that forwards events to the SCOM connector.

2. Verify that the OracleEnterpriseManager.Alert.Creator Management Pack has been imported into the SCOM server:

   1. Open the Ops Mgr 2007 console window and log in.

   2. In the Administration pane of the OpsMgr console, select Administration, then Management Packs.

   3. Verify that the "OracleEnterpriseManager Alert Creator" Management Pack is listed.

   4. If the pack is not listed, import it as specified in Installing the Alert Creator Management Pack.

3. Determine the error that Oracle Enterprise Manager has reported.

   1. To do this you need to examine the log file for errors. Perform the following steps to locate errors in the log file:

      1.) Open the emoms_pbs.trc file in a text editor. The file is located in the log directory at the following location:

      <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/

      Where, <EM_INSTANCE_BASE> is the OMS Instance Base directory. By default, the

      OMS Instance Base directory is gc_inst, which is present under the parent directory of the Oracle Middleware Home.

      For example, if the Oracle Middleware Home is /u01/app/Oracle/Middleware,

      then the instance base directory is /u01/app/Oracle/Middleware/gc_inst, and the log and trace files are available in /u01/app/Oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/ directory path.

      2.) Go to the bottom of the file and search backwards for Caused by. Lines that start with Caused by contain error information. The error information comes after the text in the line that reads oracle.sysman.emSDK.webservices.outbound.WSInvokeException: caught WebServiceException :.

4. Diagnose the problem based on the error information. See Errors from Oracle Enterprise Manager for information on troubleshooting common error events.

Errors from Oracle Enterprise Manager

This section provides cause and solution information on troubleshooting common error messages. Find the error message in Table 7–1 that matches your error message, then refer to the corresponding section(s) indicated under Possible Cause for instructions to diagnose and correct the problem.

Invalid Web Service Credentials

Cause

The user name or password for accessing the SCOM web service is incorrect.

Solution

1. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.

2. From the Enterprise Manager console, click Setup.

3. Select Extensibility in the pane.

4. Select Management Connectors from the menu. This will display the Management Connectors page.

5. Click the name of the SCOM Connector.

   This invokes edit mode, enabling you to configure the connector.

6. Correct the SCOM Web Service Username and SCOM Web Service Password fields, then click OK.

SSL Not Configured in Enterprise Manager

Cause

The SSL handshake between the Oracle Enterprise Manager Connector Framework and the SCOM web service failed. This failure occurs because Oracle Enterprise Manager is not configured correctly with the SSL certificate for the SCOM web service. The SSL certificate the SCOM web service uses must be imported into the Enterprise Manager key store. The certificate is either missing from the key store or does not match the SSL certificate provided by the SCOM web service.

Solution

Import the SSL certificate from the SCOM web service into the Enterprise Manager key store. See Configuring Enterprise Manager to use SSL for details on setting up Oracle Enterprise Manager with the SCOM SSL certificate.

SCOM Web Service Down

Cause

The SCOM web service is down.

Solution

Perform the following steps to check the status of the web service and start it if necessary.

If the SCOM web service is installed on a Unix system:

1. Open a command terminal on the system where the SCOM web service is installed.

2. Change the working directory to the adapters/bin directory in the SCOM web service installation directory.

3. Enter the following command:

   ./service.sh status

4. If the command indicates that the service is not running, enter the following command:

   ./service.sh start

If the SCOM web service is installed on a Windows system:

1. Open a command terminal on the system where the SCOM web service is installed.

2. Change the working directory to the adapters/log directory in the SCOM web service installation directory.

3. Open the framework.log file in a text editor.

4. Go to the bottom of the file and search backwards for the string iWave Adapter Framework. If the last occurrence found is iWave Adapter Framework Started, this indicates that the web service is started.

5. If the web service is not started, start the web service as specified in Running the Web Service on Windows.

Invalid IP Address

Cause

The IP address specified in the URL is invalid or the network is down.

Solution

Verify that the hostname/IP address configured for the connector is correct:

1. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.

2. From the Enterprise Manager console, click Setup.

3. Select Extensibility in the pane.

4. Select Management Connectors from the menu. This will display the Management Connectors page.

5. Click the name of the SCOM Connector.

   This invokes edit mode, enabling you to configure the connector.

6. Verify that the hostname/IP address specified in the URL for the createEvent and updateEvent operations are correct.

7. If the hostname/IP address is incorrect, provide the correct value, then click OK.

If the URLs specify a host name, make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.

If the hostname/IP address appears to be correct, try to ping the system where the SCOM web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.

Invalid Port Number

Cause

The port number specified in the URL is invalid.

Solution

Verify that the port number configured for the connector is correct:

1. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.

2. From the Enterprise Manager console, click Setup.

3. Select Extensibility in the pane.

4. Select Management Connectors from the menu. This will display the Management Connectors page.

5. Click the name of the SCOM Connector.

   This invokes edit mode, enabling you to configure the connector.

6. Verify that the port number specified in the URL for the createEvent, updateEvent, setup, initialize, and uninitialize operations are correct.

7. If the port number is incorrect, provide the correct value and click OK.

Invalid URL Path

Cause

The web service received the request and rejected it because an invalid path was specified in the URL.

Solution

Perform the following steps to test the URL the connector is using.

1. Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.

2. From the Enterprise Manager console, click Setup.

3. Select Extensibility in the pane.

4. Select Management Connectors from the menu. This will display the Management Connectors page.

5. Click the name of the SCOM Connector.

   This invokes edit mode, enabling you to configure the connector.

6. Select and copy the URL specified for the createEvent operation.

7. Open an internet browser on the system where the Oracle Enterprise Manager server is installed.

8. In the address window, enter the URL that was copied in step 6 above. Add ?wsdl to the end of the URL. The URL should appear similar to the following example:

   http://[Hostname]:8080/services/SCOM2007/EventService?wsdl

   [Hostname] is the actual host name or IP address where the SCOM web service is installed.

If the WSDL is loaded, this confirms that the URL is correct. If it fails to load, there is a problem with the URL. Perform the steps specified in Using the Correct URL for SCOM Web Service Operations to configure the connector to use the correct URL.

Firewall Blocking Access

Cause

A firewall is blocking access to the system where the SCOM Web Service is installed.

Solution

Contact your IT department to give Enterprise Manager access to the port used by the SCOM Web Service. Perform the steps specified in Using the Correct URL for SCOM Web Service Operations to determine the URL used by the SCOM Web Service. The port number specified in the URL is the port number the IT department should open in the firewall.

Unknown Host

Cause

The system does not recognize the host name specified in the URL.

Solution

You can use the following options to address this issue:

• Coordinate with the system administrator to change the system configuration to recognize the host name.

• Specify the IP address in the URL instead of the host name. To do this, perform the following steps:

  1. Determine the IP address of the system where the SCOM web service is installed.

  2. Log in to the Oracle Enterprise Manager console by entering a user name with a 'Super Administrator' role, entering the appropriate password, then click Login.

  3. Click the Setup link at the top right part of the window. The Overview of Setup page appears.

  4. Click the Management Connectors link on the left side of the window. The Management Connectors page appears, which shows the installed connectors.

  5. Click on the Configure icon associated with the Microsoft SCOM Connector. This invokes edit mode, enabling you to configure the connector.

  6. Change the host name to the IP address in the URL specified for the createEvent and updateEvent operations.

  7. Click OK.

Invalid XML Format

Cause

The connector framework could not process the request because the XSL file was formatted incorrectly. This problem should not occur unless the connector has been customized.

Solution

Examine any changes made to the XSL template files for mistakes that could have caused the problem. If you can't find the problem manually, load the XSL in a utility that performs XML validation.

SCOM Server Not Operational

Cause

The SCOM Agent could not insert the alert into SCOM because the wrong host name is configured for SCOM or the SCOM server is down.

Solution

Perform the following steps to determine and correct the root cause of the problem:

1. Verify that the host name or IP address listed in the error message is correct for the RMS system. If the host name or IP address are incorrect, perform the following steps to correct the configuration:

   1. Open Windows Explorer on the system where the SCOM Agent is located.

   2. Navigate to the bin directory in the SCOM Agent installation directory.

   3. Run the SCOMAgentConfig.exe utility to start the SCOM Agent Configuration Tool.

   4. Click Load to open a directory navigation window.

   5. Navigate to the SCOM Agent installation directory and open the SCOMAgent.cfg file.

   6. Click the Management Groups tab, then click Edit to display the Edit Management Group window.

   7. Correct the hostname/IP address in the Server field, then click Update.

   8. Click Save to save the changes to the configuration file.

   9. Click Exit to exit the utility.

   10. Stop and restart the SCOM Agent in IIS.

2. Verify that the following OpsMgr services are running based on the version of SCOM:

   preR2 OpsMgr Config Service OpsMgr Health Service OpsMgr SDK Service R2 System Center Data Access System Center Management System Center Management Configuration

Invalid SCOM API Credentials

Cause

The SCOM Agent could not send the alert to the SCOM server, because the credentials configured for accessing the SCOM API are invalid.

Solution

Perform the following steps to change the credentials for accessing the SCOM API:

1. Open Windows Explorer on the system where the SCOM Agent is located.

2. Navigate to the bin directory in the SCOM Agent installation directory.

3. Run the SCOMAgentConfig.exe utility to start the SCOM Agent Configuration Tool.

4. Click Load to open a directory navigation window.

5. Navigate to the SCOM Agent installation directory and open the SCOMAgent.cfg file.

6. Click the Management Groups tab, then click Edit to display the Edit Management Group window.

7. Correct the credential information in the Domain, Username, and Password fields, then click Update.

8. Click Save to save the changes to the configuration file.

9. Click Exit to exit the utility.

10. Stop and restart the SCOM Agent in IIS.

Wrong SCOM API Permissions

Cause

The SCOM Agent could not send the alert to the SCOM server, because the credentials configured for accessing the SCOM API do not have sufficient permissions.

Solution

Refer to Section 2.4.2, "Setting Up the Agent Account". This section provides the steps required to set up the account for accessing the SCOM API.

SCOM Agent Not Operational

Cause

The web service could not create an alert in SCOM because the SCOM Agent is not operational.

Solution

Open IIS Manager on the system where the SCOM Agent was installed, and start the web site for the Agent.

SCOM Agent Configuration

Cause

The web service could not connect to the SCOM Agent because the web service has an invalid configuration parameter. Either the URL for the SCOM Agent is incorrect or the credentials for accessing the SCOM Agent are invalid.

Solution

1. Verify that the URL for the SCOM Agent is correct. You should specify the URL that was provided at the end of the SCOM Agent installation. Note that if the host name in the URL is localhost and you are accessing it from another system, you need to replace localhost with the host name or IP address of the SCOM Agent installation machine.

   If you do not know the URL, you can determine it as follows:

   • If the SCOM Agent was installed as a web site, the address is:

     http://<IP>:<port>/Service.asmx
     

     ... where <IP> is the IP address, and <port> is the port number specified when installing the Agent.

   • If the SCOM Agent was installed as a virtual directory, the address is:

     http://<IP>:<port>/<vdir>/Service.asmx
     

     ... where <IP> is the IP address, <port> is the port number for the web service where the agent was installed, and <vdir> is the virtual directory name specified for the Agent.

2. Select a user name and password that are valid on the system where the SCOM Agent was installed.

3. Open a command window and change the working directory to adapters\endpoints\SCOM2007 in the SCOM web service installation directory.

4. Rerun the SCOM Web Service installer using the URL and credentials from the preceding steps. See Section 2.5.1.2, "Installing the Web Service on Unix" or Section 2.5.2.2, "Installing the Web Service on Windows", depending on your platform, for the procedure.

Management Pack Missing

Cause

The web service could not create an alert in SCOM because the OracleEnterpriseManager.Alert.Creator management pack has not been imported into SCOM.

Solution

Refer to Section 2.4.4, "Installing the Alert Creator Management Pack" for the steps required to import the management pack into SCOM.

Alert Create Timeout

Cause

The web service was able to insert an event in SCOM, but an alert was not created within the timeout period. This likely indicates that an error occurred in the alert generating rule and it was unloaded by SCOM. Whenever this occurs, the Ops Mgr Health Service generates an error followed by a warning in the Operations Manager log. The error entry begins with the following message:

A module reported an error 0x80070057 from a callback which was running as part of rule "Create.Default.Alert" running for instance "OracleEnterpriseManager Event Source" with id ...

The warning entry begins with the following message:

Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload ...

Solution

Restart the Windows service named “Ops Mgr Health Service” on the RMS system.