Oracle® Enterprise Manager Cloud Control Getting Started with Oracle Fusion Middleware Management 12c Release 2 (12.1.0.2) Part Number E24215-03 |
|
|
PDF · Mobi · ePub |
This chapter describes the errors you may encounter while deploying and using JVM Diagnostics and the workaround steps you can follow to resolve each of them. It contains the following sections:
This section lists the errors that show the status of the JVM Diagnostics Manager.
Table 15-1 Cross Tier Functionality Errors
Error Message | Workaround Steps |
---|---|
DBWait link not displayed on JVM Threads Real Time Analysis page. No data displayed in Top DBStates / SQLs tables. |
Cross tier functionality errors may occur due the following:
If the database credentials are incorrect:
If database agent errors occur, ensure that the database agent is running on a machine on which the database is installed with the correct IP address and port number.
|
This section lists the errors that occur when you run the deployment script.
Table 15-3 Script Execution Errors
Error Message | Workaround Steps |
---|---|
ScriptException: Error occured while performing deploy: The action you performed timed out after 600,000 milliseconds |
This error occurs when you are deploying the JVM Diagnostics Agent. To resolve this issue, check if the lock for the target WebLogic domain Administration Console has already been acquired. If it has been acquired, release it and run the script again.
|
This section lists loadheap errors.
Error Message | Workaround Steps |
---|---|
glibc detected * free(): invalid next size (fast): 0x0965d090" ./loadheap.sh: line 237: 32357 Aborted ./bin/${bindir}/processlog in=$infile hdr=${sumdata} obj=${objdata} rel=${reldata} root=${rootdata} osum=${objsumdata} rrel=${rootrel} heap=${heap _id} skip=$skipgarbage db=$dbtype $* Error processing file /tmp/heapdump6.txt |
Check if the heapdump operation has been successfully completed. Open the |
Heapdump already in progress, cannot take another heapdump |
Check if the |
loadheap.sh created unusable unique indexes. |
Run the |
This section lists the user interface errors.
Table 15-5 JVM Diagnostics UI Page Errors
Error Message | Workaround Steps |
---|---|
JAM Console: Socket timed out after recv -- client adc2100083.us.oracle.com:7001 is not Active [0] secs JAM Console jamlooptimeout = [3]JAM CONSOLE: JVM 1 is not active JAM Cons Err Processing Request: 128 JVM 1 is not active jamDAL: jamreq returned 128 return status < 0 from jamDalInst.processRequest |
To resolve this error, increase the Agent Request Timeout (secs) and Agent Loop Request Timeout (secs) |
Agent is up and running but is not displayed in the real time pages. |
If the log file shows To resolve this issue, download the latest |
You do not have the necessary privileges to view this page |
Ensure that you have the required JVM Diagnostics Administrator or User privileges to view the JVM Diagnostics data. |
This section lists some of the questions you may have while using JVM Diagnostics. It includes the following:
Location of the JVM Diagnostics Logs
JVM Diagnostics Manager Status
JVM Diagnostics Agent Status
Monitoring Status
Creating Less Privileged Users
Usage of Try Changing Threads Parameter
Significance of Optimization Levels
Manually Deploying the JVM Diagnostics Agent
Log Manager Level
Repository Space Requirements
You can find the JVM Diagnostics logs in the following locations:
The JVM Diagnostics Manager Log file is located at $EMGC_JVMDMANAGER1/logs/EMGC_JVMDMANAGER1.out
UI related errors are logged in:
$T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
$T_WORK/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.log
Communication errors between the JVM Diagnostics Manager and the Console are logged in $T_WORK/gc_inst/em/EMGC_OMS1/sysman/log/emoms.log
To check the status of the JVM Diagnostics Manager:
From the Setup menu, select Middleware Diagnostics, then click Setup JVM Diagnostics.
Check the JVM Diagnostics Agent log file to verify the connection between Agent and the Manager. If you see an error - JAM Agent ERROR: Cannot connect to Console:Connection refused
, this indicates that the JVM Diagnostics Manager is not running.
Check if the message JAM Console: Agent connection from:[Hostname]
is present in the JVM Diagnostics Manager log file. If this message appears, it indicates that the JVM Diagnostics Manager is running and is connected to the Agent.
To check the status of the JVM Diagnostics Agent:
From the Targets menu, select Middleware, then click on a Java Virtual Machine target. Select the Live Thread Analysis option from the Java Virtual Machine menu. Check the JVM Status in the Connected JVMs table.
If the status is Not Active, this indicates that the Agent is not connected to the Manager. Check the agent logs to verify if it is running and the IP address and port number of the Manager is correct.
If the status is No AD4J Agent Deployed, the JVM Diagnostics Agent must be deployed on that JVM.
If the JVM Diagnostics Agent is running, the active threads data must be visible. If the JVM Diagnostics Agent is not running, you will see a message - JVM is inactive, Please try again after some time
.
To verify if the JVM Diagnostics Manager is monitoring the data:
From the Setup menu, select Middleware Diagnostics, then click Setup JVM Diagnostics in the Middleware Diagnostics page. In the JVMD Configuration page, verify that the Enable Monitoring check box is checked.
Navigate to the Monitoring page under Setup and check if monitoring status is On for the Pool to which the JVM being monitored belongs.
Navigate to the JVM Pools page under Setup and verify if the Poll Enabled check box has been checked for the Pool to which the JVM being monitored belongs. Monitoring should now be enabled.
You can run the create_jvm_diagnostic_db_user.sh
script if you want to create less privileged users who can only load heaps using the loadHeap
script.
This parameter should be used only when the JVM is highly active.
The JVM Diagnostics Agent supports three optimization levels:
Level 0 indicates that the JVM Diagnostics Agent is using a JVMTI based engine. This level is supported for JDK 6 series on almost all supported platforms.
Level 1 is a hybrid between level 0 and level 2. It is supported only for very few JDKs on selected platforms.
Level 2 uses Runtime Object Analysis technique for monitoring as it is efficient at run time.
You can customize the JVMD Agent deployment in the production environment by running custom provisioning scripts.
After the OMS has been installed, the jvmd.zip
file can be found in the plugins/oracle.sysman.emas.oms.plugin_12.1.0.0.0
directory in the Middleware installation directory. The zip file contains a set of scripts in the customprov
directory. Details on using these scripts are described in the README.TXT
present in the same directory. To use the custom provisioning scripts, follow these steps:
From the Setup menu, select Middleware Diagnostics, then click the Setup JVM Diagnostics in the Middleware Diagnostics page. Click the Register Databases tab and download the jamagent.war
file
Make a copy of the deployment profile that includes the location of the downloaded jamagent.war
, domains, and server details.
Run the Perl script
on the deployment profile which will deploy the JVMD Agent to all the specified servers.
The default log manager level is 3. You can temporarily increase this to a higher level if you encounter some issues. Log levels 1 to 5 are supported where:
1 - Error
2 - Warning
3 - Info
4 - Debug
5 - Trace
For monitoring data, Oracle recommends 50 MB per JVM per day with the default setting of a 24 hour purge interval. This amount can vary based upon runtime factors (e.g depth of call stacks, etc.) within your environment. Hence, you must check the tablespace growth periodically and if required, you may need to change the space requirements. This will ensure that database growth due to standard monitoring will occur smoothly without sudden spikes. Tablespace sizing can be affected by the following:
Heap Dumps: Analyzing heaps requires a large amount of tablespace. As a standard practice, we recommend that you must have 5 times the size of heap dump file being loaded in your tablespace. Since you know the size of your dump file, make sure that there is adequate space to accommodate the dump file before it is loaded into the database.
Thread Traces: While these are smaller than heaps. they are loaded into the database automatically when a user initiates a trace at the console. The size of these threads can vary dramatically depending on the number of active threads during the trace, the duration of the trace, and the sample interval of the trace. This should usually be under 100MB but if several thread traces have been initiated, it could fill up the database quickly. Before initiating the traces, you must ensure that there is adequate space in the database.