How to find which Java thread on WebLogic that causes high CPU utilization on Linux

Scope

Sometimes your Linux server is having close to 100% CPU utilization. It is easy and quick to find which process is spending more CPU by running the top command. It takes a few more steps to establish which Java(TM) thread is using CPU, and what that thread is doing. This blog will explain the procedure for gather evidence to your developers that their code is too CPU intensive.

Scenarios

I have tried this procedure using the following versions of Oracle® WebLogic and Java(TM):

• Oracle® WebLogic 12.1.3 with Oracle® Java(TM) SE 7 - jdk1.7.0_79
• Oracle® WebLogic 10.3.6 with Oracle®Java(TM) SE 7 - jdk1.7.0_79
• Oracle® WebLogic 10.3.6 with Oracle® Java(TM) SE 6 - jdk1.6.0_45
• Oracle® WebLogic 10.3.6 with Oracle® JRockit(R) 6 - R28.2.7

These guidelines would also work on other servers using Oracle® Java(TM) SE.

Overview

The procedure for finding the most CPU intensive thread is basically the same for each combination of Oracle® WebLogic and Java(TM):

• Find the ID of the process that has highest CPU utilization
• Find the ID of the thread that has highest CPU utilization based on the process ID found above.
• Take a thread dump of the process based on the process ID found above.
• Search for the thread ID in the thread dump
• Identify the code that causes issues

Find the ID of the process that has highest CPU utilization

Use the top command and look at the CPU% column to identify which PID has highest CPU utilization. In the case below, the PID with 7253 uses most CPU%.

Find the ID of the thread that has highest CPU utilization based on the process ID found above

Again we can use the top command to find out which threads are spending most CPU% with the PID as input. The command will be:

top -H -p <PID>

Take a thread dump of the process based on the process ID found above

Use the jstack <pid> command with Java HotSpot(TM) or the jrcmd <pid> print_threads command with Oracle® JRockit(R) to get a thread dump. For instance:

$JAVA_HOME/bin/jstack 7253 > /tmp/stack.txt

Search for the thread ID in the thread dump file

Some tools will write the thread IDs as decimal numbers and others will write it as hexadecimal numbers. In this blog post I used Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode) which wrote the thread IDs as hexadecimal numbers. When testing Oracle® JRockit(R) however, my thread IDs were written with decimal numbers. 

From the list of thread IDs above, I converted the three thread IDs with highest CPU% to hex:

[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7255
1c57
[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7286
1c76
[oracle@localhost dom_12c_hotspot7]$ printf "%x\n" 7291
1c7b

I can now search for 1c57, 1c76 and 1c7b in the /tmp/stack.txt file I created.

Nothing exciting about 1c57 as it was only the "VM Thread":

The other thread ID 1c76 is coming from my custom code:

The third thread ID 1c7b is also coming from my custom code, and that thread is even STUCK.:

This was just an example on how to find threads with high CPU utilization. The code deployed was written poorly on purpose to cater for this article. Finding the performance issue on a real life application will probably prove to be more complex. In case there are multiple threads that fights over the CPU, then it is probably better to do live analyses using a tool like JVisualVM.