Generating JBoss Thread dumps

To diagnose issues for 100% CPU utilisation or Out of memory errors it is important to gather thread dumps at the time the problem is occurring.

The thread dumps will provide information on what process’ are holding onto resources which can then be used to identify the root cause.

In order to identify the processes, multiple thread dumps over a short period of time are required.  These will then show what threads are present over an extended period of time, rather than executing and closing as would be expected.  It is recommended that 5 consecutive thread dumps are recorded and supplied.

UNIX/Linux:

Note the process ID number of the Java process

jps –lv

With the Process ID from the previous step, save to a file and run the following bash script to generate the thread dumps:

#!/bin/sh

#

# Takes the JBoss PID as an argument.

#

# Create thread dumps a specified number of times and INTERVAL.

# Thread dumps will be in the file where stdout was redirected

# or in console output.

#

# Usage: sh ./threaddump_linux.sh <JBOSS_PID>

#

# Number of times to collect data.

LOOP=5

# Interval in seconds between data points.

INTERVAL=1

for ((i=1; i <= $LOOP; i++))

do

   kill -3  $1 > $1_$i.dump

   echo "thread dump #" $1 "." $i

   if [ $i -lt $LOOP ]; then

      echo "Sleeping..."

      sleep $INTERVAL

   fi

done

Windows:

Download PSTools from the Microsoft website: http://technet.microsoft.com/en-gb/sysinternals/bb897553.aspx and extract to c:\program files\PSTools

Create a bat file in the directory called threaddump_win.bat containing the following (changing folder locations where appropriate):

@echo off

:: Takes the JBoss PID as an argument.

::

:: Create thread dumps 5 times

:: Thread dumps will be output to the file <PID>_<#>.dump

::

:: Usage: threaddump_win.bat <JBOSS_PID>

::==================================================

:: main

::==================================================

ECHO ==================================================

ECHO  JBoss thread dump utility v2

ECHO ==================================================

ECHO.

Set JDK_BIN=C:\Program Files\Java\jdk1.7.0_07\bin

Set PSTOOLS_DIR=C:\Program Files\PSTools

Set LOG_DIR=C:\ThreadDump\Dumps

:: Set the PID for the threaddumps

CALL :SETPID

:: Check the environment

SET ENVIRONMENT_OK="TRUE"

CALL :ENVIRONMENT_CHECK

IF %ENVIRONMENT_OK%=="TRUE" (

ECHO environment check complete

ECHO.

) ELSE (

ECHO environment check failed, exiting script

GOTO END

)

:: Execute the thread dumps

SET STAMP=%DATE:/=%_%TIME:~0,2%%TIME:~3,2%%TIME:~6,2%

for %%i in (1 2 3 4 5) do "%PSTOOLS_DIR%\PSExec.exe" -s "%JDK_BIN%\jstack.exe" %PID% > "%LOG_DIR%\%STAMP: =0%_%PID%_%%i.dump"

:: Exit the script

GOTO END

::==================================================

:: check the environment setup

::==================================================

:ENVIRONMENT_CHECK

ECHO Checking environment 

:: Check for PSExec.exe

ECHO ...%PSTOOLS_DIR%\PSExec.exe

CALL :CHECK_EXIST %PSTOOLS_DIR%\PSExec.exe     

:: Check for jstack.exe

ECHO ...%JDK_BIN%\jstack.exe

CALL :CHECK_EXIST %JDK_BIN%\jstack.exe  

:: Check for output directory

ECHO ...%LOG_DIR%

CALL :CHECK_EXIST %LOG_DIR%

:: Exit the script

GOTO END

:CHECK_EXIST

IF NOT EXIST "%*" (

        SET ENVIRONMENT_OK="FALSE"

        ECHO.

        ECHO **ERROR**

        ECHO Could not locate %*

        ECHO identify the correct location and update this script

        ECHO.

        GOTO :END

)

ECHO ok

ECHO.

GOTO END

::==================================================

:: set the PID

::==================================================

:SETPID

Set PID=%1

IF "%PID%" == "" CALL :GET_TASKLIST_PID

IF "%PID%" == "" (

        ECHO could not find the PID try entering it as an arguement to the script:

        ECHO "threaddump_win.bat <JBOSS_PID>"

        GOTO END

) ELSE (

        ECHO Generating threaddumps from %PID%

        ECHO.

)

GOTO END

:: Determine the PID from the output of tasklist

:GET_TASKLIST_PID

ECHO No PID specified, attempting to identify the jboss process automatically

ECHO.

SET /a mem = 0

:: for each java process

FOR /F "tokens=*" %%a in ('TASKLIST /fi "IMAGENAME eq java.exe"') DO CALL :SET_TASKLIST_PID %%a

GOTO END

:SET_TASKLIST_PID

::skip if the line doesn't contains a memory value

IF "%5"=="" GOTO END

if %5 GTR %mem% (

        :: set the new largest pid and save the new memory usage

        SET /A MEM = %1

        SET /A PID = %2

)

GOTO END

::==================================================

:END

Open cmd.exe as administrator

From the command line run threaddump_win.bat optionally using the pid as a parameter as in the following example:

threaddump_win.bat 1234