Error Message

A memory leak is also known as a heap leak. Heap is dynamic memory of the running process space. A process generally allocates and frees dynamic memory during run time based on the application requirements. A Heap leak is typically caused by some code in the process space not freeing dynamically allocated memory after its usage. Over time it results in process memory growth, which leads to sluggish response and eventually an application hang or crash.

To detect the code that is responsible for a memory leak implemented by a binary (exe or dll) requires collecting several log and dump files. These files are necessary for Veritas developers to determine the memory issues created by the binary file and to create a fix. 

The following section contains steps on how to collect necessary data for a process space that is exhibiting the memory growth over time for analysis. These steps use the Veritas Beserver service as an example process space and these steps are also applicable to any running process space on the system that exhibits a memory leak issue.

Required Tools

1. Download Microsoft debugging tool windbg on to the problem server and install it. Refer the following Microsoft link to download windbg tool based on the problem server Windows version (32-bit/64-bit).
https://msdn.microsoft.com/en-us/windows/hardware/hh852363 (Do not install .Net 4.5 and only check the box for 'Debugging Tools for Windows' during the install, no other options are needed)

2. Download another Microsoft tool userdump.exe on to the problem server. Use following link to download the tool.
https://www.microsoft.com/downloads/details.aspx?FamilyID=E089CA41-6A87-40C8-BF69-28AC08570B7E&displaylang=en

It is a self extracting executable and it contains all the necessary versions of userdump.exe based on Windows version. It extracts all the necessary files to the kktools\userdump8.1 folder path. Use an x86 folder on Windows 32-bit operating systems. Use an x64 folder on Windows 64-bit operating systems. Use an ia64 folder on Windows that runs on Intel Itanium processors. 

Note: Do not run Setup.exe found in these folders; just use the userdump.exe from these respective folders from the DOS window to create the user dumps of the process space.

a. To list all the running processes with associated PID (Process Identifier) on the system, run userdump /p from the DOS window. This helps to identify the PID of the process you might need to dump and PID is used as a command line parameter for userdump.exe to create the user dump.

b. To create the user dump of a suspicious running process on the system, run userdump PID from the DOS window.

3. Enable the  User Mode Stack Trace Database  debugging flag on Beserver.exe service using the  gflags.exe  tool. Gflags.exe is installed by Microsoft Windbg tool setup. 

You can use this tool either from a DOS command prompt or using Gflags Graphical User Interface to set this flag on Beserver service. To launch the Graphical User Interface go to Start > AllPrograms > Debugging Tools for Windows (x86)

a. From a DOS command prompt, run gflags /i Beserver.exe +ust

b. Using the Gflag Graphical User Interface: 

4. To make sure flag is been enabled proper , use Registry Editor to review the following Key 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\beserver.exe

Name: GlobalFlag
Type: REG_DWORD
Value: 0x00001000

Caution!  This procedure requires you to edit the registry. Using Registry Editor incorrectly can cause serious problems that may require you to reinstall your operating system. Veritas cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use the Registry Editor at your own risk. Back up the registry before you edit it.

How to Collect the UMDH logs and user dump files for leaking process:

1. After enabling the User Mode Stack Trace Database debugging flag on Beserver service, make sure to restart the service, other wise the flag will not be enabled on the exe. So it’s very import to restart the service or re launch the application to apply this flag.

2. After service is been restarted, immediately create the user dump of the beserver service and run the UMDH.exe tool to create the stack trace log file. UMDH.exe is also installed by the Microsoft Windbg tool setup. 

Please open 2 elevated command prompts and navigate one to the install location of the UDMH.exe(default is C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64) and the other command prompt to Userdump.exe(default is C:\kktools\userdump8.1\x64). The command lines to create the user dump and UMDH log files are as follows.

               >>userdump PID of Beserver  c:\Beserver_dump1.dmp
               >>umdh -p:PID of Beserver -f:c:\Beserver_allocation1.txt
 

3. Launch the task manager and note the initial memory usage of the Beserver service (generally it is around 100mb).

4. Wait until task manager reports “Initial value +150 MB” memory usage for Beserver service and create another set of user dumps of Beserver and UMDH log.

The command lines to create the user dump and UMDH log file

                >>userdump PID of Beserver  c:\Beserver_dump2.dmp
                >>umdh -p:PID of Beserver -f:c:\Beserver_allocation2.txt

5. Repeat step 4 at “Initial value +300 MB” and “Initial value +450 MB” increments and make sure to name these files uniquely and appropriately.

6. Zip and send these 3 sets of user dumps and UMDH log files to the Veritas technical support agent to identify the leaking code.
 

Disabling the User Mode Stack Trace Database debugging flag:

After collecting UMDH log files and user dumps of the process space, it’s important to disable the User Mode Stack Trace Database debugging flag. Run the following command from the DOS command prompt and restart the service. 

gflags /i Beserver.exe -ust

 

Sometimes executables and services, while running on the system, exhibit gradual memory usage increases over time. This is a typical symptom of memory leak in running process space. This fact can be monitored through the Task Manager. When executable and services experience the gradual memory growth over time, they typically exhibit slow response, a hung situation or eventually crash. If Dr. Watson is enabled on the system to collect full user dump, and user dump file size of crashing process is MB or more, then it is an indication of a possible memory leak that can lead to a crash.