How to Troubleshoot High Memory Usage - Part 1

Problem

How do I troubleshoot high memory usage and memory performance issues?

Solution

Use Windows Task Manager to monitor memory

1. Press CTRL+ALT+DELETE and select Task Manager.
2. Click the Performance tab.
3. Monitor the following over time:
   
   • Under Physical Memory (K), check to see if the Available value decreases. If so, you might have a memory leak.
   • Under Kernel Memory (K), observe changes in Paged and Non-paged memory to identify if it is a kernel paged or non-paged memory leak. 
4. If you identify a leak, click the Processes tab, and select View.
5. Select Columns and enable the following:
   
   • Page Faults
   • Virtual Memory Size
   • Paged Pool
   • Non-paged Pool
   • Handle Count
   • Thread Count.
6. In the Processes tab, click Mem Usage to bring the process using the most memory on the top.
   
   NOTE: If you identify a process using high memory and not releasing it, use the following information to help troubleshoot the issue. You might also be requested to provide a process dump to help identify the cause.  

PoolMon and PerfMon
For a more in depth and accurate analysis, run PoolMon and PerfMon at the same time.

PoolMon
IMPORTANT: This applies if you want to use PoolMon on Windows XP or earlier. You must enable Gflags.exe to enable pool tagging. Pool tagging is permanently enabled on Windows Server 2003 and later.

1. If you are using XP, enable pool tagging as follows. If you are using Windows 2003 or later, skip to Step 2.
   
   • Enable pool tagging by using a dialog box:
     1. Click Start, Run, and type Gflags.
     2. In the dialog box, enable Enable Pool Tagging.
     3. Restart your computer.
         
   • Enable pool tagging by using the command line:
     1. Click Start, Run, type cmd, and press ENTER.
     2. Type the following command and press ENTER:
        
        gflags /r +ptg
         
     3. Restart your computer.
         
2. Prepare to run PoolMon:
   1. Poolmon.exe is contained in Microsoft Windows Driver Kit (WDK). You can download the WDK from http://www.microsoft.com/download/en/details.aspx?id=11800. 
   2. Install PoolMon on the computer you want to test, following the Microsoft product instructions.
       
3. This example outlines a procedure for using PoolMon to detect a memory leak:
    
   1. Click Start, Run, type cmd, and press ENTER.
   2. Navigate to the PoolMon directory.
   3. Type the following command and press ENTER:
      
      IMPORTANT: To obtain the most accurate results, follow the instructions below accurately. Starting PoolMon changes the data, therefore you must let it run until it reaches a steady state and the data is reliable.
      
      poolmon -b -p -r -n <filename>.log
      Let PoolMon run for at least few hours, sometimes it might need to run for few days.
       
   4. Stop PoolMon, wait for a 30 minutes, and then restart PoolMon.
      
      IMPORTANT: Repeat this every 30 minutes for at least two hours.
       
   5. To be able to take multiple snapshots over time the below script can assist:
      
      @ECHO off
      :LOOP
      ECHO %DATE %TIME% >>filename.log
      Poolmon -b -p -r -n filename.log
      Ping -n seconds 127.0.0.1 >NULL
      GOTO LOOP
      
      NOTE: for the seconds, McAfee recommends every 15 minutes.
       
   6. When data collection is complete, examine the following values for each tag, and note any that continually increase:
      • Diff (allocations minus free bytes)
      • Bytes (number of bytes allocated minus number of bytes freed)
         
   1. Examine the allocations that were increasing, and determine whether the bytes are now freed. Allocations that have still not been freed, or have continued to increase in size are the likely cause.

For more details on PoolMon usage, see http://msdn.microsoft.com/en-us/library/ff547083(v=vs.85).aspx.
 

PerfMon
PerfMon offers several methods to save captured data. However, McAfee uses Microsoft Binary Performance Log (BLG) format to troubleshoot performance issues.

Windows 7 users

1. Click Start, Run, type cmd, and press ENTER.
2. Type the following command and press ENTER:
   
   perfmon
    
3. Click Data Collector Sets, User Defined.
4. Right-click User Defined, select New, and select Data Collector Set.
5. Type a name (for example, McAfee <date_timestamp>), select Create manually, and click OK.
6. Under Create data logs, only select Performance counter, and click Next.
7. Click Add. In the next page from the drop-down list, select Processor, select <All instances>,  and click Add.
8. Select Memory from the drop-down list and select <All instances>,  then click Add.
9. Select Process from the drop-down list, and select <All instances> from the next drop-down list.
10. Click Add, OK, Next, Next, then select Start this data collector set now, then click Finish.
    Wait for the data to run long enough to capture the information and collect the log while reproducing the issue.
11. Right-click User Defined, select and right-click the <log name> and click Stop.
12. Retrieve the log from C:\Perflogs\Admin\examplename\computername_date-time\DataColletor01.blg.
    

Windows XP users

1. Click Start, Run, type cmd, and press ENTER.
2. Type the following command and press ENTER:
   
   perfmon
    
3. Click Performance Logs and Alerts.
4. Right-click Counter logs, and select New Log Settings.
5. Type a name (for example, McAfee <date_timestamp>) and click OK.
6. Click Add Objects, select Processor, and click Add.
7. Select Memory, and click Add, Close, Add counters.
8. Under Performance Object,, select process from the drop-down list.
9. Select All counters and select All instances, Add, Close.
10. Select Apply and click OK to continue.
    Wait for the data to run long enough to capture the information and collect the log while reproducing the issue.
11. Click the Stop icon on the menu bar.
12. Retrieve the log  from C:\perflogs\logs.blg