| Subject: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/10/2008 10:21:17 AM |
| From: Megan Kielman [Email Address Protection] |
All - If I were to add the Work Set column in Process Explorer, should the difference between total physical memory and working set equal the perfmon counter Available MBytes? Thanks! Megan |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/11/2008 9:23:39 PM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> If I were to add the Work Set column in Process Explorer, should the > difference between total physical memory and working set equal the perfmon > counter Available MBytes? No. Total working set is actually a meaningless counter, because it counts shared pages multiple times. In addition to that, there are other kinds of memory that are neither part of available bytes nor part of any working set. This includes things like non-pageable kernel data and code, pages on the modified list, etc. The Windows Internals book by Russinovich and Solomon has more details on this in its memory managment chapter. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/16/2008 4:00:29 PM |
| From: Megan Kielman [Email Address Protection] |
Pavel - Thank you for the book suggestion. I have read the parts of that chapter but still find myself having lots of questions. I guess what I am looking for is a good way to determine how much physical memory each process was using. All this time I thought working set was a good counter, but then I realized that the working set column in ProcessExplorer did not add up to total Physical memory. I realize that the kernel uses some of the memory as well. I just really want to be able to look at what each process is using and consider all other components that use memory, add them all together and get a sum equal to total physical memory. Is this possible? My problem right now is that I have a server that hovers around having 100MB of Available Memory, but I can't figure out which process is using it all because Working Set shows that the highest memory user is 120MB. The system has 4GB. Additionally I don't understand what the VM column in Process Explorer means. I have found several places on the internet that talk about it, but they are all saying something different! Please help, Megan Pavel Lebedinsky [MSFT] wrote: >> If I were to add the Work Set column in Process Explorer, should the >> difference between total physical memory and working set equal the perfmon >> counter Available MBytes? > > > No. Total working set is actually a meaningless counter, because it > counts shared pages multiple times. In addition to that, there are > other kinds of memory that are neither part of available bytes > nor part of any working set. This includes things like non-pageable > kernel data and code, pages on the modified list, etc. > > The Windows Internals book by Russinovich and Solomon has > more details on this in its memory managment chapter. > |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/18/2008 12:38:49 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> Thank you for the book suggestion. I have read the parts of that chapter > but still find myself having lots of questions. I guess what I am looking > for is a good way to determine how much physical memory each process was > using. All this time I thought working set was a good counter, but then I > realized that the working set column in ProcessExplorer did not add up to > total Physical memory. I realize that the kernel uses some of the memory > as well. I just really want to be able to look at what each process is > using and consider all other components that use memory, add them all > together and get a sum equal to total physical memory. Is this possible? Only approximately. Windows Internals has a section that talks about this, called "EXPERIMENT: Accounting for Physical Memory Use": http://book.itzero.com/read/microsoft/0507/Microsoft.Press.Microsoft.Windows.Internals.Fourth.Edition.Dec.2004.internal.Fixed.eBook-DDU_html/0735619174/ch07lev1sec1.html#ch07table15 > My problem right now is that I have a server that hovers around having > 100MB of Available Memory, but I can't figure out which process is using > it all because Working Set shows that the highest memory user is 120MB. > The system has 4GB. Typically, most of the remaining memory would be in the system working set. The size of the system working set is represented by the Memory\Cache Bytes counter in perfmon, and it should be equal to the sum of these 4 counters: Memory\Pool Paged Resident Bytes Memory\System Code Resident Bytes Memory\System Driver Resident Bytes Memory\System Cache Resident Bytes System Cache Resident Bytes is typically the largest part. > Additionally I don't understand what the VM column in Process Explorer > means. I have found several places on the internet that talk about it, but > they are all saying something different! VM Size in task manager is the process commit charge, also known as "private bytes" or "pagefile bytes", event though these last two names are not quite accurate. Process commit charge is basically the amount of system virtual memory that is reserved for use by the process. The total (system-wide) commit charge cannot exceed the sum of the RAM and all pagefiles. Commit is typically charged when applications allocate memory by calling VirtualAlloc(MEM_COMMIT). Some of this committed memory may contain actual data (either in RAM or in the pagefile). The rest is purely "virtual", meaning it doesn't consume any physical storage. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/21/2008 12:51:44 PM |
| From: Megan Kielman [Email Address Protection] |
Thank you Pavel. I have some additional questions: If VM Size is the same as "Private Bytes", why doesn't Private Bytes and VM equal one another in Process Explorer? With regard to Cache Bytes - which of the counters you mentioned below refers to File System Cache? Pavel Lebedinsky [MSFT] wrote: >> Thank you for the book suggestion. I have read the parts of that chapter >> but still find myself having lots of questions. I guess what I am looking >> for is a good way to determine how much physical memory each process was >> using. All this time I thought working set was a good counter, but then I >> realized that the working set column in ProcessExplorer did not add up to >> total Physical memory. I realize that the kernel uses some of the memory >> as well. I just really want to be able to look at what each process is >> using and consider all other components that use memory, add them all >> together and get a sum equal to total physical memory. Is this possible? > > Only approximately. Windows Internals has a section that talks about this, > called "EXPERIMENT: Accounting for Physical Memory Use": > > http://book.itzero.com/read/microsoft/0507/Microsoft.Press.Microsoft.Windows.Internals.Fourth.Edition.Dec.2004.internal.Fixed.eBook-DDU_html/0735619174/ch07lev1sec1.html#ch07table15 > >> My problem right now is that I have a server that hovers around having >> 100MB of Available Memory, but I can't figure out which process is using >> it all because Working Set shows that the highest memory user is 120MB. >> The system has 4GB. > > Typically, most of the remaining memory would be in the system working set. > The size of the system working set is represented by the Memory\Cache Bytes > counter in perfmon, and it should be equal to the sum of these 4 counters: > > Memory\Pool Paged Resident Bytes > Memory\System Code Resident Bytes > Memory\System Driver Resident Bytes > Memory\System Cache Resident Bytes > > System Cache Resident Bytes is typically the largest part. > >> Additionally I don't understand what the VM column in Process Explorer >> means. I have found several places on the internet that talk about it, but >> they are all saying something different! > > VM Size in task manager is the process commit charge, also known as > "private bytes" or "pagefile bytes", event though these last two names > are not quite accurate. > > Process commit charge is basically the amount of system virtual memory > that is reserved for use by the process. The total (system-wide) commit > charge cannot exceed the sum of the RAM and all pagefiles. > > Commit is typically charged when applications allocate memory by > calling VirtualAlloc(MEM_COMMIT). Some of this committed > memory may contain actual data (either in RAM or in the pagefile). > The rest is purely "virtual", meaning it doesn't consume any physical > storage. > > |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/22/2008 2:18:27 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> If VM Size is the same as "Private Bytes", why doesn't Private Bytes and > VM equal one another in Process Explorer? "Virtual Size" in Process Explorer is the same counter as Process\Virtual Bytes in perfmon. It is the sum of all regions in the virtual address space that are either committed or reserved, regardless of region type (private, shared or executable). Private Bytes is a subset of Virtual Bytes, because it is comprised mostly of private committed regions (plus a few other things, like shared file views that are mapped as copy-on-write). > With regard to Cache Bytes - which of the counters you mentioned below > refers to File System Cache? Memory\System Cache Resident Bytes. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/22/2008 3:45:57 PM |
| From: Megan Kielman [Email Address Protection] |
Pavel - Thank you for your help! I have been going through the "Accounting for Physical Memory" and cannot find the perfmon counter that shows the Modified List. Do you know? Regarding "Virtual Size" in process explorer, the memory that is marked as "reserved" but not "committed" does not subtract from our physical memory until it is backed by actual memory, correct? Therefore, I should not worry about that counter when I am trying to figure out what process(es) are using all the memory? Pavel Lebedinsky [MSFT] wrote: >> If VM Size is the same as "Private Bytes", why doesn't Private Bytes and >> VM equal one another in Process Explorer? > > "Virtual Size" in Process Explorer is the same counter as Process\Virtual > Bytes in perfmon. It is the sum of all regions in the virtual address space > that are either committed or reserved, regardless of region type (private, > shared or executable). Private Bytes is a subset of Virtual Bytes, because > it is comprised mostly of private committed regions (plus a few other > things, like shared file views that are mapped as copy-on-write). > >> With regard to Cache Bytes - which of the counters you mentioned below >> refers to File System Cache? > > Memory\System Cache Resident Bytes. > |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/22/2008 3:52:05 PM |
| From: Megan Kielman [Email Address Protection] |
Can you further explain the committed memory that doesn't consume physical storage. If it isn't in physical memory or on the hard disk, where is it? Megan Kielman wrote: > Thank you Pavel. I have some additional questions: > > If VM Size is the same as "Private Bytes", why doesn't Private Bytes and > VM equal one another in Process Explorer? > > With regard to Cache Bytes - which of the counters you mentioned below > refers to File System Cache? > > > > > Pavel Lebedinsky [MSFT] wrote: >>> Thank you for the book suggestion. I have read the parts of that >>> chapter but still find myself having lots of questions. I guess what >>> I am looking for is a good way to determine how much physical memory >>> each process was using. All this time I thought working set was a >>> good counter, but then I realized that the working set column in >>> ProcessExplorer did not add up to total Physical memory. I realize >>> that the kernel uses some of the memory as well. I just really want >>> to be able to look at what each process is using and consider all >>> other components that use memory, add them all together and get a sum >>> equal to total physical memory. Is this possible? >> >> Only approximately. Windows Internals has a section that talks about >> this, >> called "EXPERIMENT: Accounting for Physical Memory Use": >> >> http://book.itzero.com/read/microsoft/0507/Microsoft.Press.Microsoft.Windows.Internals.Fourth.Edition.Dec.2004.internal.Fixed.eBook-DDU_html/0735619174/ch07lev1sec1.html#ch07table15 >> >> >>> My problem right now is that I have a server that hovers around >>> having 100MB of Available Memory, but I can't figure out which >>> process is using it all because Working Set shows that the highest >>> memory user is 120MB. The system has 4GB. >> >> Typically, most of the remaining memory would be in the system working >> set. >> The size of the system working set is represented by the Memory\Cache >> Bytes >> counter in perfmon, and it should be equal to the sum of these 4 >> counters: >> >> Memory\Pool Paged Resident Bytes >> Memory\System Code Resident Bytes >> Memory\System Driver Resident Bytes >> Memory\System Cache Resident Bytes >> >> System Cache Resident Bytes is typically the largest part. >> >>> Additionally I don't understand what the VM column in Process >>> Explorer means. I have found several places on the internet that talk >>> about it, but they are all saying something different! >> >> VM Size in task manager is the process commit charge, also known as >> "private bytes" or "pagefile bytes", event though these last two names >> are not quite accurate. >> >> Process commit charge is basically the amount of system virtual memory >> that is reserved for use by the process. The total (system-wide) commit >> charge cannot exceed the sum of the RAM and all pagefiles. >> >> Commit is typically charged when applications allocate memory by >> calling VirtualAlloc(MEM_COMMIT). Some of this committed >> memory may contain actual data (either in RAM or in the pagefile). >> The rest is purely "virtual", meaning it doesn't consume any physical >> storage. >> |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/22/2008 3:57:26 PM |
| From: Megan Kielman [Email Address Protection] |
Last question for today with regard to System Cache Resident Bytes, how do I know which portion of it is dedicated to the File System? When I think of the "File System Cache" I am thinking that it is a chunk of memory that has data files loaded into it for quick access. Is this true? If so, is the more RAM available for a file server better for performance of serving files? Pavel Lebedinsky [MSFT] wrote: >> If VM Size is the same as "Private Bytes", why doesn't Private Bytes and >> VM equal one another in Process Explorer? > > "Virtual Size" in Process Explorer is the same counter as Process\Virtual > Bytes in perfmon. It is the sum of all regions in the virtual address space > that are either committed or reserved, regardless of region type (private, > shared or executable). Private Bytes is a subset of Virtual Bytes, because > it is comprised mostly of private committed regions (plus a few other > things, like shared file views that are mapped as copy-on-write). > >> With regard to Cache Bytes - which of the counters you mentioned below >> refers to File System Cache? > > Memory\System Cache Resident Bytes. > |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/24/2008 2:28:38 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> I have been going through the "Accounting for Physical Memory" and cannot > find the perfmon counter that shows the Modified List. Do you know? Vista/WS08 have Memory\Modified Page List Bytes counter. On XP/WS03 the only way to get it is using a kernel debugger. > Regarding "Virtual Size" in process explorer, the memory that is marked as > "reserved" but not "committed" does not subtract from our physical memory > until it is backed by actual memory, correct? Therefore, I should not > worry about that counter when I am trying to figure out what process(es) > are using all the memory? Correct. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/24/2008 3:31:07 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> Last question for today with regard to System Cache Resident Bytes, how do > I know which portion of it is dedicated to the File System? All of it. (Technically, system components other than the cache manager can also map views of files in the system cache space, but data in such views can still be considered part of "file cache"). > When I think of the "File System Cache" I am thinking that it is a chunk > of memory that has data files loaded into it for quick access. Is this > true? To some extent, yes. On Windows things are a little more complicated than that because in addition to the system working set, many cached file pages can also be found on the standby page list. This is again described in more detail in the Windows Internals book. > If so, is the more RAM available for a file server better for performance > of serving files? Yes. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/24/2008 3:41:33 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> Can you further explain the committed memory that doesn't consume physical > storage. If it isn't in physical memory or on the hard disk, where is it? Physical memory is allocated on demand when an application accesses a committed page for the first time. Until a page has been accessed it does not consume any physical storage. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/24/2008 2:56:01 PM |
| From: Megan Kielman [Email Address Protection] |
Pavel - Is there a point in which the System Cache and Standby list are flushed or do items remain present until there is a need for space? Also, in the case of the File Server, per the Windows Internals book, the remainder of free physical memory can be used up by the standby list. If a process comes along that needs memory, will it push things off the standby list? Thank you so much! Things are much more sense to me now! Megan Pavel Lebedinsky [MSFT] wrote: >> Last question for today with regard to System Cache Resident Bytes, how do >> I know which portion of it is dedicated to the File System? > > All of it. (Technically, system components other than the cache > manager can also map views of files in the system cache space, > but data in such views can still be considered part of "file cache"). > >> When I think of the "File System Cache" I am thinking that it is a chunk >> of memory that has data files loaded into it for quick access. Is this >> true? > > To some extent, yes. On Windows things are a little more complicated > than that because in addition to the system working set, many cached > file pages can also be found on the standby page list. This is again > described in more detail in the Windows Internals book. > >> If so, is the more RAM available for a file server better for performance >> of serving files? > > Yes. > |
| Back |
| Subject: Re: Process Explorer Working Set Total |
| Group: microsoft.public.winternals |
| Date: 4/25/2008 3:01:12 AM |
| From: "Pavel Lebedinsky [MSFT]" [Email Address Protection] |
> Is there a point in which the System Cache and Standby list are flushed or > do items remain present until there is a need for space? Standby pages are basically available memory. They already have an up-to-date copy of their contents on disk, so if necessary the memory manager can simply erase their contents and use them for something else. That means there is never any need to flush them. System cache is a more complicated case. It consumes physical memory and kernel address space, both of which are limited resources. Some of the pages it holds can be dirty, which means that even if they are removed from the working set they will not be immediately reusable until they are first written to their backing store. So there is a complicated set of rules that control the working set size, the address space usage and the number of dirty pages in the cache. > Also, in the case of the File Server, per the Windows Internals book, the > remainder of free physical memory can be used up by the standby list. If a > process comes along that needs memory, will it push things off the standby > list? Yes. When a process requests a new page, the memory manager will first look in the zeroed page list, then in the free list, and finally if both the free and the zeroed lists are empty, it will take a standby page and zero it. -- This posting is provided "AS IS" with no warranties, and confers no rights. |
| Back |