| Subject: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/12/2007 9:44:59 AM |
| From: Megan Kielman [Email Address Protection] |
Is this statement correct? I continue to read articles (like this one http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory that talk about a processes "own" address space. Assuming 2GB in UM... When a process starts up, it is presented with all 2GB of UM address space, but it can request to actually use much less. The memory manager determines which memory is available for use. The amount of memory used by this process is reflected in the Working Set and Private Bytes counters. |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/12/2007 11:10:17 AM |
| From: "Ben Voigt [C++ MVP]" [Email Address Protection] |
"Megan Kielman" <megan.kielman@gmail.com> wrote in message news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... > Is this statement correct? I continue to read articles (like this one > http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory > that talk about a processes "own" address space. > > Assuming 2GB in UM... > > When a process starts up, it is presented with all 2GB of UM address > space, but it can request to actually use much less. The memory manager > determines which memory is available for use. The amount of memory used by > this process is reflected in the Working Set and Private Bytes counters. Every process in 32-bit Windows has 2GB of user-mode address space. Most of the space is not mapped to physical memory. The page mappings are stored in the TLB. You may find that http://en.wikipedia.org/wiki/Category:Virtual_memory provides more information than random blogs. |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/12/2007 12:52:31 PM |
| From: "the_el_vez" [Email Address Protection] |
Ok - so let's try to take this topic from the top down. There are structures maintained in the kernel - for each process in the system - called page tables (PTE's and PDE's on x86). These tables are constructed such that a virtual address in a process can be broken down and decoded into a physical address in RAM by the processor. Each process (NOT PROCESSOR) in Windows has a separate set of these tables. In addition to the page tables, each process has a virtual address space that is 4gb in size (0 - 0xffffffff). So now the real question you are asking is how do these 2 things relate. RAM in the machine and virtual address space are totally independent of each other. Theoretically a process can be using no RAM at all - but always has its 4gb address space. So now what happens when we want to allocate some of our 4gb of virtual address space? Well the memory manager creates something called a virtual address descriptor (VAD) in response to a request to allocate some of our process VA. A VAD describes a region of VA space in a single process. There is a seperate set of VAD's for each process. The VAD still has no relation to RAM. Let's say our VAD describes address 0x1000 in our process's VA. So at this point we have a 4gb VA space, a VAD describing some of that VA space. So now you execute some code in your process that writes to address 0x1000. The first thing that happens is the processor takes the address 0x1000 and looks in the process's page tables to find the RAM that backs it. Since there is no page table entry for 0x1000, we get a page fault. A page fault occurs when the processor references a virtual address that isn't backed by RAM - meaning that there is no page table entry for our process that decodes the VA to RAM. The page fault triggers code in MM. The MM code looks at the address that faulted (in our case 0x1000). MM then looks for a VAD in our process's VAD's that describes that address. Lucky for us it finds one (if it didn't find one we would wind up with an access violation exception). When MM finds the VAD it says "oops - this process is supposed to have some memory here but I am a big fat liar and didn't back this address with any RAM yet because I didn't think the process would really use it.. I'd better do that now!". Then the MM finds some available page of RAM (say RAM address 0x30000) and updates the page tables (for the process that page faulted) to use the RAM page 0x30000 as backing for VA 0x1000 in the current process and re-executes the instruction that started this whole chain of events. BTW - the reason MM doesn't immediately back the address with RAM when you call allocate() - is because most allocated VA is never used. So MM waits till you PROVE that you need the memory (by page faulting). Some important things to note about this scenario: 1.) Only address 0x1000 in THIS process are backed by the RAM page 0x30000 (THIS process means the process we are talking about in the scenario above). 2.) Another process's VA 0x1000 will be backed by a different RAM page or not backed at all. Does this make this more clear? "Megan Kielman" <megan.kielman@gmail.com> wrote in message news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... > Is this statement correct? I continue to read articles (like this one > http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory > that talk about a processes "own" address space. > > Assuming 2GB in UM... > > When a process starts up, it is presented with all 2GB of UM address > space, but it can request to actually use much less. The memory manager > determines which memory is available for use. The amount of memory used by > this process is reflected in the Working Set and Private Bytes counters. |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/12/2007 12:59:39 PM |
| From: "the_el_vez" [Email Address Protection] |
The TLB is a small fast look up for already translated VA mappings. Page Directories/Page Tables are what actually do the mapping. "Ben Voigt [C++ MVP]" <rbv@nospam.nospam> wrote in message news:uDbSboOPIHA.3516@TK2MSFTNGP02.phx.gbl... > > "Megan Kielman" <megan.kielman@gmail.com> wrote in message > news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >> Is this statement correct? I continue to read articles (like this one >> http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory >> that talk about a processes "own" address space. >> >> Assuming 2GB in UM... >> >> When a process starts up, it is presented with all 2GB of UM address >> space, but it can request to actually use much less. The memory manager >> determines which memory is available for use. The amount of memory used >> by this process is reflected in the Working Set and Private Bytes >> counters. > > Every process in 32-bit Windows has 2GB of user-mode address space. Most > of the space is not mapped to physical memory. The page mappings are > stored in the TLB. > > You may find that http://en.wikipedia.org/wiki/Category:Virtual_memory > provides more information than random blogs. > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/13/2007 3:06:11 PM |
| From: Megan Kielman [Email Address Protection] |
Yes this makes things infinitely more clear. Thank you for your patience with my questions! You mentioned that when a process goes to access the memory in the VAD that isn't backed by physical RAM, it creates a page fault. It makes me wonder then when looking at the Page Fault counter in Process Explorer, I always assumed this meant that the process was doing a log of paging (to the page file) but it could also mean that the process is attempting to access memory that has never been backed by memory. Does that make the Page Fault counter not a good way to determine if the system is paging out to page file a lot? Also, I am trying to understand how kernel memory allocation fits in. Because User mode memory is always mapped to the lower half of the address space and kernel mode is always mapped to the upper half, does that always mean that no matter how much physical RAM, 1/2 will be dedicated to kernal? I am really sorry if my questions aren't clear. I am doing the best I can to articulate what I am trying to ask. the_el_vez wrote: > Ok - so let's try to take this topic from the top down. > > There are structures maintained in the kernel - for each process in the > system - called page tables (PTE's and PDE's on x86). These tables are > constructed such that a virtual address in a process can be broken down > and decoded into a physical address in RAM by the processor. Each > process (NOT PROCESSOR) in Windows has a separate set of these tables. > In addition to the page tables, each process has a virtual address space > that is 4gb in size (0 - 0xffffffff). So now the real question you are > asking is how do these 2 things relate. RAM in the machine and virtual > address space are totally independent of each other. Theoretically a > process can be using no RAM at all - but always has its 4gb address > space. So now what happens when we want to allocate some of our 4gb of > virtual address space? Well the memory manager creates something called > a virtual address descriptor (VAD) in response to a request to allocate > some of our process VA. A VAD describes a region of VA space in a single > process. There is a seperate set of VAD's for each process. The VAD > still has no relation to RAM. Let's say our VAD describes address 0x1000 > in our process's VA. So at this point we have a 4gb VA space, a VAD > describing some of that VA space. So now you execute some code in your > process that writes to address 0x1000. The first thing that happens is > the processor takes the address 0x1000 and looks in the process's page > tables to find the RAM that backs it. Since there is no page table entry > for 0x1000, we get a page fault. A page fault occurs when the processor > references a virtual address that isn't backed by RAM - meaning that > there is no page table entry for our process that decodes the VA to RAM. > The page fault triggers code in MM. The MM code looks at the address > that faulted (in our case 0x1000). MM then looks for a VAD in our > process's VAD's that describes that address. Lucky for us it finds one > (if it didn't find one we would wind up with an access violation > exception). When MM finds the VAD it says "oops - this process is > supposed to have some memory here but I am a big fat liar and didn't > back this address with any RAM yet because I didn't think the process > would really use it.. I'd better do that now!". Then the MM finds some > available page of RAM (say RAM address 0x30000) and updates the page > tables (for the process that page faulted) to use the RAM page 0x30000 > as backing for VA 0x1000 in the current process and re-executes the > instruction that started this whole chain of events. BTW - the reason MM > doesn't immediately back the address with RAM when you call allocate() - > is because most allocated VA is never used. So MM waits till you PROVE > that you need the memory (by page faulting). Some important things to > note about this scenario: > > 1.) Only address 0x1000 in THIS process are backed by the RAM page > 0x30000 (THIS process means the process we are talking about in the > scenario above). > 2.) Another process's VA 0x1000 will be backed by a different RAM page > or not backed at all. > > Does this make this more clear? > > > "Megan Kielman" <megan.kielman@gmail.com> wrote in message > news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >> Is this statement correct? I continue to read articles (like this one >> http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory >> that talk about a processes "own" address space. >> >> Assuming 2GB in UM... >> >> When a process starts up, it is presented with all 2GB of UM address >> space, but it can request to actually use much less. The memory >> manager determines which memory is available for use. The amount of >> memory used by this process is reflected in the Working Set and >> Private Bytes counters. > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/13/2007 4:06:12 PM |
| From: "the_el_vez" [Email Address Protection] |
No. It is pretty much fully dynamic (in Vista and beyond). Obviously there are kernel areas that are always backed (non-paged pool, etc.) but the memory manager can give and take away as required - kernel or user VA backing. As far as page faults go - the first time a page is backed by memory that is a special kind of fault - it is not read from the page file. Once it is trimmed and put into the page file - the next fault (if one occurs for that address) will be read in from the page file. An interesting app to write is one that allocates VA, writes to and frees it. If you watch the page fault counters when you run this - you will see the pagefaults are pretty much linear with the size of MAX_ALLOCS. Here is some code I just whipped up (should compile :) - I wrote it in the mail editor!): #include <windows.h> #include <stdio.h> #include <stdlib.h> #include <conio.h> #define PAGE_SIZE 0x1000 #define MAX_ALLOCS 100 PULONG Data[MAX_ALLOCS]; int __cdecl main(int argc, char* argv[]) { volatile int a = 0; ULONG Count; printf("Allocating blocks...\n"); // // Allocate the blocks // for (Count = 0; Count < MAX_ALLOCS; Count++) { Data[Count] = (PULONG)VirtualAlloc(NULL, MEM_COMMIT | MEM_RESERVE, PAGE_SIZE, PAGE_READWRITE); if (!Data[Count]) { printf("Failed to allocate Data[%X]. Error == %d\n", Count, GetLastError()); } } printf("Writing blocks...\n"); // // Write to the blocks // for (Count = 0; Count < MAX_ALLOCS; Count++) { if (Data[Count]) { *Data[Count] = 0x1234; } } printf("Freeing blocks...\n"); // // Free the blocks // for (Count = 0; Count < MAX_ALLOCS; Count++) { if (Data[Count]) { VirtualFree(Data[Count], PAGE_SIZE, MEM_RELEASE); } } printf("Press \'y\' to continue\n"); while (a != 'y') { a = _getch(); } return 0; } There will be some floor (probably around 300 pages - just to get the app loaded and running), but after that you should be able to adjust the size of MAX_ALLOCS and see the increase/decrease. This posting is provided "AS IS" with no warranties, and confers no rights "Megan Kielman" <megan.kielman@gmail.com> wrote in message news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... > Yes this makes things infinitely more clear. Thank you for your patience > with my questions! > > You mentioned that when a process goes to access the memory in the VAD > that isn't backed by physical RAM, it creates a page fault. It makes me > wonder then when looking at the Page Fault counter in Process Explorer, I > always assumed this meant that the process was doing a log of paging (to > the page file) but it could also mean that the process is attempting to > access memory that has never been backed by memory. Does that make the > Page Fault counter not a good way to determine if the system is paging out > to page file a lot? > > Also, I am trying to understand how kernel memory allocation fits in. > Because User mode memory is always mapped to the lower half of the address > space and kernel mode is always mapped to the upper half, does that always > mean that no matter how much physical RAM, 1/2 will be dedicated to > kernal? > > I am really sorry if my questions aren't clear. I am doing the best I can > to articulate what I am trying to ask. > > > > the_el_vez wrote: >> Ok - so let's try to take this topic from the top down. >> >> There are structures maintained in the kernel - for each process in the >> system - called page tables (PTE's and PDE's on x86). These tables are >> constructed such that a virtual address in a process can be broken down >> and decoded into a physical address in RAM by the processor. Each process >> (NOT PROCESSOR) in Windows has a separate set of these tables. In >> addition to the page tables, each process has a virtual address space >> that is 4gb in size (0 - 0xffffffff). So now the real question you are >> asking is how do these 2 things relate. RAM in the machine and virtual >> address space are totally independent of each other. Theoretically a >> process can be using no RAM at all - but always has its 4gb address >> space. So now what happens when we want to allocate some of our 4gb of >> virtual address space? Well the memory manager creates something called a >> virtual address descriptor (VAD) in response to a request to allocate >> some of our process VA. A VAD describes a region of VA space in a single >> process. There is a seperate set of VAD's for each process. The VAD still >> has no relation to RAM. Let's say our VAD describes address 0x1000 in our >> process's VA. So at this point we have a 4gb VA space, a VAD describing >> some of that VA space. So now you execute some code in your process that >> writes to address 0x1000. The first thing that happens is the processor >> takes the address 0x1000 and looks in the process's page tables to find >> the RAM that backs it. Since there is no page table entry for 0x1000, we >> get a page fault. A page fault occurs when the processor references a >> virtual address that isn't backed by RAM - meaning that there is no page >> table entry for our process that decodes the VA to RAM. The page fault >> triggers code in MM. The MM code looks at the address that faulted (in >> our case 0x1000). MM then looks for a VAD in our process's VAD's that >> describes that address. Lucky for us it finds one (if it didn't find one >> we would wind up with an access violation exception). When MM finds the >> VAD it says "oops - this process is supposed to have some memory here but >> I am a big fat liar and didn't back this address with any RAM yet because >> I didn't think the process would really use it.. I'd better do that >> now!". Then the MM finds some available page of RAM (say RAM address >> 0x30000) and updates the page tables (for the process that page faulted) >> to use the RAM page 0x30000 as backing for VA 0x1000 in the current >> process and re-executes the instruction that started this whole chain of >> events. BTW - the reason MM doesn't immediately back the address with RAM >> when you call allocate() - is because most allocated VA is never used. So >> MM waits till you PROVE that you need the memory (by page faulting). Some >> important things to note about this scenario: >> >> 1.) Only address 0x1000 in THIS process are backed by the RAM page >> 0x30000 (THIS process means the process we are talking about in the >> scenario above). >> 2.) Another process's VA 0x1000 will be backed by a different RAM page or >> not backed at all. >> >> Does this make this more clear? >> >> >> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>> Is this statement correct? I continue to read articles (like this one >>> http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory >>> that talk about a processes "own" address space. >>> >>> Assuming 2GB in UM... >>> >>> When a process starts up, it is presented with all 2GB of UM address >>> space, but it can request to actually use much less. The memory manager >>> determines which memory is available for use. The amount of memory used >>> by this process is reflected in the Working Set and Private Bytes >>> counters. >> |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/13/2007 4:19:51 PM |
| From: "David Craig" [Email Address Protection] |
Read the book I recommeded. The answer is no. Just to confuse you more there is a 3GB switch available that will change the partitioning of the logical address space from 2/2 to 3/1 where the kernel only has one GB of address space. This is used with SQL server frequently since SQL databases are so huge and since SQL server does its own caching, it needs the memory address space far more than the kernel. All of these questions are answered in the book and other articles from Microsoft. Start with the book or a course offered by OSR or another of the better companies that provide programmer educational courses. For example, another use of paging is the executable itself. That is why Windows can only execute programs that reside on file systems that use the memory manager and cache manager. The address space for the program is created when the process is created, but code from the executable is memory mapped into the correct addresses without any or much of the physical code being actually read into memory. When the process creation is completed, control is passed to the starting address within the process's address space and if nothing has been loaded, a page fault occurs and then the code is paged in, one or more pages at a time, and execution resumed as if it never page faulted. When, for example, the startup code has become unused for a while, depending upon system memory load, the memory manager can mark those pages as no longer being in memory and reuse those blocks for that process or any other process including kernel requests for memory. If you are running something such as Microsoft Word and you see a lot of disk activity even when you are not doing saves, reads, etc. then the document plus the size of Word program are placing stress on memory and you can see a lot of disk activity. The pages are of two basic types. 1) Program code that is not self-modifying so that the page does not need to be written to the page file but can be retrieved from the executable when it is needed. 2) Pages that have been modified or created from 'nothing stored on mass storage' and those pages are written to the pagefile(s) when they have to be freed for handle other memory needs. The hardware supports a 'dirty' flag capability that lets the OS know the page has been changed since it was read in or last saved to the pagefile. "Megan Kielman" <megan.kielman@gmail.com> wrote in message news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... > Yes this makes things infinitely more clear. Thank you for your patience > with my questions! > > You mentioned that when a process goes to access the memory in the VAD > that isn't backed by physical RAM, it creates a page fault. It makes me > wonder then when looking at the Page Fault counter in Process Explorer, I > always assumed this meant that the process was doing a log of paging (to > the page file) but it could also mean that the process is attempting to > access memory that has never been backed by memory. Does that make the > Page Fault counter not a good way to determine if the system is paging out > to page file a lot? > > Also, I am trying to understand how kernel memory allocation fits in. > Because User mode memory is always mapped to the lower half of the address > space and kernel mode is always mapped to the upper half, does that always > mean that no matter how much physical RAM, 1/2 will be dedicated to > kernal? > > I am really sorry if my questions aren't clear. I am doing the best I can > to articulate what I am trying to ask. > > > > the_el_vez wrote: >> Ok - so let's try to take this topic from the top down. >> >> There are structures maintained in the kernel - for each process in the >> system - called page tables (PTE's and PDE's on x86). These tables are >> constructed such that a virtual address in a process can be broken down >> and decoded into a physical address in RAM by the processor. Each process >> (NOT PROCESSOR) in Windows has a separate set of these tables. In >> addition to the page tables, each process has a virtual address space >> that is 4gb in size (0 - 0xffffffff). So now the real question you are >> asking is how do these 2 things relate. RAM in the machine and virtual >> address space are totally independent of each other. Theoretically a >> process can be using no RAM at all - but always has its 4gb address >> space. So now what happens when we want to allocate some of our 4gb of >> virtual address space? Well the memory manager creates something called a >> virtual address descriptor (VAD) in response to a request to allocate >> some of our process VA. A VAD describes a region of VA space in a single >> process. There is a seperate set of VAD's for each process. The VAD still >> has no relation to RAM. Let's say our VAD describes address 0x1000 in our >> process's VA. So at this point we have a 4gb VA space, a VAD describing >> some of that VA space. So now you execute some code in your process that >> writes to address 0x1000. The first thing that happens is the processor >> takes the address 0x1000 and looks in the process's page tables to find >> the RAM that backs it. Since there is no page table entry for 0x1000, we >> get a page fault. A page fault occurs when the processor references a >> virtual address that isn't backed by RAM - meaning that there is no page >> table entry for our process that decodes the VA to RAM. The page fault >> triggers code in MM. The MM code looks at the address that faulted (in >> our case 0x1000). MM then looks for a VAD in our process's VAD's that >> describes that address. Lucky for us it finds one (if it didn't find one >> we would wind up with an access violation exception). When MM finds the >> VAD it says "oops - this process is supposed to have some memory here but >> I am a big fat liar and didn't back this address with any RAM yet because >> I didn't think the process would really use it.. I'd better do that >> now!". Then the MM finds some available page of RAM (say RAM address >> 0x30000) and updates the page tables (for the process that page faulted) >> to use the RAM page 0x30000 as backing for VA 0x1000 in the current >> process and re-executes the instruction that started this whole chain of >> events. BTW - the reason MM doesn't immediately back the address with RAM >> when you call allocate() - is because most allocated VA is never used. So >> MM waits till you PROVE that you need the memory (by page faulting). Some >> important things to note about this scenario: >> >> 1.) Only address 0x1000 in THIS process are backed by the RAM page >> 0x30000 (THIS process means the process we are talking about in the >> scenario above). >> 2.) Another process's VA 0x1000 will be backed by a different RAM page or >> not backed at all. >> >> Does this make this more clear? >> >> >> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>> Is this statement correct? I continue to read articles (like this one >>> http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory >>> that talk about a processes "own" address space. >>> >>> Assuming 2GB in UM... >>> >>> When a process starts up, it is presented with all 2GB of UM address >>> space, but it can request to actually use much less. The memory manager >>> determines which memory is available for use. The amount of memory used >>> by this process is reflected in the Working Set and Private Bytes >>> counters. >> |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/14/2007 12:10:16 PM |
| From: Megan Kielman [Email Address Protection] |
Ok so the address space is split in half, but the amount of RAM allocated to each is fully dynamic. Makes sense. Thank you! Thank you for the code. I need to find someone that knows how to compile C so I can run it. Anyway, so the fact that the page faults is linear, if I am looking at a processes Page Fault it could mean one of two things, either the process constantly asks for, then frees memory OR it is doing a lot of paging. Is that right? IF so, what is a good indicator that your system does not have enough RAM and is relying on the page file too much? Comparing Private Bytes and Working Set? Again, your help has been so great! the_el_vez wrote: > No. It is pretty much fully dynamic (in Vista and beyond). Obviously > there are kernel areas that are always backed (non-paged pool, etc.) but > the memory manager can give and take away as required - kernel or user > VA backing. > > As far as page faults go - the first time a page is backed by memory > that is a special kind of fault - it is not read from the page file. > Once it is trimmed and put into the page file - the next fault (if one > occurs for that address) will be read in from the page file. > > An interesting app to write is one that allocates VA, writes to and > frees it. If you watch the page fault counters when you run this - you > will see the pagefaults are pretty much linear with the size of > MAX_ALLOCS. Here is some code I just whipped up (should compile :) - I > wrote it in the mail editor!): > > #include <windows.h> > #include <stdio.h> > #include <stdlib.h> > #include <conio.h> > > #define PAGE_SIZE 0x1000 > #define MAX_ALLOCS 100 > > > PULONG Data[MAX_ALLOCS]; > > int > __cdecl > main(int argc, > char* argv[]) > { > > volatile int a = 0; > ULONG Count; > > > > printf("Allocating blocks...\n"); > > > // > // Allocate the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > Data[Count] = (PULONG)VirtualAlloc(NULL, > MEM_COMMIT | MEM_RESERVE, > PAGE_SIZE, > PAGE_READWRITE); > > if (!Data[Count]) { > printf("Failed to allocate Data[%X]. Error == %d\n", > Count, > GetLastError()); > } > } > > > printf("Writing blocks...\n"); > > // > // Write to the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > if (Data[Count]) { > *Data[Count] = 0x1234; > } > > } > > > printf("Freeing blocks...\n"); > > // > // Free the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > if (Data[Count]) { > VirtualFree(Data[Count], > PAGE_SIZE, > MEM_RELEASE); > } > > } > > > printf("Press \'y\' to continue\n"); > > while (a != 'y') { > a = _getch(); > } > > > return 0; > } > > > There will be some floor (probably around 300 pages - just to get the > app loaded and running), but after that you should be able to adjust the > size of MAX_ALLOCS and see the increase/decrease. > > > This posting is provided "AS IS" with no warranties, and confers no rights > > "Megan Kielman" <megan.kielman@gmail.com> wrote in message > news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... >> Yes this makes things infinitely more clear. Thank you for your >> patience with my questions! >> >> You mentioned that when a process goes to access the memory in the VAD >> that isn't backed by physical RAM, it creates a page fault. It makes >> me wonder then when looking at the Page Fault counter in Process >> Explorer, I always assumed this meant that the process was doing a log >> of paging (to the page file) but it could also mean that the process >> is attempting to access memory that has never been backed by memory. >> Does that make the Page Fault counter not a good way to determine if >> the system is paging out to page file a lot? >> >> Also, I am trying to understand how kernel memory allocation fits in. >> Because User mode memory is always mapped to the lower half of the >> address space and kernel mode is always mapped to the upper half, does >> that always mean that no matter how much physical RAM, 1/2 will be >> dedicated to kernal? >> >> I am really sorry if my questions aren't clear. I am doing the best I >> can to articulate what I am trying to ask. >> >> >> >> the_el_vez wrote: >>> Ok - so let's try to take this topic from the top down. >>> >>> There are structures maintained in the kernel - for each process in >>> the system - called page tables (PTE's and PDE's on x86). These >>> tables are constructed such that a virtual address in a process can >>> be broken down and decoded into a physical address in RAM by the >>> processor. Each process (NOT PROCESSOR) in Windows has a separate set >>> of these tables. In addition to the page tables, each process has a >>> virtual address space that is 4gb in size (0 - 0xffffffff). So now >>> the real question you are asking is how do these 2 things relate. RAM >>> in the machine and virtual address space are totally independent of >>> each other. Theoretically a process can be using no RAM at all - but >>> always has its 4gb address space. So now what happens when we want to >>> allocate some of our 4gb of virtual address space? Well the memory >>> manager creates something called a virtual address descriptor (VAD) >>> in response to a request to allocate some of our process VA. A VAD >>> describes a region of VA space in a single process. There is a >>> seperate set of VAD's for each process. The VAD still has no relation >>> to RAM. Let's say our VAD describes address 0x1000 in our process's >>> VA. So at this point we have a 4gb VA space, a VAD describing some of >>> that VA space. So now you execute some code in your process that >>> writes to address 0x1000. The first thing that happens is the >>> processor takes the address 0x1000 and looks in the process's page >>> tables to find the RAM that backs it. Since there is no page table >>> entry for 0x1000, we get a page fault. A page fault occurs when the >>> processor references a virtual address that isn't backed by RAM - >>> meaning that there is no page table entry for our process that >>> decodes the VA to RAM. The page fault triggers code in MM. The MM >>> code looks at the address that faulted (in our case 0x1000). MM then >>> looks for a VAD in our process's VAD's that describes that address. >>> Lucky for us it finds one (if it didn't find one we would wind up >>> with an access violation exception). When MM finds the VAD it says >>> "oops - this process is supposed to have some memory here but I am a >>> big fat liar and didn't back this address with any RAM yet because I >>> didn't think the process would really use it.. I'd better do that >>> now!". Then the MM finds some available page of RAM (say RAM address >>> 0x30000) and updates the page tables (for the process that page >>> faulted) to use the RAM page 0x30000 as backing for VA 0x1000 in the >>> current process and re-executes the instruction that started this >>> whole chain of events. BTW - the reason MM doesn't immediately back >>> the address with RAM when you call allocate() - is because most >>> allocated VA is never used. So MM waits till you PROVE that you need >>> the memory (by page faulting). Some important things to note about >>> this scenario: >>> >>> 1.) Only address 0x1000 in THIS process are backed by the RAM page >>> 0x30000 (THIS process means the process we are talking about in the >>> scenario above). >>> 2.) Another process's VA 0x1000 will be backed by a different RAM >>> page or not backed at all. >>> >>> Does this make this more clear? >>> >>> >>> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >>> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>>> Is this statement correct? I continue to read articles (like this >>>> one http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about >>>> memory that talk about a processes "own" address space. >>>> >>>> Assuming 2GB in UM... >>>> >>>> When a process starts up, it is presented with all 2GB of UM address >>>> space, but it can request to actually use much less. The memory >>>> manager determines which memory is available for use. The amount of >>>> memory used by this process is reflected in the Working Set and >>>> Private Bytes counters. >>> > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/18/2007 8:56:10 AM |
| From: Megan Kielman [Email Address Protection] |
Is the type of page fault you are demonstrating with your program called a "Soft" Page Fault? the_el_vez wrote: > No. It is pretty much fully dynamic (in Vista and beyond). Obviously > there are kernel areas that are always backed (non-paged pool, etc.) but > the memory manager can give and take away as required - kernel or user > VA backing. > > As far as page faults go - the first time a page is backed by memory > that is a special kind of fault - it is not read from the page file. > Once it is trimmed and put into the page file - the next fault (if one > occurs for that address) will be read in from the page file. > > An interesting app to write is one that allocates VA, writes to and > frees it. If you watch the page fault counters when you run this - you > will see the pagefaults are pretty much linear with the size of > MAX_ALLOCS. Here is some code I just whipped up (should compile :) - I > wrote it in the mail editor!): > > #include <windows.h> > #include <stdio.h> > #include <stdlib.h> > #include <conio.h> > > #define PAGE_SIZE 0x1000 > #define MAX_ALLOCS 100 > > > PULONG Data[MAX_ALLOCS]; > > int > __cdecl > main(int argc, > char* argv[]) > { > > volatile int a = 0; > ULONG Count; > > > > printf("Allocating blocks...\n"); > > > // > // Allocate the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > Data[Count] = (PULONG)VirtualAlloc(NULL, > MEM_COMMIT | MEM_RESERVE, > PAGE_SIZE, > PAGE_READWRITE); > > if (!Data[Count]) { > printf("Failed to allocate Data[%X]. Error == %d\n", > Count, > GetLastError()); > } > } > > > printf("Writing blocks...\n"); > > // > // Write to the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > if (Data[Count]) { > *Data[Count] = 0x1234; > } > > } > > > printf("Freeing blocks...\n"); > > // > // Free the blocks > // > > for (Count = 0; Count < MAX_ALLOCS; Count++) { > > if (Data[Count]) { > VirtualFree(Data[Count], > PAGE_SIZE, > MEM_RELEASE); > } > > } > > > printf("Press \'y\' to continue\n"); > > while (a != 'y') { > a = _getch(); > } > > > return 0; > } > > > There will be some floor (probably around 300 pages - just to get the > app loaded and running), but after that you should be able to adjust the > size of MAX_ALLOCS and see the increase/decrease. > > > This posting is provided "AS IS" with no warranties, and confers no rights > > "Megan Kielman" <megan.kielman@gmail.com> wrote in message > news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... >> Yes this makes things infinitely more clear. Thank you for your >> patience with my questions! >> >> You mentioned that when a process goes to access the memory in the VAD >> that isn't backed by physical RAM, it creates a page fault. It makes >> me wonder then when looking at the Page Fault counter in Process >> Explorer, I always assumed this meant that the process was doing a log >> of paging (to the page file) but it could also mean that the process >> is attempting to access memory that has never been backed by memory. >> Does that make the Page Fault counter not a good way to determine if >> the system is paging out to page file a lot? >> >> Also, I am trying to understand how kernel memory allocation fits in. >> Because User mode memory is always mapped to the lower half of the >> address space and kernel mode is always mapped to the upper half, does >> that always mean that no matter how much physical RAM, 1/2 will be >> dedicated to kernal? >> >> I am really sorry if my questions aren't clear. I am doing the best I >> can to articulate what I am trying to ask. >> >> >> >> the_el_vez wrote: >>> Ok - so let's try to take this topic from the top down. >>> >>> There are structures maintained in the kernel - for each process in >>> the system - called page tables (PTE's and PDE's on x86). These >>> tables are constructed such that a virtual address in a process can >>> be broken down and decoded into a physical address in RAM by the >>> processor. Each process (NOT PROCESSOR) in Windows has a separate set >>> of these tables. In addition to the page tables, each process has a >>> virtual address space that is 4gb in size (0 - 0xffffffff). So now >>> the real question you are asking is how do these 2 things relate. RAM >>> in the machine and virtual address space are totally independent of >>> each other. Theoretically a process can be using no RAM at all - but >>> always has its 4gb address space. So now what happens when we want to >>> allocate some of our 4gb of virtual address space? Well the memory >>> manager creates something called a virtual address descriptor (VAD) >>> in response to a request to allocate some of our process VA. A VAD >>> describes a region of VA space in a single process. There is a >>> seperate set of VAD's for each process. The VAD still has no relation >>> to RAM. Let's say our VAD describes address 0x1000 in our process's >>> VA. So at this point we have a 4gb VA space, a VAD describing some of >>> that VA space. So now you execute some code in your process that >>> writes to address 0x1000. The first thing that happens is the >>> processor takes the address 0x1000 and looks in the process's page >>> tables to find the RAM that backs it. Since there is no page table >>> entry for 0x1000, we get a page fault. A page fault occurs when the >>> processor references a virtual address that isn't backed by RAM - >>> meaning that there is no page table entry for our process that >>> decodes the VA to RAM. The page fault triggers code in MM. The MM >>> code looks at the address that faulted (in our case 0x1000). MM then >>> looks for a VAD in our process's VAD's that describes that address. >>> Lucky for us it finds one (if it didn't find one we would wind up >>> with an access violation exception). When MM finds the VAD it says >>> "oops - this process is supposed to have some memory here but I am a >>> big fat liar and didn't back this address with any RAM yet because I >>> didn't think the process would really use it.. I'd better do that >>> now!". Then the MM finds some available page of RAM (say RAM address >>> 0x30000) and updates the page tables (for the process that page >>> faulted) to use the RAM page 0x30000 as backing for VA 0x1000 in the >>> current process and re-executes the instruction that started this >>> whole chain of events. BTW - the reason MM doesn't immediately back >>> the address with RAM when you call allocate() - is because most >>> allocated VA is never used. So MM waits till you PROVE that you need >>> the memory (by page faulting). Some important things to note about >>> this scenario: >>> >>> 1.) Only address 0x1000 in THIS process are backed by the RAM page >>> 0x30000 (THIS process means the process we are talking about in the >>> scenario above). >>> 2.) Another process's VA 0x1000 will be backed by a different RAM >>> page or not backed at all. >>> >>> Does this make this more clear? >>> >>> >>> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >>> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>>> Is this statement correct? I continue to read articles (like this >>>> one http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about >>>> memory that talk about a processes "own" address space. >>>> >>>> Assuming 2GB in UM... >>>> >>>> When a process starts up, it is presented with all 2GB of UM address >>>> space, but it can request to actually use much less. The memory >>>> manager determines which memory is available for use. The amount of >>>> memory used by this process is reflected in the Working Set and >>>> Private Bytes counters. >>> > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/18/2007 6:21:00 PM |
| From: "the_el_vez" [Email Address Protection] |
It is specifically called a "demand zero" page fault, which means it is a page fault that requires a zero filled page. A "hard" page fault is when a page has to be read in from disk. A "soft" page fault is a page fault that is satisfied by an already existent RAM page. So in this case the faults would be demand zero page faults, which are a type of soft fault. "Megan Kielman" <megan.kielman@gmail.com> wrote in message news:eoWgD6YQIHA.4880@TK2MSFTNGP03.phx.gbl... > Is the type of page fault you are demonstrating with your program called a > "Soft" Page Fault? > > > the_el_vez wrote: >> No. It is pretty much fully dynamic (in Vista and beyond). Obviously >> there are kernel areas that are always backed (non-paged pool, etc.) but >> the memory manager can give and take away as required - kernel or user VA >> backing. >> >> As far as page faults go - the first time a page is backed by memory that >> is a special kind of fault - it is not read from the page file. Once it >> is trimmed and put into the page file - the next fault (if one occurs for >> that address) will be read in from the page file. >> >> An interesting app to write is one that allocates VA, writes to and frees >> it. If you watch the page fault counters when you run this - you will see >> the pagefaults are pretty much linear with the size of MAX_ALLOCS. Here >> is some code I just whipped up (should compile :) - I wrote it in the >> mail editor!): >> >> #include <windows.h> >> #include <stdio.h> >> #include <stdlib.h> >> #include <conio.h> >> >> #define PAGE_SIZE 0x1000 >> #define MAX_ALLOCS 100 >> >> >> PULONG Data[MAX_ALLOCS]; >> >> int >> __cdecl >> main(int argc, >> char* argv[]) >> { >> >> volatile int a = 0; >> ULONG Count; >> >> >> >> printf("Allocating blocks...\n"); >> >> >> // >> // Allocate the blocks >> // >> >> for (Count = 0; Count < MAX_ALLOCS; Count++) { >> >> Data[Count] = (PULONG)VirtualAlloc(NULL, >> MEM_COMMIT | MEM_RESERVE, >> PAGE_SIZE, >> PAGE_READWRITE); >> >> if (!Data[Count]) { >> printf("Failed to allocate Data[%X]. Error == %d\n", >> Count, >> GetLastError()); >> } >> } >> >> >> printf("Writing blocks...\n"); >> >> // >> // Write to the blocks >> // >> >> for (Count = 0; Count < MAX_ALLOCS; Count++) { >> >> if (Data[Count]) { >> *Data[Count] = 0x1234; >> } >> >> } >> >> >> printf("Freeing blocks...\n"); >> >> // >> // Free the blocks >> // >> >> for (Count = 0; Count < MAX_ALLOCS; Count++) { >> >> if (Data[Count]) { >> VirtualFree(Data[Count], >> PAGE_SIZE, >> MEM_RELEASE); >> } >> >> } >> >> >> printf("Press \'y\' to continue\n"); >> >> while (a != 'y') { >> a = _getch(); >> } >> >> >> return 0; >> } >> >> >> There will be some floor (probably around 300 pages - just to get the app >> loaded and running), but after that you should be able to adjust the size >> of MAX_ALLOCS and see the increase/decrease. >> >> >> This posting is provided "AS IS" with no warranties, and confers no >> rights >> >> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >> news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... >>> Yes this makes things infinitely more clear. Thank you for your patience >>> with my questions! >>> >>> You mentioned that when a process goes to access the memory in the VAD >>> that isn't backed by physical RAM, it creates a page fault. It makes me >>> wonder then when looking at the Page Fault counter in Process Explorer, >>> I always assumed this meant that the process was doing a log of paging >>> (to the page file) but it could also mean that the process is attempting >>> to access memory that has never been backed by memory. Does that make >>> the Page Fault counter not a good way to determine if the system is >>> paging out to page file a lot? >>> >>> Also, I am trying to understand how kernel memory allocation fits in. >>> Because User mode memory is always mapped to the lower half of the >>> address space and kernel mode is always mapped to the upper half, does >>> that always mean that no matter how much physical RAM, 1/2 will be >>> dedicated to kernal? >>> >>> I am really sorry if my questions aren't clear. I am doing the best I >>> can to articulate what I am trying to ask. >>> >>> >>> >>> the_el_vez wrote: >>>> Ok - so let's try to take this topic from the top down. >>>> >>>> There are structures maintained in the kernel - for each process in the >>>> system - called page tables (PTE's and PDE's on x86). These tables are >>>> constructed such that a virtual address in a process can be broken >>>> down and decoded into a physical address in RAM by the processor. Each >>>> process (NOT PROCESSOR) in Windows has a separate set of these tables. >>>> In addition to the page tables, each process has a virtual address >>>> space that is 4gb in size (0 - 0xffffffff). So now the real question >>>> you are asking is how do these 2 things relate. RAM in the machine and >>>> virtual address space are totally independent of each other. >>>> Theoretically a process can be using no RAM at all - but always has its >>>> 4gb address space. So now what happens when we want to allocate some of >>>> our 4gb of virtual address space? Well the memory manager creates >>>> something called a virtual address descriptor (VAD) in response to a >>>> request to allocate some of our process VA. A VAD describes a region of >>>> VA space in a single process. There is a seperate set of VAD's for each >>>> process. The VAD still has no relation to RAM. Let's say our VAD >>>> describes address 0x1000 in our process's VA. So at this point we have >>>> a 4gb VA space, a VAD describing some of that VA space. So now you >>>> execute some code in your process that writes to address 0x1000. The >>>> first thing that happens is the processor takes the address 0x1000 and >>>> looks in the process's page tables to find the RAM that backs it. Since >>>> there is no page table entry for 0x1000, we get a page fault. A page >>>> fault occurs when the processor references a virtual address that isn't >>>> backed by RAM - meaning that there is no page table entry for our >>>> process that decodes the VA to RAM. The page fault triggers code in MM. >>>> The MM code looks at the address that faulted (in our case 0x1000). MM >>>> then looks for a VAD in our process's VAD's that describes that >>>> address. Lucky for us it finds one (if it didn't find one we would wind >>>> up with an access violation exception). When MM finds the VAD it says >>>> "oops - this process is supposed to have some memory here but I am a >>>> big fat liar and didn't back this address with any RAM yet because I >>>> didn't think the process would really use it.. I'd better do that >>>> now!". Then the MM finds some available page of RAM (say RAM address >>>> 0x30000) and updates the page tables (for the process that page >>>> faulted) to use the RAM page 0x30000 as backing for VA 0x1000 in the >>>> current process and re-executes the instruction that started this whole >>>> chain of events. BTW - the reason MM doesn't immediately back the >>>> address with RAM when you call allocate() - is because most allocated >>>> VA is never used. So MM waits till you PROVE that you need the memory >>>> (by page faulting). Some important things to note about this scenario: >>>> >>>> 1.) Only address 0x1000 in THIS process are backed by the RAM page >>>> 0x30000 (THIS process means the process we are talking about in the >>>> scenario above). >>>> 2.) Another process's VA 0x1000 will be backed by a different RAM page >>>> or not backed at all. >>>> >>>> Does this make this more clear? >>>> >>>> >>>> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >>>> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>>>> Is this statement correct? I continue to read articles (like this one >>>>> http://msexchangeteam.com/archive/2005/12/07/415733.aspx) about memory >>>>> that talk about a processes "own" address space. >>>>> >>>>> Assuming 2GB in UM... >>>>> >>>>> When a process starts up, it is presented with all 2GB of UM address >>>>> space, but it can request to actually use much less. The memory >>>>> manager determines which memory is available for use. The amount of >>>>> memory used by this process is reflected in the Working Set and >>>>> Private Bytes counters. >>>> >> |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/19/2007 11:25:55 AM |
| From: Megan Kielman [Email Address Protection] |
Thank you! the_el_vez wrote: > It is specifically called a "demand zero" page fault, which means it is > a page fault that requires a zero filled page. A "hard" page fault is > when a page has to be read in from disk. A "soft" page fault is a page > fault that is satisfied by an already existent RAM page. So in this case > the faults would be demand zero page faults, which are a type of soft > fault. > > "Megan Kielman" <megan.kielman@gmail.com> wrote in message > news:eoWgD6YQIHA.4880@TK2MSFTNGP03.phx.gbl... >> Is the type of page fault you are demonstrating with your program >> called a "Soft" Page Fault? >> >> >> the_el_vez wrote: >>> No. It is pretty much fully dynamic (in Vista and beyond). Obviously >>> there are kernel areas that are always backed (non-paged pool, etc.) >>> but the memory manager can give and take away as required - kernel or >>> user VA backing. >>> >>> As far as page faults go - the first time a page is backed by memory >>> that is a special kind of fault - it is not read from the page file. >>> Once it is trimmed and put into the page file - the next fault (if >>> one occurs for that address) will be read in from the page file. >>> >>> An interesting app to write is one that allocates VA, writes to and >>> frees it. If you watch the page fault counters when you run this - >>> you will see the pagefaults are pretty much linear with the size of >>> MAX_ALLOCS. Here is some code I just whipped up (should compile :) - >>> I wrote it in the mail editor!): >>> >>> #include <windows.h> >>> #include <stdio.h> >>> #include <stdlib.h> >>> #include <conio.h> >>> >>> #define PAGE_SIZE 0x1000 >>> #define MAX_ALLOCS 100 >>> >>> >>> PULONG Data[MAX_ALLOCS]; >>> >>> int >>> __cdecl >>> main(int argc, >>> char* argv[]) >>> { >>> >>> volatile int a = 0; >>> ULONG Count; >>> >>> >>> >>> printf("Allocating blocks...\n"); >>> >>> >>> // >>> // Allocate the blocks >>> // >>> >>> for (Count = 0; Count < MAX_ALLOCS; Count++) { >>> >>> Data[Count] = (PULONG)VirtualAlloc(NULL, >>> MEM_COMMIT | MEM_RESERVE, >>> PAGE_SIZE, >>> PAGE_READWRITE); >>> >>> if (!Data[Count]) { >>> printf("Failed to allocate Data[%X]. Error == %d\n", >>> Count, >>> GetLastError()); >>> } >>> } >>> >>> >>> printf("Writing blocks...\n"); >>> >>> // >>> // Write to the blocks >>> // >>> >>> for (Count = 0; Count < MAX_ALLOCS; Count++) { >>> >>> if (Data[Count]) { >>> *Data[Count] = 0x1234; >>> } >>> >>> } >>> >>> >>> printf("Freeing blocks...\n"); >>> >>> // >>> // Free the blocks >>> // >>> >>> for (Count = 0; Count < MAX_ALLOCS; Count++) { >>> >>> if (Data[Count]) { >>> VirtualFree(Data[Count], >>> PAGE_SIZE, >>> MEM_RELEASE); >>> } >>> >>> } >>> >>> >>> printf("Press \'y\' to continue\n"); >>> >>> while (a != 'y') { >>> a = _getch(); >>> } >>> >>> >>> return 0; >>> } >>> >>> >>> There will be some floor (probably around 300 pages - just to get the >>> app loaded and running), but after that you should be able to adjust >>> the size of MAX_ALLOCS and see the increase/decrease. >>> >>> >>> This posting is provided "AS IS" with no warranties, and confers no >>> rights >>> >>> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >>> news:OOFGgRdPIHA.3388@TK2MSFTNGP03.phx.gbl... >>>> Yes this makes things infinitely more clear. Thank you for your >>>> patience with my questions! >>>> >>>> You mentioned that when a process goes to access the memory in the >>>> VAD that isn't backed by physical RAM, it creates a page fault. It >>>> makes me wonder then when looking at the Page Fault counter in >>>> Process Explorer, I always assumed this meant that the process was >>>> doing a log of paging (to the page file) but it could also mean that >>>> the process is attempting to access memory that has never been >>>> backed by memory. Does that make the Page Fault counter not a good >>>> way to determine if the system is paging out to page file a lot? >>>> >>>> Also, I am trying to understand how kernel memory allocation fits >>>> in. Because User mode memory is always mapped to the lower half of >>>> the address space and kernel mode is always mapped to the upper >>>> half, does that always mean that no matter how much physical RAM, >>>> 1/2 will be dedicated to kernal? >>>> >>>> I am really sorry if my questions aren't clear. I am doing the best >>>> I can to articulate what I am trying to ask. >>>> >>>> >>>> >>>> the_el_vez wrote: >>>>> Ok - so let's try to take this topic from the top down. >>>>> >>>>> There are structures maintained in the kernel - for each process in >>>>> the system - called page tables (PTE's and PDE's on x86). These >>>>> tables are constructed such that a virtual address in a process >>>>> can be broken down and decoded into a physical address in RAM by >>>>> the processor. Each process (NOT PROCESSOR) in Windows has a >>>>> separate set of these tables. In addition to the page tables, each >>>>> process has a virtual address space that is 4gb in size (0 - >>>>> 0xffffffff). So now the real question you are asking is how do >>>>> these 2 things relate. RAM in the machine and virtual address space >>>>> are totally independent of each other. Theoretically a process can >>>>> be using no RAM at all - but always has its 4gb address space. So >>>>> now what happens when we want to allocate some of our 4gb of >>>>> virtual address space? Well the memory manager creates something >>>>> called a virtual address descriptor (VAD) in response to a request >>>>> to allocate some of our process VA. A VAD describes a region of VA >>>>> space in a single process. There is a seperate set of VAD's for >>>>> each process. The VAD still has no relation to RAM. Let's say our >>>>> VAD describes address 0x1000 in our process's VA. So at this point >>>>> we have a 4gb VA space, a VAD describing some of that VA space. So >>>>> now you execute some code in your process that writes to address >>>>> 0x1000. The first thing that happens is the processor takes the >>>>> address 0x1000 and looks in the process's page tables to find the >>>>> RAM that backs it. Since there is no page table entry for 0x1000, >>>>> we get a page fault. A page fault occurs when the processor >>>>> references a virtual address that isn't backed by RAM - meaning >>>>> that there is no page table entry for our process that decodes the >>>>> VA to RAM. The page fault triggers code in MM. The MM code looks at >>>>> the address that faulted (in our case 0x1000). MM then looks for a >>>>> VAD in our process's VAD's that describes that address. Lucky for >>>>> us it finds one (if it didn't find one we would wind up with an >>>>> access violation exception). When MM finds the VAD it says "oops - >>>>> this process is supposed to have some memory here but I am a big >>>>> fat liar and didn't back this address with any RAM yet because I >>>>> didn't think the process would really use it.. I'd better do that >>>>> now!". Then the MM finds some available page of RAM (say RAM >>>>> address 0x30000) and updates the page tables (for the process that >>>>> page faulted) to use the RAM page 0x30000 as backing for VA 0x1000 >>>>> in the current process and re-executes the instruction that started >>>>> this whole chain of events. BTW - the reason MM doesn't immediately >>>>> back the address with RAM when you call allocate() - is because >>>>> most allocated VA is never used. So MM waits till you PROVE that >>>>> you need the memory (by page faulting). Some important things to >>>>> note about this scenario: >>>>> >>>>> 1.) Only address 0x1000 in THIS process are backed by the RAM page >>>>> 0x30000 (THIS process means the process we are talking about in the >>>>> scenario above). >>>>> 2.) Another process's VA 0x1000 will be backed by a different RAM >>>>> page or not backed at all. >>>>> >>>>> Does this make this more clear? >>>>> >>>>> >>>>> "Megan Kielman" <megan.kielman@gmail.com> wrote in message >>>>> news:OhjaW5NPIHA.1164@TK2MSFTNGP02.phx.gbl... >>>>>> Is this statement correct? I continue to read articles (like this >>>>>> one http://msexchangeteam.com/archive/2005/12/07/415733.aspx) >>>>>> about memory that talk about a processes "own" address space. >>>>>> >>>>>> Assuming 2GB in UM... >>>>>> >>>>>> When a process starts up, it is presented with all 2GB of UM >>>>>> address space, but it can request to actually use much less. The >>>>>> memory manager determines which memory is available for use. The >>>>>> amount of memory used by this process is reflected in the Working >>>>>> Set and Private Bytes counters. >>>>> >>> > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/20/2007 6:41:09 PM |
| From: "David Craig" [Email Address Protection] |
I think there are two or more reasons. 1) a page was allocated and never written to, so since it never became dirty the page can be recreated by just grabbing any free page and mapping it into that process. 2) when running driver verifier one option is to have it force every page that is not part of the non-paged pool or not locked down out by just marking the page tables as if the page wasn't really there. Since it usually has the real memory space available during this test, it just leaves the pages in memory as they were to minimize disk activity. Driver Verifier just wants to know if you touched a page that was not "resident" or could be non-resident in normal operation while at elevated IRQL. On the first reason, it could be that the memory manager was trying to free up pages, but something such as an interrupt/DPC occurred and the driver scheduled a worker thread to run and it needs the pages in the process of being paged out. You could get headaches trying to figure out all possible reasons, especially without source code to the OS. "Melvin (math) Klassen" <mklassen@Math.UVic.CA> wrote in message news:fkf1s1$17058$1@casa.comp.uvic.ca... >> A "hard" page fault is when a page has to be read in from disk. > >> A "soft" page fault is a page fault that is satisfied by an already > existent RAM page. > > Why is a "soft" page fault generated, when the RAM page already is > existent > ? > Is that page of RAM already being used by a different process? > > Thanks. > > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/20/2007 9:24:39 PM |
| From: "the_el_vez" [Email Address Protection] |
A page fault occurs when a virtual address is referenced by the processor and that virtual address is not backed by a RAM page. If the RAM page has to be filled from a page on disk in order to satisfy the fault (like it would in the case where a page was paged out to disk and then brought back in) it is called a "hard fault". If the fault can be satisfied without filling a RAM page from disk (like in the case where a process needs a page filled with zeros) then it is called a "soft fault". Other times when a soft fault can occur is when a page is on its way to disk (the paging process occurs in several stages). If a page is referenced again while it is still in the paging process it can be given back to the process. This is also called a "soft fault". There are a lot of stages and semantics in the paging process (all covered in the "Windows Internals" book) if you want more specifics. "Melvin (math) Klassen" <mklassen@Math.UVic.CA> wrote in message news:fkf1s1$17058$1@casa.comp.uvic.ca... >> A "hard" page fault is when a page has to be read in from disk. > >> A "soft" page fault is a page fault that is satisfied by an already > existent RAM page. > > Why is a "soft" page fault generated, when the RAM page already is > existent > ? > Is that page of RAM already being used by a different process? > > Thanks. > > |
| Back |
| Subject: Re: Still confused about memory allocation |
| Group: microsoft.public.winternals |
| Date: 12/20/2007 9:25:51 PM |
| From: "the_el_vez" [Email Address Protection] |
A page fault occurs when a virtual address is referenced by the processor and that virtual address is not backed by a RAM page. If the RAM page has to be filled from a page on disk in order to satisfy the fault (like it would in the case where a page was paged out to disk and then brought back in) it is called a "hard fault". If the fault can be satisfied without filling a RAM page from disk (like in the case where a process needs a page filled with zeros) then it is called a "soft fault". Other times when a soft fault can occur is when a page is on its way to disk (the paging process occurs in several stages). If a page is referenced again while it is still in the paging process it can be given back to the process. This is also called a "soft fault". There are a lot of stages and semantics in the paging process (all covered in the "Windows Internals" book) if you want more specifics. "Melvin (math) Klassen" <mklassen@Math.UVic.CA> wrote in message news:fkf1s1$17058$1@casa.comp.uvic.ca... >> A "hard" page fault is when a page has to be read in from disk. > >> A "soft" page fault is a page fault that is satisfied by an already > existent RAM page. > > Why is a "soft" page fault generated, when the RAM page already is > existent > ? > Is that page of RAM already being used by a different process? > > Thanks. > > |
| Back |