QUOTE: (dencorso @ Dec 31 2007, 03:03 AM)
Some time ago, Tihiy (in this post) pointed to a russian language page, by Igor Petrovitch Leyko (Nov 03, 2005), about win 98 memory problems (link), and while no automatic translator did a good job at translating it, Google Language Tools gave me a good start, and from that start I've done my best to produce a good enough English translation of a short excerpt of that page, which I find relevant to the present discussion:
dencorso,
Well done job!
It inspired me to expand the translation a bit further, with your piece copied and pasted almost in its entirety (the ending of the last sentence was slightly modified to more accurately reflect the original.) This translation is approximately half of the original text (the second part of it.) If /when I have more inspiration, I will translate the introductory part of the article as well. It should be mentioned here, that in Russian forums Igor Leyko (Microsoft MVP) is a well regarded person, respected for his deep and intimate knowledge of Win Oses.
GreyPhound
Windows 9x and lots of memory:
Let's start with traditional "excuses". The functions in question, OS work specifics and applied algorithms are documented extremely poorly, or not documented at all. The information I have mined from various sources seems to be partially contradictory. I did my best to put all the pieces together, but no guarantee can be given that I’m not mistaken in some way. My attempts to reach out to people with good knowledge of internal modules of the OSes mentioned here have failed, as too many years passed. And in some cases nothing but reconstruction had to be done, similar to how with a few found bones appearance of a dinosaur would be identified. Therefore, if you find a mistake or inaccuracies, your comments would be appreciated. Now, let's get back to the subject at hand.
Although Windows 98 and Windows Me can surely be described as outdated systems, they still run on quite a lot of computers. For various reasons: computer power is not enough to "haul" a modern OS, or because of programs and hardware that will not work under Windows 2000 and XP.
And if old machines will have only one problem with memory - its deficiency, the situation with newer ones seems to be more complicated.
For example a company has a certain accounting program on an old machine written many years ago, and it still suites the needs of the user. If for some ill luck the old computer is broken, on a new one the program would not work under XP. Or someone has an old scanner, which is used occasionally and the owner is happy with it. But WinXP drivers for the scanner do not exist, you grudge buying a new one - and why shouldn't you, if the old one works!
The most evident solution in these circumstances is to install Windows 98 or Windows Me on a new computer, either in a dual boot configuration, or as a single system.
However, if your computer has a fair amount of memory, then these OSes could fail to install at all, or hang during work every now and then. It is not for nothing, that in forums and discussion groups, you hear statements about Windows 98 being unable to utilize more than 512 Mb of RAM.
Let's try to find out what's the problem here. Let's exclude at once situations when new hardware doesn't have drivers for Windows 98 (it is presumed here that Win98, if not otherwise mentioned, will comprise three operating systems Win98, Win98SE and WinMe), hyperthreading is not switched off in Pentium 4 processors and so on. We are talking about only one thing: specifics of how Windows 98 handles large amounts of installed RAM.
As a rule, with memory not more than 512 Mb you are unlikely to run into problems. With an extremely unfortunate combination of hardware and programs in use, that occupy a small amount of memory, but are processing large amounts of data, a system may hang on rare occasions even with smaller amounts of RAM, however this situation will not happen often and will usually be chalked up to system glitches.
But if memory is more than 512 Mb, problems will occur more often, leading to a complete hang, or making it impossible to install a system or load it when memory is added after installation. Or the computer will go into cyclical rebooting.
Extraction of "redundant" modules will usually prove to be a radically effective remedy, and the problems will be gone as if by magic. But this solution can hardly be called satisfying - not everyone will agree to open his computer and extract memory before Win98 launch and put it back when starting WinXP.
So what a smart user would do in this juncture? That's right: he will go to Microsoft knowledgebase site. And with some search experience he will quickly find the articles which describe this problem or similar ones: 184447, 253912, 304943, 311871.
It may well happen that after reading them your head will be swimming. Therefore, let's figure out what is written in them and how one is to comprehend that.
Carefully analyzed, the article 311871 can be set aside. Its style and comprehensiveness makes it stand out from the other ones, and it can be presumed with certainty that it was written by one of my MVP colleagues, undoubtedly with good knowledge of Windows 98 but not familiar enough with its inner layouts.
An assertion that this system is not meant to use 1 Gb of RAM is incorrect. It was designed to be used with up to 2 Gb, but as a rule, the actual limit, due to solutions inherited from preceding versions is less.
I witnessed (by correspondence, in a discussion group) the very first case of Windows 98 with a large amount of memory working abnormally. In February 1998, one of the beta testers attempted to install Windows 98 (probably, it was RC0, not beta already) on a server with one gigabyte of memory. During installation a blue screen with a message about insufficient memory to initialize Windows popped up. The installation could only be continued after the memory had been reduced to 928 Mb.
Not a single developer has ever mentioned that the system was not designed to utilize a gigabyte. Quite the contrary, the answer was something like: "the system is designed to use up to 2 Gb of RAM, therefore it must be a mistake of some sort, but for lack of time we will not look into it right now due to shipping deadline, and then it's unlikely an average user will ever encounter this issue." Undoubtedly a fair statement, if you remember a typical computer configuration back at that time. Suffice it to say that the gigabyte in that ill-fated server was made up using 16 Mb modules.
Other knowledgebase articles say that disk cache should be limited to 512 Mb, and the amount of RAM to 1 Gb, 768 Mb or even to 512 Mb - various articles will have it differently.
The article 253912 mentions briefly allocation of addresses to cache in the fourth gigabyte, it also says about a possibility of limiting the size of cache. But article 304943 says that in some cases limiting cache doesn't help, and that the amount of used RAM should be reduced.
So the knowledge base acknowledges that with large amounts of RAM Windows may refuse to boot, or will work unreliably. But what's the issue then if the developers talked about two gigabytes with confidence?
So it's clear that nothing is clear. Nobody seems to know just everything, and if somebody does - then he wouldn't tell you, and "trimming" or physical reduction of RAM is often presented as the only viable option. Let's get down to the gist of the problem.
Let's start with how Windows utilizes memory.
32 bit address space in 80386 (and up) processors is divided into memory pages sized 4 Kb each. Each of these pages is managed independently form one another. There maybe many instances of such address space, but actual (physical) address space is available only to the system kernel and drivers, and only they can directly address physical memory. Other system components and all application programs work in virtual address space, with virtual addresses.
The concept of "virtual memory" is somewhat ambiguous. Quite often it is used as contradistinction to the concept of "physical memory", but in general, virtualization should be understood as a form of abstracting from a real device. Obviously, a program can only be executed from physical memory - and not when its code is unloaded to the swap file. However, a program itself usually can not determine if its pieces are in physical memory, or unloaded to the disk. Nor can it determine where exactly in physical memory its code and data is - the operating system translates RAM addresses to which a program calls in its virtual address space, into those addresses that a program actually occupies in physical memory. If part of the virtual memory addressed by a program is unloaded to the disk, the OS halts the program and loads a needed page to the physical memory, and the program afterwards will proceed as if that page was always in the memory.
To provide for such a translation, the operating system maintains a special table of pages, in which currently mapped virtual pages, physical memory areas and swap file areas where data is really kept are indexed.
There are several such tables - for every program a separate table is kept. Different programs may use one and the same virtual addresses, but each program will have its own map of virtual and real pages, so that in reality different physical memory areas will be addressed.
Since we have touched memory allocation, let me mention here, that memory is allocated to a program only on its calls. The operating system can deny a program memory allocation, but it can not allocate memory to a program more than it has called for. Therefore, questions asked sometimes: "Why does not the OS release all the memory to my favorite program, when we still have lots of free memory" are pointless. To "overfeed" a program with memory, that is - force it into using more than it wants itself is impossible.
So, we have figured out that physical addresses space should be distinguished from virtual one. One of the operating system kernel components - memory manager, among other things takes care of translating virtual addresses used by programs and other kernel components to physical memory actual addresses.
In Windows 98 memory is allocated as follows (see the diagram below.) On the left - actual allocation of address space of programs and the system is shown, on the right side, an example of allocation of physical memory.
All programs virtual address space is divided into four areas. The first area occupies the range of addresses from zero to 4 Mb and is not addressable by Windows programs. For DOS programs and drivers the situation is slightly different, but discussing those specifics will distract us from the topic, therefore we will not go into details here.
The area from 4 Mb to 2 Gb is the working area for programs. From a programs’ standpoint this area is filled with RAM up to the quantity needed by a program (and of course, not more than 2 Gb), as already mentioned, each program has its own memory, which has nothing to do with the memory allocated to other programs.
The third gigabyte is used as virtual address space, common for all the programs and the system. Windows loads its graphical core into this area, as well as dynamic libraries. Obviously, in reality these modules are located in the actually available physical memory in areas of significantly lower addresses, but a mechanism is needed to enable programs to address the system modules. Virtualization is such a mechanism.
The fourth gigabyte (addresses from c0000000-ffffffff) is also common for all programs and the system, but it is used for several purposes. Firstly, the system loads its kernel there, drivers and other modules which work on kernel level (if there are such modules in the system.) This area is virtualized just as the third gigabyte is.
Secondly, in this address space an area of memory is allocated to be used by disk cache (VCACHE), also virtualized.
Third,in this same gigabyte there are memory addresses (actual, not virtualized) of those IO devices that provide for direct access to their memory.
Fourth, it is here that address space is allocated for DOS virtual machines.
Let's see more details of how the fourth gigabyte is used.
The system kernel is allocated address space in the same way as common modules, and doesn't need additional explanations; therefore let's get straight to talking about the disk cache.
When more than ten years ago, Microsoft started developing its dynamically changing disk cache (VCACHE module first appeared in Windows 3.11 for work groups, released in 1993), gigabyte memory sizes seemed as something from science fiction. For those who didn't know, the first IBM PC, released in 1980 had 64 Kb of RAM, and in 1993 computers with four megabytes were the norm. Extrapolating to our time, it would seem that in 2006 one would expect 256 Mb of RAM in mass manufactured computers. Besides, no one even thought, that this OS and its direct offspring would last this long.
So programmers had no reason to be concerned with economizing address space, and openhandedly the fourth gigabyte was allotted among other things, also to address space of the disk cache. It turned out to be less to one's cost to allot linear virtual address space to variable size cache, and afterwards map it through a memory manager to actually used physical memory, than constantly taking stock of allocated memory in the cache module itself.
For this reason, during system boot, part of the address space in the fourth gigabyte is reserved for use by the disk cache. The size of this part depends on the operating system version, some hardware characteristics, and amount of RAM installed on the computer. Usually it equals the amount of RAM, but has an upper limit of 800 Mb.
Another purpose for using the fourth gigabyte is to provide access to device memory. If you lived in the times of DOS, you should know that to boost work speed, it was not uncommon for programs to write their data directly to device memory, video adapter for example (EGA or VGA.)
With advent of 32 bit processor work mode, nothing has changed in this regard. As before, direct write remains the quickest way of calling a device, only device memory addresses have been removed from the first megabyte to the end of the area being addressed, to that same fourth gigabyte. And also, there is usually a layer in the form of a driver
between a program and a device.
DOS virtual machines also have their address allocation in this very fourth gigabyte, where your DOS programs are executed. With all this going on, there comes the need for a dual retranslation: when a program calls an address in the first megabyte, instead of it a corresponding address of a virtual machine is given, (in the fourth gigabyte), and since this is a virtual address, a second redirection is taking place - to an address of the physical memory, where a required page of the virtual machine is located. Such is the cost of support for old programs. However, the actual burden isn't that big.
But let's get back to the common problems of the fourth gigabyte.
Unlike Windows XP and other NT based OSes, Windows 98 has one, and only one address space for two upper gigabytes. It turns out to be shared by both virtual memory and device addresses, and it has to be sufficient to be used in all modes, otherwise the work of the system will be broken.
As a rule, the OS can't influence the use of addresses of the device memory - these addresses, in most cases, are fixed by the hardware.
Let's see now how memory is allocated during Windows boot. The very first phase is DOS boot. Afterwards, from there Windows kernel is loaded - VMM32.VXD. This file has many modules, and one of them is VM manager VMM.VXD (in particular, it manages memory allocation,) by its structure it's a plain DOS program. But only by its structure. And only to make it possible to be started from DOS. Once started, it shifts the processor to protected mode and uses DOS mainly as a driver for various services, first of all the disk and video adapter.
Then VM manager starts loading into memory other modules (drivers) which are located in VMM32.VXD, drivers which are referenced in the registry or in system.ini, and also those called earlier by loaded DOS programs (as an example, disk caching program smartdrv can be given, which on Windows boot passes a demand to it to load the smartdrv.vxd module, which is kept in the body of smartdrv.exe.)
But here we have a vicious circle. To load a driver to memory, this memory needs to be allocated to it, and while being allocated - correspondingly indexed in the pages table. And to allocate space for pages table, memory needs to be distributed in advance. Therefore, VM manager performs allocation in two moves: first, a small, but absolutely available area of memory is allocated, into which driver bodies are loaded. A table of pages is also built there. In the second phase of actual initialization, drivers' requirements for memory are analyzed and they are allocated virtual addresses in the fourth gigabyte.
If your boot log is on during Windows boot, these phases will be clearly visible there (bootlog.txt). First come pairs of messages Loading and LoadSuccess for each driver, VM manager and drivers initialize: SYSCRITINIT and SYSCRITINTSUCCESS.
The main "hog" of address space, undeniably is VCACHE. As mentioned earlier, it demands for itself address space approximately equal to the amount of RAM, but not more than 800 Mb. As result, total demands of this and other modules can exceed the gigabyte allotted to them. WinMe in this respect is more effective and more efficiently allocates address space, therefore given a choice, it is more preferable to chose it over the other.
But if it's still impossible to allot virtual memory required by the system and all the drivers, then as a rule, Windows 98 will give a warning about insufficient memory, and Windows Me will reboot (although it can be the other way round too). But since there will not be enough address space next time - rebooting becomes cyclic. At that, there is no initialization entry in the log, it ends when drivers are loaded.
If address space deficiency is not too big, it's possible that the system will load in safety mode, when the number of drivers and the required address space decrease. But don't hold your breath.
Sometimes you have enough address space for initialization, but you have practically, almost run out of stock. In this case the system will boot normally, but an attempt to start a DOS program will lead to a mistake warning. If free address space still remains, but not too much of it, then some DOS programs may start, but those that require more memory – will fail.
But what can be done if there's not enough address space and the system will not boot, or DOS programs do not start? Alas, the problem can only be solved in two ways. If you have Windows 98, you may try switching to Windows Me. The second, more universal, but less productive way is to limit the amount of the used memory (detailed instructions are in the end of this article).
Apparently, you have a question: if the issue is disk cache that does not have enough virtual address space, then why not shrink its size, as Microsoft knowledgebase would suggest? Unfortunately, it will not help. As a matter of fact, memory allocation is performed by VM manager, and it looks only to its own section of the system.ini file [386Enh]. And cache size limitation is determined in some other section, and it affects only how cache itself will work, in particular how it will use its allocated address space.
However, shrinking the size of cache most probably will prove to be a useful operation. Why? That's what exactly we'll talk about now.
So, the system boots, DOS programs can start, but no work can be performed - shortly thereafter the system either hangs or it shows a blue screen of mistake. What else does it need?
Let’s draw an approximated address space layout in the fourth gigabyte (this is how it looks on authors' Windows 98 computer, some details have been omitted for simplification); at that we will have two diagrams: one for physical space, the other for virtual one.
Note: To avoid misunderstanding, it should be mentioned that booting only Windows kernel without the graphical interface has nothing to do with booting with BootGUI=0 parameter set. In the latter case, Windows as a matter of fact does not load at all; it is only MS-DOS, which is part of it that loads. However, some documentation mentions also "real kernel mode Windows".
Also, note that part of the addresses range occupied by an AGP video adapter has been occupied in two instances (if the adapter uses PCI-E bus, the picture will change less, and therefore we will limit ourselves with the most common situation.)
If this address area is being used by a video adapter, then why the system has allocated it to the cache virtual memory?
The answer is very simple: modern display drivers usually consist of two parts - the core, in the form of a virtual device driver (vxd) and a main part, in the form of dynamically loaded libraries. When the system kernel loads, only the core of a video adapter is loaded too. It only provides for realization of basic functions of the video adapter, approximately at the VGA level, that's why it does quite well with corresponding addresses area in the first megabyte or in addition to that, small areas of memory occupied by the adapter in the upper addresses of the fourth gigabyte. When initializing memory VM manager allocates to the video adapter core the area it has called for, but it can not get the information that later at graphical subsystem and all driver files load, this driver will need some more areas of the address space. So the manager allots free areas to VCACHE address space.
Afterwards the graphical core of Windows is started, and with it comes the video adapter driver in full. This driver initializes all the functionality of the video adapter and in the addresses zone used by the adapter a physical device appears. If the same zone is also allotted to the disk cache, when VCACHE is trying to address it a conflict arises between the device driver and cache. Normal work of the system is interrupted.
It's important to know that the conflict arises not because cache has been allocated this area of addresses, but because it starts to work with it. This makes avoiding conflicts pretty simple: all that needs to be done is to limit a maximum cache size.
As a rule, AGP devices use addresses starting from E0000000, that is 3,5 Gb. Disk cache at that can use a half gigabyte. This is exactly what determines a recommendation to limit cache to the value of 524288 Kb (512 Mb). Normally, such a limit will be effective. But not always a video adapter will be allocated exactly this address area, besides other devices may behave in similar ways. TV tuners, for example. Therefore, more radical limitation of cache may be required.
Let's get to practical recommendations now - how to install Windows 98 on a computer with more than 512 Mb of RAM.
1. What's needed done first is to forget, once and for all, EMM386 and other memory managers, more or less stable work, or even (system) booting can't be guaranteed with them.
It is also desirable not to place an explicit call for himem.sys in config.sys - let Windows load it automatically. Although, I have no explanation whatsoever for this piece of advice and can't myself understand why does it have to be this way, but in some cases, it does work.
2. When installing WinMe, have a boot disk, or some other bootable media. It should come with a text editor, as you may need to edit the system.ini. file.
For a Win98 installation no diskette is required, as DOS can always be loaded (Command prompt only) and edit started from there.
3. Let's start an installation. Three options are available:
1. Leave less than, or 512 Mb, in the computer, do a system install and tune up, and only afterwards add all the memory, followed by another tune up.
2. Tune up is performed in advance - stricter than required - and after the installation, reduce the limitations so long as the system still works.
3. Start a system install, and adjust the parameters as needed.
To my personal liking, point 3.3 is the way to go.
What is needed for point 3.1 is pretty clear: take nearly all the memory out of the machine. If a RAM module is 1 Gb, or more - this option is obviously not acceptable.
To follow point 3.2, prior to installation, the system.ini file needs to be created, with a single short entry. On the disk with Windows to be installed, create an installation directory (usually it's Windows). Create a file named system.ini there, with two lines:
[386Enh]
MaxPhysPage=10000
Once done, you can proceed with installation. These lines will force VM manager to use from the outset only 256 Mb of RAM. If this amount of RAM is what you need, you can keep this setting for your work and not waste your time on adjusting it to the current configuration.
To install as in point 3.3, nothing needs to be done in advance if you are installing Windows 98; however, it's a must to have a bootable diskette (or other media) in case of a WinMe installation.
Start a normal installation. If Windows gives a warning about insufficient memory for initialization, or goes into reloading - in Windows 98, press Ctrl key during load, and select Command prompt only from menu. In Windows Me you will have to boot from a diskette or other media.
Open system.ini file for editing, go to [386Enh] and a line MaxPhysPage=value. In Windows 98 you can start with the value 40000, in WinMe - 60000. Save the file and reload the computer, to try proceed with installation. In case of a failure, re-edit the file and reduce the first digit of the value by one. Continue this procedure till a successful installation.
4. Once the installation is complete, it makes sense to sort out a maximum amount of physical RAM. To do this, repeat once again the procedure in point 3c (?) - only this time increasing the value, until the system stops loading.
Afterwards, go back to the previous value and increase it by a smaller amount. It will be convenient to use an increment reduced by a factor of two. For example, if the system loads with the value of 40000 and does not load at 50000, it's worth trying an average value 48000 (this value is hexadecimal - and it shows a number of pages of physical memory the system can use.) If the system loads - try 4c000, if it doesn't - 44000 and so on. Don't go too far however - one in the fourth order means 16 Mb, and therefore, three zeroes in the low order are not worth touching - the benefit will be scanty.
1. Running programs under DOS may require an additional cap on the amount of memory. Run your DOS programs or games to check on how they work. If a program doesn't load or crashes, reduce the value of the parameter by 1-2 units in the fourth order. Most of the time this will be enough.
5. Setting the disk cache.
This setting does not require matching, but preferably it should be done after all the drivers are installed.
Right-click on the My Computer icon, choose Properties from the menu, then the Device Manager tab. Right-click the Computer icon (at the top of the list) and then choose Properties. In the window that appears, choose Memory. In the resulting list of addresses assigned to devices locate the first entry from the fourth gigabyte (= the range C0000000-FFFFFFFF). The entry you want is the one that has the address nearer to C0000000 or farther from FFFFFFFF (which are two ways of saying the same thing). Subtract C0000000 from that hexadecimal number you just found (using the standard Windows calculator's Hex mode), convert the result to decimal (by selecting the calculator's Dec mode) and divide by 1024 (all of the calculation is easily done with the calculator in scientific mode). The resulting number is the maximum allowable size of the disk cache (it is stated in kilobytes). Open the system.ini file, find the section [VCache] and fill it with a string MaxFileCache = xxx, where xxx is the number you just calculated. If you later add another hardware device to the computer, it may require the repetition of these calculations, and further shrinking of the cache size.
Now, you shouldn't have problems caused by large amounts of memory.
If Windows 98 is the only OS on your computer, let me give you a piece of advice. Do not try to increase the amount of memory above what Windows can work with. In some cases a converse effect can be observed: after adding another (RAM) module, you may need to set a new, lower limit of the usable RAM.
Sometimes, another gigabyte added to a machine that works with 1 Gb of RAM, forces the amount of used RAM to a limit of approximately 900 Mb - in other words not only the amount of used RAM does not increase, but it even decreases. On my computer with 2 Gb, WinMe needs just a slight limiting of memory, and adding a third gigabyte of physical memory, the amount used by Windows has to be limited to 1.5 gigabytes. That is, increasing the actual amount of RAM reduces the amount that the system can use.
So before investing in memory, try checking first how exactly your computer and OS will react to its expansion. In reality, the upper limit most of the time is slightly more than one gigabyte for Windows 98, and slightly less than 2 Gb for WinMe.
Igor Leyko
Nov 03, 2005