Hibernating when you have a large uncached extension

One of the main issues I've faced while developing a software RAID miniport device driver was allocating enough memory during normal runtime (for handling RAID 5 operations) and still being called for hibernation by Windows. If you ever try this out, you'll quickly discover that allocating an uncached extension larger than 900KB will prevent diskdump.sys from even calling your miniport's DriverEntry.

The whole algorithm is completely unknown as to how diskdump decides if it can start the miniport driver or not, in fact even the Microsoft storage devs are at a loss to explain the criteria diskdump has to crank up a miniport for hibernation or crashdump. After many, many hours of experimentation, I have found a way to satisfy diskdump's "criteria" to start up a miniport driver for hibernation or crashdump.

In my situation, my miniport driver was allocating 42MB of uncached memory during normal runtime. This of course prevented diskdump from even calling my DriverEntry routine for hibernation; not a good thing if you intend to get WHQL certification.

So here's the trick to hibernate while allocating a huge uncached extension:


Get rid of your device extension.

This seems to be the key factor for diskdump to "grant" hibernation to continue. How do you do this? Store your device extension data in your driver binary.

• Simply create a global variable in your driver using the structure type of your device extension.

• When specifiying the device extension size during DriverEntry, specify the size of a pointer (remember: the size of a pointer is different for 32-bit and 64-bit drivers). i.e. DeviceExtensionSize = sizeof(u32);

• In your FindAdapter routine, use the pointer space that was allocated by Windows for your device extension to point to your global variable. i.e. *HwDeviceExtension = (u32)global_variable;

• In all other entry points of your miniport, dereference the pointer to get access to your global variable. i.e. devExt_t *devext = (devExt_t *)*HwDeviceExtension;
  

Following this method of having a near-zero size device extension, you can have a very large uncached extension and will still be called for hibernation. Please note that the minimum memory requirements for the product I worked on was 1GB of RAM. I did test this implementation down to 512MB of RAM, but got mixed results.

Also note that the miniport driver I worked on had a very small SRB extension size (size of a pointer), so again, this may be a factor to look at when designing your miniport for this type of scenario.

During this implementation, a bug in the Windows Memory Manager was discovered that prevented this implementation for working properly on certain system configurations. Basically what happens is that after each hibernation, there is a memory leak (true for any miniport). This can build up over many subsequent hibernations (without shutting down in-between), but has not been seen yet in the field because most miniport drivers allocate a very small amount of memory (i.e. 64KB). Because the average size of a miniports uncached extension is extremely small, the corresponding memory leak is very small and does not affect hibernating thousands of times in a row.

In my case, because my uncached extension was 42MB, the memory leak became very noticeable after several hibernations, eventually preventing subsequent hibernations (maybe around 5-10 in a row before hitting the failure). I received a 0xC000009A error (STATUS_INSUFFICIENT_RESOURCES) that hibernation could not be performed; this happened because diskdump.sys couldn't allocate enough resources for itself to get up a running.

This bug in Windows has been fixed by Microsoft and a hotfix is available for download. However, please note that if you follow this implementation, you will then have to require your customers to install this hotfix for hibernation to work. Again, this is a Microsoft bug that is only exagerrated because of the large mount of memory allocated for your uncached extension.

Hibernation / Crashdump details

Having spent numerous hours of trial-and-error discovery of the limits of hibernation and crashdump resources, I thought I'd post them here to save myself time later (and help anyone else searching for these answers). Note that this information was verified on Windows 2000 SP4, Windows XP SP2, and Windows 2003 SP1 & R2 using a ScsiPort miniport driver.

• MaximumTransferLength: The maximum I/O size you'll get from ScsiPort during hibernation and crashdump is 4KB (PAGE_SIZE). This will change in Windows Vista to 64KB.

• The maximum device extension size that can be allocated during hibernation and crashdump is 16KB.
  

• The maximum uncached extension size that can be allocated during hibernation and crashdump is 63KB (despite the DDK documentation claiming the max is 32KB).

• The maximum device extension size that can be allocated during normal runtime, while allowing Windows to call your miniport to hibernate or crashdump, is 900KB.

• The maximum uncached extension size that can be allocated during normal runtime, while allowing Windows to call your miniport to hibernate or crashdump, is 900KB.