Life of every driver developer is complicated by the fact that the code you write should be stable (read: bugs free) and compatible with any other third party drivers. It is very important for any driver to be bugs free, as any error in kernel leads (directly or indirectly) to a blue screen of death (BSOD).
To ease the life of kernel developers Microsoft has implemented quite a few useful tools which helps to locate driver errors and make the code better. Some of them run in runtime (Driver Verifier), some of them are static (prefast) and some of them represent a complex system of test (like DTM for example). In this post, I will concentrate on Driver Verifier, a tool which is making verification tests of driver in runtime. Let me explain what is Driver Verifier, and why it is so useful tool. Taken from http://support.microsoft.com/kb/244617:
Driver Verifier is included in Windows 2000, Windows XP, and Windows Server 2003 to promote stability and reliability; you can use this tool to troubleshoot driver issues. Windows kernel-mode components can cause system corruption or system failures as a result of an improperly written driver, such as an earlier version of a Windows Driver Model (WDM) driver.
So, in general, Driver Verifier is a tool which verifies driver and makes sure it is not buggy. Taking into consideration words above, anything that does not pass Driver Verifier test is not stable and reliable. A complete list of features can be read from http://support.microsoft.com/kb/244617 . Now, when it comes to compatibility of drivers (for example, filter drivers in stack) there is unsaid rule, which says: “if any third party driver does not pass Driver Verifier test, there is no chances to make your driver compatible with it”. Well, words are just words … But in real world it could result into pessimistic situation.
Let’s imagine you develop a driver which resides in some stack and filters it. Your driver is Driver Verifier compatible. You even spent some time and money in order to pass WHQL tests and you got “Windows Logo Certification” for you driver. Now, you are happy as you are sure it is compatible with particular Windows version and it would not BSOD it. You deploy your driver onto customers machine, and it BSODs. You start to dig, and you find out that there is another driver in filtering stack which is causing BSOD when both drivers (yours and another) are installed. As your driver is Driver Verifier compatible the next logical step is to check if the other third party driver is also Driver Verifier compatible. You set up verification, and the third party driver does not pass verification tests and BSODs machine. Well, the problem is located. The other third party driver is guilty, and has to be un-installed, right?
Now comes the politics. The faulty driver is a part of a big product from well known company which is on the market since 1982 and has quite a big piece of a cake of it. You can’t uninstall this product, as customer payed money for it 🙂 Customer would better uninstall your product as from the customer’s point of view, the problem started right away when they installed your product. At this point, a good question appears, why would that big company did not make it’s product as much stable as it could pass Driver Verifier test? Hopefully we will be able to get answers.
To name things. I came over Norton Antivirus recently, and discovered, that it does not pass Driver Verifier test under minimal stress test conditions. It could be easily demonstrated over a small video clip I took at Windows Vista SP1 French edition. I just installed Norton Antivirus, setup Driver Verifier to verify all it’s drivers and after I run a small application which is connecting in cycle into some dummy IP. It BSODs. The clip can be seen here: (make sure you click HD and run clip in high definition!)
The resulting memory dump can be analyzed in WinDbg, and it produces the following crash trace:
0: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * *******************************************************************************BAD_POOL_CALLER (c2) The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc. Arguments: Arg1: 00000007, Attempt to free pool which was already freed Arg2: 0000110b, (reserved) Arg3: 00000002, Memory contents of the pool block Arg4: 93f39014, Address of the block of pool being deallocated Debugging Details: ------------------ MODULE_NAME: ccHPx86 FAULTING_MODULE: 81811000 nt DEBUG_FLR_IMAGE_TIMESTAMP: 499b2a24 BUGCHECK_STR: 0xc2_7 DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT CURRENT_IRQL: 0 LAST_CONTROL_TRANSFER: from 818fe184 to 818deb0d STACK_TEXT: WARNING: Stack unwind information not available. Following frames may be wrong. 99555014 818fe184 000000c2 00000007 0000110b nt!KeBugCheckEx+0x1e 99555088 92cf584e 93f39014 00000000 995550d4 nt!ExFreePoolWithTag+0x17f 99555098 92cf59be 93f39018 8830a88c 92cf78cf ccHPx86+0x1784e 995550d4 92cf46bf 00000001 92d43244 92d46000 ccHPx86+0x179be 995550fc 92ce0089 00000000 99555124 923abdb8 ccHPx86+0x166bf 99555140 92ce0272 81308f48 81308f20 00000000 ccHPx86+0x2089 9955516c 81e927d9 81308f48 00000000 00000004 ccHPx86+0x2272 9955519c 81e95820 81308ec8 81308ec8 81308ec8 SYMEFA+0x427d9 995551f4 81e95476 81308ec8 92d8aecc 81856508 SYMEFA+0x45820 9955520c 92d6ed41 00000004 99555230 00000000 SYMEFA+0x45476 99555270 92d6f255 00000001 995553b8 00000000 BHDrvx86+0x15d41 99555294 92d63c76 99555350 a6a83f88 00000001 BHDrvx86+0x16255 995552a8 92d69b43 00000001 995553b8 995552e0 BHDrvx86+0xac76 995552b8 92d69f65 995553b8 00000001 85d841f8 BHDrvx86+0x10b43 995552e0 92d679db 99555564 89660788 995553a4 BHDrvx86+0x10f65 99555368 92d67ed8 99555540 92d67ed8 99555564 BHDrvx86+0xe9db 99555514 92d68fac 99555564 99555540 99555564 BHDrvx86+0xeed8 99555538 92d73ee2 8967daf8 923799cc 8967dac8 BHDrvx86+0xffac 99555554 92d740f9 99555564 8967dae8 92d8c970 BHDrvx86+0x1aee2 995555a4 92d743d1 995555d4 89f0aff0 92a37908 BHDrvx86+0x1b0f9 995555c8 92d74627 995555d4 995555f0 00000004 BHDrvx86+0x1b3d1 995555e8 92a1d70f 00000000 9955562c aac6af5c BHDrvx86+0x1b627 99555608 92a212a2 00000000 9955562c 87ffaf48 SYMTDI!isUserLoggedIn+0x127f 99555630 92a2276f aafb0fa8 99555654 8702cb10 SYMTDI!GetGUIDevice+0xc62 99555658 92a22861 922dc438 87ffaf48 9955568c SYMTDI!GetGUIDevice+0x212f 99555668 81af36be 922dc438 87ffaf48 a9fb6f10 SYMTDI!GetGUIDevice+0x2221 9955568c 8185592d 87ffafdc 9233e8a4 922dc438 nt!PoSetHiberRange+0xc952 995556a0 81a500d5 519a70e6 883582e4 92232428 nt!IofCallDriver+0x1b 99555770 81a3e521 92232440 00000000 88358240 nt!CcMapData+0x133d 99555800 81a4baa2 00000000 99555858 00000240 nt!SeUnlockSubjectContext+0x62d 99555864 81a511dc 99555ab0 00000000 ffffff00 nt!ObOpenObjectByName+0x13c 995558d8 81a55ffa 99555af0 02000000 99555ab0 nt!SeSetAccessStateGenericMapping+0x7a2 99555934 92ab3603 99555af0 02000000 99555ab0 nt!IoCreateFileEx+0x9e 99555b78 92ab12ea 882f5838 831d3998 00000016 afd+0x18603 99555bfc 92ab9040 87078530 a9fb6f00 99555c30 afd+0x162ea 99555c0c 81af36be 922f1448 a9fb6f00 87076718 afd+0x1e040 99555c30 8185592d a9fb6fdc a9fb6f00 922f1448 nt!PoSetHiberRange+0xc952 99555c44 81a576a1 87076718 a9fb6f00 a9fb6fdc nt!IofCallDriver+0x1b 99555c64 81a57e46 922f1448 87076718 01baf800 nt!FsRtlAreNamesEqual+0x2b9 99555d00 81a58f10 922f1448 a9fb6f00 00000000 nt!FsRtlAreNamesEqual+0xa5e 99555d34 8185bc7a 00000278 000000b8 00000000 nt!NtDeviceIoControlFile+0x2a 99555d64 77d15e74 badb0d00 01baf830 00000000 nt!ZwQueryLicenseValue+0xbc6 99555d68 badb0d00 01baf830 00000000 00000000 0x77d15e74 99555d6c 01baf830 00000000 00000000 00000000 0xbadb0d00 99555d70 00000000 00000000 00000000 00000000 0x1baf830 STACK_COMMAND: kb FOLLOWUP_IP: ccHPx86+1784e 92cf584e 5d pop ebp SYMBOL_STACK_INDEX: 2 SYMBOL_NAME: ccHPx86+1784e FOLLOWUP_NAME: MachineOwner IMAGE_NAME: ccHPx86.sys BUCKET_ID: WRONG_SYMBOLS Followup: MachineOwner ---------
The most interesting parts of stack trace are in bold. So, two drivers being verified are in guilty crash stack. SYMTDI.SYS and SYMEFA.SYS. However, the crash originates by ccHPx86.sys. Let’s try to get some info about this driver:
0: kd> lmv m ccHPx86 start end module name 92cde000 92d59000 ccHPx86 (no symbols) Loaded symbol image file: ccHPx86.sys Image path: \??\C:\Windows\system32\drivers\NAV\1005000.086\ccHPx86.sys Image name: ccHPx86.sys Timestamp: Tue Feb 17 22:20:36 2009 (499B2A24) CheckSum: 00083665 ImageSize: 0007B000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
The path containst NAV, which I believe stands for Norton Anti Virus. A little googling shows that this driver belongs to Norton Antivirus Package. Let’s check the second unknown driver named BHDrvx86.sys:
0: kd> lmv m BHDrvx86 start end module name 92d59000 92d9b000 BHDrvx86 (no symbols) Loaded symbol image file: BHDrvx86.sys Image path: \??\C:\Windows\system32\drivers\NAV\1005000.086\BHDrvx86.sys Image name: BHDrvx86.sys Timestamp: Wed Jan 28 00:23:38 2009 (497F977A) CheckSum: 00041D27 ImageSize: 00042000 Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
It is also a part of Norton Antivirus. Obviously, something wrong happens inside those drivers and this caused the verifier to react. Taking into consideration that Driver Verifier is a standard tool to test drivers it is really strange to see such results with popular Antivirus vendor. I hope this post will be a good addition to NAV testing process 🙂