All 32 bit versions of Microsoft Windows includes a 16 bit emulator subsystem, which allows one to run standard DOS and Win16 application like one could in Windows 3.1. 64 bit versions of Windows no longer includes the 16 bit emulator subsystem.
Many regards these subsystems as possible security holes, and prefere to disable them to avoid any surprises. On Windows XP / 2003 and newer the ntvdm.exe can be stopped from running with this Group Policy:
If running Windows NT/2000 (or want to ensure no one starts ntvdm.exe), then one can change the NTFS permissions to the file, so it no longer can be executed.
Computer Configuration\Administrative Templates\Windows Components\Application Compatibility
Prevent access to 16-bit applications
HKEY_LOCAL_MACHINE \Software \Policies \Microsoft \Windows \AppCompat]
VDMDisallowed = 0
If just want to disable the ability to run MSDOS and Win16 applications without disabling the subsystem, then one can modify the following registry key (To WOW2):
Note if disabling the DOS subsystem then any batch scripts with the .BAT file extension, should be renamed to use the .CMD file extension.
[HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Control \WOW]
CmdLine = "..."
WowCmdLine = "..."
More Info MS KB220159
More Info SecurityFocus.com (Exploit sample code)