Smallvoid.com

A great advantage of Microsoft Windows is that old software usually continue to work even if upgrading to the next version of Microsoft Windows.

All 32 bit versions of Microsoft Windows includes a 16 bit emulator subsystem, which allows one to run standard DOS and Win16 application like one could in Windows 3.1. 64 bit versions of Windows no longer includes the 16 bit emulator subsystem.

Microsoft Windows also once included a OS/2 and a POSIX emulator, but they were removed with Windows XP/2003. More Info MS KB Q308259

Many regards these subsystems as possible security holes, and prefere to disable them to avoid any surprises. On Windows XP / 2003 and newer the ntvdm.exe can be stopped from running with this Group Policy:

Computer Configuration\Administrative Templates\Windows Components\Application Compatibility
Prevent access to 16-bit applications

HKEY_LOCAL_MACHINE \Software \Policies \Microsoft \Windows \AppCompat]
VDMDisallowed = 0

If running Windows NT/2000 (or want to ensure no one starts ntvdm.exe), then one can change the NTFS permissions to the file, so it no longer can be executed.

If just want to disable the ability to run MSDOS and Win16 applications without disabling the subsystem, then one can modify the following registry key (To WOW2):

[HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Control \WOW]
CmdLine = "..."
WowCmdLine = "..."

More Info MS KB Q220159

Note if disabling the DOS subsystem then any batch scripts with the .BAT file extension, should be renamed to use the .CMD file extension.

More Info SecurityFocus.com (Exploit sample code)