The FIN scan sends a packet with only the FIN flag set, the Xmas Tree scan sets the FIN, URG and PUSH flags (see a good TCP/IP book for more details) and the Null scan sends a packet with no flags switched on.
These scan types will work against any system where the TCP/IP implementation follows RFC 793. Microsoft Windows does not follow the RFC, and will ignore these packets even on closed ports. This technicality allows you to detect an MS Windows system by running SYN along with one of these scans. If the SYN scan shows open ports, and the FIN/NUL/XMAS does not, chances are you're looking at a Windows box (though OS Fingerprinting is a much more reliable way of determining the OS running on a target!)
The sample below shows a SYN scan and a FIN scan, performed against a Linux system. The results are, predictably, the same, but the FIN scan is less likely to show up in a logging system.
[chaos]# nmap -sS 127.0.0.1 Starting Nmap 4.01 at 2006-07-06 17:23 BST Interesting ports on chaos (127.0.0.1): (The 1668 ports scanned but not shown below are in state: closed) PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 631/tcp open ipp 6000/tcp open X11 Nmap finished: 1 IP address (1 host up) scanned in 0.207 seconds [chaos]# nmap -sF 127.0.0.1 Starting Nmap 4.01 at 2006-07-06 17:23 BST Interesting ports on chaos (127.0.0.1): (The 1668 ports scanned but not shown below are in state: closed) PORT STATE SERVICE 21/tcp open|filtered ftp 22/tcp open|filtered ssh 631/tcp open|filtered ipp 6000/tcp open|filtered X11 Nmap finished: 1 IP address (1 host up) scanned in 1.284 seconds