When a TCP connection is made between two systems, a process known as a "three way handshake" occurs. This involves the exchange of three packets, and synchronises the systems with each other (necessary for the error correction built into TCP. Refer to a good TCP/IP book for more details.
The system initiating the connection sends a packet to the system it wants to connect to. TCP packets have a header section with a flags field. Flags tell the receiving end something about the type of packet, and thus what the correct response is.
Here, I will talk about only four of the possible flags. These are SYN (Synchronise), ACK (Acknowledge), FIN (Finished) and RST (Reset). SYN packets include a TCP sequence number, which lets the remote system know what sequence numbers to expect in subsequent communication. ACK acknowledges receipt of a packet or set of packets, FIN is sent when a communication is finished, requesting that the connection be closed, and RST is sent when the connection is to be reset (closed immediately).
To initiate a TCP connection, the initiating system sends a SYN packet to the destination, which will respond with a SYN of its own, and an ACK, acknowledging the receipt of the first packet (these are combined into a single SYN/ACK packet). The first system then sends an ACK packet to acknowledge receipt of the SYN/ACK, and data transfer can then begin.
SYN or Stealth scanning makes use of this procedure by sending a SYN packet and looking at the response. If SYN/ACK is sent back, the port is open and the remote end is trying to open a TCP connection. The scanner then sends an RST to tear down the connection before it can be established fully; often preventing the connection attempt appearing in application logs. If the port is closed, an RST will be sent. If it is filtered, the SYN packet will have been dropped and no response will be sent. In this way, Nmap can detect three port states - open, closed and filtered. Filtered ports may require further probing since they could be subject to firewall rules which render them open to some IPs or conditions, and closed to others.
Modern firewalls and Intrusion Detection Systems can detect SYN scans, but in combination with other features of Nmap, it is possible to create a virtually undetectable SYN scan by altering timing and other options (explained later).