They see me scannin'; they hatin'

(Part 1 of a 2-part series on IDS evasion with Nmap. Part 2 is here.)

One hour into your pentest and you're already getting calls from the Blue Team:

"We see your Nmap scans. Do you want to just give up now, or..."

Impossible, I used "SYN Stealth scan" and scanned really slowly!

It's not impossible: Nmap out-of-the-box is really not that hard to spot if you know what you are looking for. Here are the most-common ways that Nmap scans get detected by IDS.

Quirks of Nmap's raw packets. Nmap uses raw sockets (or raw Ethernet access on Windows) to craft and send customized packets for rapid scanning. Because these packets are unique, they can be detected easily. Some specific examples:

• ICMP Echo Request with no payload. Snort IDS recognizes this with SID 469.
• TCP options in port scan. This is how p0f recognizes Nmap's SYN scan:  TCP window size a multiple of 1024, and only the MSS option supported with a value of 1460.
• TCP options in OS detection. These options were chosen specifically to test edge cases of target TCP/IP stacks, and must be consistent in order to produce consistent results. That means it's a stationary target for IDS signature writers. Snort and p0f both catch this easily.

Invalid or unexpected TCP packet sequence. Nmap's more unusual scan options like the FIN, NULL, XMAS, Window, and Maimon scans send packets which are invalid, since they are not part of any existing TCP session. That means they can be detected immediately by an IDS. It's also very likely that the packets will simply be blocked by a stateful firewall rule.

Even the standard SYN scan can readily be detected by this method as soon as it scans an open port, since the SYN, SYN/ACK, RST behavior is highly unusual.

Excessive fragmentation. Some of the previous detection points can be avoided by fragmenting the scan packets into tiny bits that cannot be completely inspected. The -f and --mtu options enable this behavior. But this can have unintended consequences: routers may just drop your traffic; IDS engines can trigger alerts based on the unusually small fragments; and IDS engines can use preprocessors like Snort's frag3 to reassemble the fragments before applying rules.

Timing thresholds. These generic rules will catch pretty much every port scanner ever, as long as they try to operate at any reasonable speed. The most valuable thresholds count the number of failed connections (TCP RST or ICMP error responses) in a time window and trigger when there are too many. Snort's sfPortscan preprocessor is a good study in this kind of heuristic detection.

UDP payloads. UDP port scanning is tricky, since there's no guarantee that a listening service will respond to an empty packet; in fact, most will not. For this reason, Nmap sends special data payloads to some well-known ports to try to get a real response. These payloads are static, and could be used to identify Nmap. Some, like the SNMP version 1 query with the "public" community string, trigger IDS alerts that are intended to catch generic bad behavior, which have no idea they've caught Nmap.

Service probes. Nmap's service probes are similar to the UDP payloads, but there are more of them and they get sent to more ports, both TCP and UDP. Some of them are highly Nmap-specific, including an SSL probe with a static (old!) date and a SIP probe that identifies itself as Nmap.

NSE behaviors. Nmap Scripting Engine (NSE) scripts are great gatherers of information, but they don't do a great job of hiding what they are doing. The HTTP scripts all use a User-Agent header that identifies as "Nmap Scripting Engine." Not to mention that any of the brute-force or exploit scripts will naturally be detected as those specific attacks.

Next: some Nmap options to reduce the impact of these detection techniques.