They see me scannin' (part 2)

In the first post in this series, we looked at 10 different ways that IDS and other security systems can identify an Nmap scan and ruin your day. Don't despair! With a little ingenuity and knowhow, we can eliminate most of these trouble spots.

There are 3 cardinal rules when trying to avoid detection that apply no matter what tools you are using:

1. Less is more. The less traffic you generate, the less the IDS has to match its rules against. You can also avoid packet-counting thresholds this way.
2. Blend in. If your traffic looks like legitimate traffic, it will be harder to catch without drowning in false-positives.
3. Do your research. Reading this blog post is insufficient. You have to know what kind of setup your target is using and tune your approach to what works best.

Before we even get to the trouble spots from last time, we can apply these rules to our scan plan. You do have a plan of what you want to accomplish, right? With Less is more in mind, your first step should be to gather as much information as you can from non-attributable sources; only use Nmap for the things only Nmap can tell you. Some examples:

• Use public DNS records from sites like Robtex for host discovery.
• Use hosted scan data and scanners like Shodan, scans.io, and HackerTarget.com.
• Use search engines and Google dorking to look for interesting stuff.
• Use nmap -sL to do reverse-DNS lookups on big subnets looking for interesting names.

Once you've done these things, you can try to eliminate unnecessary phases from your Nmap scan. The simplest one is host discovery: using pre-assembled lists of addresses or hostnames via -iL, you can avoid sweeping large areas of network for active hosts. You probably should verify once that they are up, and afterwards you can use -Pn to skip the host discovery phase. If you save all your output with -oA then you can split your scan into different runs of Nmap without duplicating effort between them. Then you can let some parts (like a broad host discovery scan) run very slowly while you move on with other phases and examining interesting results.

The other important thing is to do your research regarding your targets and their defenses. By searching job postings, using passive detection, and mapping rules and thresholds, you should get a good idea of what kind of defense systems are being used, how they are configured, and what the typical response actions are. Without this information, you are working blind and just guessing what behaviors to change. Take full advantage of weaknesses: if their IDS is vulnerable to fragmentation bypass, use fragmentation and don't worry about most of these restrictions!

Now let's examine the trouble spots we identified in Nmap's behavior.

First, there's the general problem of Nmap's raw packets being unique and fingerprintable. A good way to avoid this for TCP scans is to use the TCP Connect scan (-sT) instead of the SYN scan (-sS). This uses your OS's native socket calls to make connections, which means that the packets will blend in with other connections from valid clients. There are downsides in performance and level of detail (TTL info that can be used to map firewall rules is not available, for instance) but these are usually minimal in an IDS-evasion scenario.

"But wait," you say, "Isn't SYN scan the 'stealth' option?" Yes it is—in 1997. If your defense technology consists of watching daemon logs for aborted connection attempts, then TCP SYN scan is completely invisible. But as we discussed last time, SYN scan sticks out like a sore thumb to a stateful IDS. The combination of unique TCP options and unusual packet sequence make it a poor choice for IDS evasion.

Side note: A patch to allow a user to override the TCP Window size in SYN scan was just posted to the Nmap development list. I haven't tested it at all, but it would help with one aspect of this detection.

Next, there's the specific problem of Nmap's ICMP Echo Request host discovery option (-PE, used by default with root privilege) sending packets without a body. There are a few different options that can be used here: --data-length has been around since before Nmap 3.00 and appends a specified number of random bytes to the end of all raw packets. This avoids the signature directly, but still could look suspicious because most ping tools use a well-known static string. In the latest versions (beginning with 6.49BETA1), you can also use --data or --data-string to specify particular payloads to use. You'll have to separate the ICMP host discovery Nmap commands from the other uses of Nmap, though, since the data gets appended to every packet, which is unexpected and weird.

I'm mostly going to skip over OS detection, because there's no way to make it more stealthy. Because it depends on sending a bunch of strange-looking packets in a specific order, it will always be easy to detect. Avoid -O and -A if you are at all concerned with avoiding detection.

The cautions regarding excessive fragmentation are best handled by doing your research. Not all IDS will complain about this, and many of them will actually be bypassed by it (which is why the -f option was created in the first place). If you probe your target's defenses before trying large-scale enumeration, you'll know whether or not -f will help or harm you.

The performance impact of TCP Connect scan should not cause you problems because you will be artificially slowing down your progress to avoid timing thresholds. A good way to do this is to start with the timing templates (-T[0-5]) and then apply some well-researched tweaks. -T3 is the default, so you obviously want to go slower than that, but -T0 slows you down to 4 packets per minute, which is ridiculous. Here are some options that are helpful in combination with -T2 and lower:

• --max-retries - At very reduced speeds, you can't often afford to spend extra packets confirming that a port is filtered. Set this to 1 or 2 and just accept that you might miss something due to a dropped packet here or there.
• --max-rate - The primary effect of -T[0-2] is to slow down the packet rate. If you don't remember or don't like the rate one of the templates uses, override it with this.

For UDP payloads that are getting you in trouble, the easiest way to avoid the pain is to not scan for UDP ports. That advice isn't very helpful, though, so let's look at your options for blending in and reducing the noise. The first and simplest option is to use --data-length 0 to disable the payload feature. Actually, any of the --data* options will override the UDP payload feature, but --data-length 0 is a special case that has no other effect. As you might expect from a fast-simple option, this one loses you the most functionality; no payloads means slower UDP scans and more retries.

Another, more difficult way to hide your UDP scans is to actually edit the nmap-payloads file to change what payloads are used. This can be beneficial in the long run, but requires that you do your research. Since these are actually sent to service daemons on your target, proceed with caution. You could end up crashing something.

As always, the ultimate solution to IDS detection of -sV is not to use the option. That said, there are other ways to reduce the noise a bit. Using the --version-intensity option, you can reduce the number of different probes that Nmap will try before giving up on a service. This means losing some version information, but you can usually get an idea of the type of service running on a port if it is a common port for that service. Setting this option to 0 will send only the Null probe (connect and wait for banner) and any probes that have been specifically listed as pertaining to the scanned port in nmap-service-probes. As always, do your research and understand what kinds of probes will be sent to what ports.

In addition to the probes in the nmap-service-probes file, -sV implies "--script version" or "run all the NSE scripts in the 'version' category." In versions of Nmap prior to 6.49BETA1, you could only prevent these scripts from running by removing them from the script.db catalog or by building Nmap without NSE support (./configure --without-liblua). Now, however, the --version-intensity option passes through to the version scripts, which all check to be sure that it is not less than 7 before running.

If you are wild enough to try NSE scripts against an IDS-protected target, you should know how to read Lua, since the script sources are the final authority on what data is sent. But if you're just looking to get a little better at blending in, these tips should help:

• Use --script-args-file to pass script arguments to Nmap from a file. This will keep your command line clean and make it harder to accidentally miss one of the options you choose
• Obviously avoid dos, intrusive, and exploit category scripts.
• Use scripts by name instead of by category, so that you know exactly what will be run.
• Thoroughly read the documentation for each script you intend to use.
• Set http.useragent to something believable that blends in.

I hope this was interesting and useful. Stay safe and happy scanning!

They see me scannin'; they hatin'

(Part 1 of a 2-part series on IDS evasion with Nmap. Part 2 is here.)

One hour into your pentest and you're already getting calls from the Blue Team:

"We see your Nmap scans. Do you want to just give up now, or..."

Impossible, I used "SYN Stealth scan" and scanned really slowly!

It's not impossible: Nmap out-of-the-box is really not that hard to spot if you know what you are looking for. Here are the most-common ways that Nmap scans get detected by IDS.

Quirks of Nmap's raw packets. Nmap uses raw sockets (or raw Ethernet access on Windows) to craft and send customized packets for rapid scanning. Because these packets are unique, they can be detected easily. Some specific examples:

• ICMP Echo Request with no payload. Snort IDS recognizes this with SID 469.
• TCP options in port scan. This is how p0f recognizes Nmap's SYN scan:  TCP window size a multiple of 1024, and only the MSS option supported with a value of 1460.
• TCP options in OS detection. These options were chosen specifically to test edge cases of target TCP/IP stacks, and must be consistent in order to produce consistent results. That means it's a stationary target for IDS signature writers. Snort and p0f both catch this easily.

Invalid or unexpected TCP packet sequence. Nmap's more unusual scan options like the FIN, NULL, XMAS, Window, and Maimon scans send packets which are invalid, since they are not part of any existing TCP session. That means they can be detected immediately by an IDS. It's also very likely that the packets will simply be blocked by a stateful firewall rule.

Even the standard SYN scan can readily be detected by this method as soon as it scans an open port, since the SYN, SYN/ACK, RST behavior is highly unusual.

Excessive fragmentation. Some of the previous detection points can be avoided by fragmenting the scan packets into tiny bits that cannot be completely inspected. The -f and --mtu options enable this behavior. But this can have unintended consequences: routers may just drop your traffic; IDS engines can trigger alerts based on the unusually small fragments; and IDS engines can use preprocessors like Snort's frag3 to reassemble the fragments before applying rules.

Timing thresholds. These generic rules will catch pretty much every port scanner ever, as long as they try to operate at any reasonable speed. The most valuable thresholds count the number of failed connections (TCP RST or ICMP error responses) in a time window and trigger when there are too many. Snort's sfPortscan preprocessor is a good study in this kind of heuristic detection.

UDP payloads. UDP port scanning is tricky, since there's no guarantee that a listening service will respond to an empty packet; in fact, most will not. For this reason, Nmap sends special data payloads to some well-known ports to try to get a real response. These payloads are static, and could be used to identify Nmap. Some, like the SNMP version 1 query with the "public" community string, trigger IDS alerts that are intended to catch generic bad behavior, which have no idea they've caught Nmap.

Service probes. Nmap's service probes are similar to the UDP payloads, but there are more of them and they get sent to more ports, both TCP and UDP. Some of them are highly Nmap-specific, including an SSL probe with a static (old!) date and a SIP probe that identifies itself as Nmap.

NSE behaviors. Nmap Scripting Engine (NSE) scripts are great gatherers of information, but they don't do a great job of hiding what they are doing. The HTTP scripts all use a User-Agent header that identifies as "Nmap Scripting Engine." Not to mention that any of the brute-force or exploit scripts will naturally be detected as those specific attacks.

Next: some Nmap options to reduce the impact of these detection techniques.

XML output for Nmap's NSE scripts

The Nmap Scripting Engine (NSE) is a great feature of Nmap that turns it into a data-collecting, banner-grabbing, vulnerability-scanning, exploit-finding platform. It gathers information like SSL certs, enumerates users, brute-forces authentication, exploits back-doors, spiders websites, and more. But though Nmap's port scan, version detection, and OS fingerprinting output all have their own elements in Nmap's XML output, NSE script output has been stored in a blob of text, formatted in any arbitrary fashion, and shoved into the output attribute of the <script> element.

For the last year, I've been working on various prototypes of structured output for NSE scripts. The long and sordid history is mostly traced out on this SecWiki page, but after many rewrites, forks, and the closest thing I've seen to a flame war within the Nmap family, the functionality was finally merged into the trunk in r29570!

The basic idea is that NSE scripts can now return a data structure in addition to or instead of the usual string output, and that data structure will be recursively converted into XML. What follows is my attempt at pulling together into one place the way the system works and how it is supposed to be used.

What it does

The NSE engine converts Lua data structures into XML using a recursive algorithm. Lua really only has one data structure, the table, which can be used to represent arrays/lists and associative arrays/hashes/hashmaps. The only difference is whether the indices are integers (array) or any other value (hash). In keeping with this simplicity, the XML output consists of only 2 new elements: <table> and <elem>, each of which can have a key attribute. The string version of the output is still kept in the output attribute of the <script> element. A good explanation of this is given in the NSE API section of the Nmap documentation.

How to use it

There are basically 4 ways to return output under the new system:

1. Plain string output (ye Olde Way)

The new system is completely backwards compatible, so your old scripts will work just fine. The string output will continue to be put into the output attribute of the script tag, but no new XML will be generated.

 2. Automatically-stringified output

As your script executes, build a table of output elements, then return that. NSE will automatically create a string output with indentation, "key: value" pairs, and newline-separated lists. This is the easiest way to get XML output, but it can take up some extra screen real estate, since each element of output is displayed on its own line in the Normal output.

3. Structured AND string output

Your script can return 2 values: a table to be converted into XML, and a string to use for Normal output. It's best if these contain the same information, but the XML must never contain less information than the string output. This is a handy method for wordy output containing explanations of what's been discovered; this information isn't needed in machine-readable XML.

4. Self-stringifying output

Your script can return a table that contains its own instructions for turning it into a string. This is done by overriding the __tostring metamethod for the table or any of the nested tables within it. If this sounds confusing, don't worry. I'll be adding a formatting library that hides all this Lua meta-magic behind easy-to-use function calls. The underlying functionality will be the same, though. This method is useful for scripts that want to take advantage of the outline-flavored stringification of option (2) for much of their output, but have special formatting needs for subsections, like a file listing (tabular output) or a horizontal, comma-separated list. This also helps ensure that the XML and string outputs both contain the same information.

Tips for script authors

Changing the way things work can be confusing. Here are some tips to help you write scripts that work well with XML script output:

• Use stdnse.output_table() to create a table that remembers the order you assigned keys. Regular Lua tables order their keys in an unpredictable way.
• Break down your data into the smallest reasonable chunks. If your script returns a username/domain combination, return it as {username="joe", domain="EXAMPLECOM"} instead if simply {username="EXAMPLECOM\joe"}.
• Make sure your script returns nil when you don't want output. Returning false used to work, but now it will show up as "false".
• Use the automatic stringification (option 2 above) for testing. Choose one of the other options once you know what output you want, and only if the default doesn't look right.