XML output for Nmap's NSE scripts

The Nmap Scripting Engine (NSE) is a great feature of Nmap that turns it into a data-collecting, banner-grabbing, vulnerability-scanning, exploit-finding platform. It gathers information like SSL certs, enumerates users, brute-forces authentication, exploits back-doors, spiders websites, and more. But though Nmap's port scan, version detection, and OS fingerprinting output all have their own elements in Nmap's XML output, NSE script output has been stored in a blob of text, formatted in any arbitrary fashion, and shoved into the output attribute of the <script> element.

For the last year, I've been working on various prototypes of structured output for NSE scripts. The long and sordid history is mostly traced out on this SecWiki page, but after many rewrites, forks, and the closest thing I've seen to a flame war within the Nmap family, the functionality was finally merged into the trunk in r29570!

The basic idea is that NSE scripts can now return a data structure in addition to or instead of the usual string output, and that data structure will be recursively converted into XML. What follows is my attempt at pulling together into one place the way the system works and how it is supposed to be used.

What it does

The NSE engine converts Lua data structures into XML using a recursive algorithm. Lua really only has one data structure, the table, which can be used to represent arrays/lists and associative arrays/hashes/hashmaps. The only difference is whether the indices are integers (array) or any other value (hash). In keeping with this simplicity, the XML output consists of only 2 new elements: <table> and <elem>, each of which can have a key attribute. The string version of the output is still kept in the output attribute of the <script> element. A good explanation of this is given in the NSE API section of the Nmap documentation.

How to use it

There are basically 4 ways to return output under the new system:

1. Plain string output (ye Olde Way)

The new system is completely backwards compatible, so your old scripts will work just fine. The string output will continue to be put into the output attribute of the script tag, but no new XML will be generated.

 2. Automatically-stringified output

As your script executes, build a table of output elements, then return that. NSE will automatically create a string output with indentation, "key: value" pairs, and newline-separated lists. This is the easiest way to get XML output, but it can take up some extra screen real estate, since each element of output is displayed on its own line in the Normal output.

3. Structured AND string output

Your script can return 2 values: a table to be converted into XML, and a string to use for Normal output. It's best if these contain the same information, but the XML must never contain less information than the string output. This is a handy method for wordy output containing explanations of what's been discovered; this information isn't needed in machine-readable XML.

4. Self-stringifying output

Your script can return a table that contains its own instructions for turning it into a string. This is done by overriding the __tostring metamethod for the table or any of the nested tables within it. If this sounds confusing, don't worry. I'll be adding a formatting library that hides all this Lua meta-magic behind easy-to-use function calls. The underlying functionality will be the same, though. This method is useful for scripts that want to take advantage of the outline-flavored stringification of option (2) for much of their output, but have special formatting needs for subsections, like a file listing (tabular output) or a horizontal, comma-separated list. This also helps ensure that the XML and string outputs both contain the same information.

Tips for script authors

Changing the way things work can be confusing. Here are some tips to help you write scripts that work well with XML script output:

• Use stdnse.output_table() to create a table that remembers the order you assigned keys. Regular Lua tables order their keys in an unpredictable way.
• Break down your data into the smallest reasonable chunks. If your script returns a username/domain combination, return it as {username="joe", domain="EXAMPLECOM"} instead if simply {username="EXAMPLECOM\joe"}.
• Make sure your script returns nil when you don't want output. Returning false used to work, but now it will show up as "false".
• Use the automatic stringification (option 2 above) for testing. Choose one of the other options once you know what output you want, and only if the default doesn't look right.