Network Programming and Distributed Scripting with newLISP

by ax0n

newLISP is a relative newcomer to the interpreted language arena in terms of popularity.  While it had its humble beginnings back in 1991 when Lutz Mueller started working on it, only in the last four years has development been consistently active.

newLISP is everything that old-school LISP languages are, with a lot of modern features.  First off, it's a scripting language that's extremely fast.  It has networking ability that's powerful enough to write TCP or UDP client or server applications.

Then, to top that off, it has a command called net-eval() which makes newLISP stand out from the crowd by giving it the unique ability to easily distribute tasks to other nodes over a network connection.

Binaries (under 200 kilobytes) are available for Windows, BSD, Linux, Mac OS X, Solaris and a host of other platforms.  It is released under the GPL.  Performance is also second to none.  newLISP has been topping the charts on script interpreter benchmarks in several categories thanks to it's small size and efficient C code.  It outruns PHP, Perl, and even Ruby.

newLISP also has some other tricks up its sleeve that make it an excellent system administration scripting language.  It has decent filesystem support, so it can see if files or directories exist, or determine if a file's permissions are acceptable for reading or writing.  And it also has very powerful text processing ability using Perl Compatible Regular Expressions (PCRE).

Finally, it's also worth mentioning that newLISP can easily import whole functions from dynamic libraries such as libmysqlclient (instant MySQL access from within newLISP), Tcl/Tk (for creating graphical applications in newLISP) and zlib (for compression and decompression) just to name a few.  This makes newLISP one of the most robust and flexible languages around.

As you can tell, newLISP is a formidable choice for hackers, geeks, network admins or security professionals wishing to create scripted programs to do network operations or distributed computing with minimal effort.

I am lucky to have been able to work directly with Lutz, the founder and creator of newLISP.  I got a few direct lessons from him and, from there, started tinkering with it on my own.

With that, the first thing I did was create a makeshift port scanner.

I learn easiest by example, so here is what I came up with:

# port.lsp - port scanner
(set 'params (main-args))
(if (< (length params) 5) 
    (println "USAGE: port.lsp host begin-port end-port")
(set 'host (nth 2 params))
(set 'bport (int (nth 3 params)))
(set 'eport (int (nth 4 params)))
(for (port bport eport)
   (set 'socket (net-connect host port))
   (if socket (println port " open"))

The first part simply assigns the command line arguments into a list called params, then makes sure that four parameters were given (program name, host, begin port and ending port).  If not, it displays a usage tip before exiting.

The second part assigns elements of the list to appropriate variables, then uses a for loop to iterate through the ports, displaying open port numbers that are open.  Note that on machines with packet filters that "drop" packets, this port scan will take a very long time.  Nmap is a much more robust port scanner, however this little script demonstrates the power of newLISP's network commands.

We'll run this as a test just for fun:

$ ./port.lsp 1 200
21 open
22 open
23 open
25 open
79 open
111 open

Now, let's look into distributed computing, shall we?

The core command behind newLISP's distributed computing power - called net-eval() - operates on a list of lists (similar to a three dimensional array).

The innermost list is a list of host, port, and a string representing the command(s) you wish to run on the remote node.

The outermost list can contain as many host-port-command lists as your heart desires, allowing you to run many distributed processes at once and get the results back all at the same time.

Then, outside those lists is a timeout in milliseconds.  If a result isn't returned in the timeout period, the operation returns nil (that is, false).

To clarify, net-eval() syntax is as follows:

(net-eval (list (list "host" port-number command-string)) timeout)

On each remote node, you must have a newLISP listener, which is simply started by running newlisp -c -d port-number from the command line.

On UNIX environments, you may put an ampersand (&) at the end to launch it in the background, or you may even wish to use set NOHUP and log off to leave it running in the background indefinitely.

In my example, I went to my Solaris box and launched it, listening on port 31337 as follows:

$ newlisp -c -d 31337 &

I also launched newLISP listeners on various other machines on my home network, including a few OpenBSD machines, and my wife's MUD/BBS server running Windows Server 2003 with the "Windows Services for UNIX" tools installed.

Now, care must be taken.  It is a bad idea to have a newLISP listener running on a public IP address, because commands like process() or exec() can launch shell processes on the newLISP node, which is just as good as giving away an unprotected shell account on your network.  I advise using newLISP listener nodes only behind a NAT or firewall, or on a segregated network.

Let's run a test script, shall we?

In LISP, Boolean and math operations are always performed by placing the operator first, followed by the symbols to apply it to.  In addition, the symbols are numbers, but they could easily be strings or lists with some operations.

Adding 1 + 2 in LISP is as simple as: (+ 1 2)

I will start by running a quick addition operation on one remote node with a 3000 ms (3 second) timeout:

# net-eval-test.lsp
(set 'evalstring "(+ 1 2)")
(println (net-eval (list (list "" 31337 evalstring)) 3000))

When we run it, we get the answer to this mind-boggling math problem:

$ ./net-eval-test.lsp

Now, to expand this even more, I have added three other nodes into the mix, which shows more clearly how the nested list syntax of net-eval() works, and I'll demonstrate remote command execution at the same time, using the exec() command.

Notice how the quotes around the command to be run is escaped with backslashes.  This is needed to keep from confusing the interpreter.  To put quotes inside a quoted string, you need to escape them.  This is almost universal to all programming languages.  On UNIX-like platforms, uname is used to get information about the operating system and architecture.

This command will list the OS that's running, the hostname, and the machine architecture: uname -s -n -m

# uname.lsp
(set 'evalstring "(exec \"uname -s -n -m\")")
(println (net-eval (list 
        (list "localhost" 31337 evalstring)
        (list "" 31337 evalstring)
        (list "" 31337 evalstring)
        (list "" 31337 evalstring)
        ) 3000))

The result is a newLISP list of strings, containing the results of running the command:

$ ./uname.lsp
(("SunOS sparky sun4u") ("OpenBSD compy386 i386") ("OpenBSD bouncer sparc") ("Windows mudbbs x86"))

The online documentation for newLISP is very extensive, and features a few rather advanced demonstration scripts, including a working web server written entirely in newLISP.

While learning a new programming language is never easy, newLISP is more than mature enough in both implementation and documentation to make it a pretty easy language to add to your list.

Links - newLISP Website, full of demonstration newLISP programs, documentation, binaries for many platforms, and newLISP source code. - (NewLISPer) is a journal, or blog, written by a guy who was just learning newLISP.  It's turned into a bunch of newLISP tutorials with some philosophy tossed in as well. - Norman's code snippets is a website full of newLISP programs and snippets for Linux (not tested on other platforms).  There is a lot of really interesting applications and widgets available to download.

Code: port.lsp

Code: net-eval-test.lsp

Code: uname.lsp

Return to $2600 Index