tokenize - split string into words

tokenize: fn(s, delim: string): (int, list of string);

## returns (number of words, list of the words) on success; 

(0, nil) on failure.

Description

The returned tuple, (n, list), holds the number of words, n, and an ordered list, list, of those words. The first item in list (for example, hd list) is the leftmost word from s.

Words are delimited by the maximal sequences of any character from the delim string. The tokenize function skips delimiter characters at the beginning and end of s, so each element in the returned list has non-zero length.

If s is nil or contains no words, tokenize returns a count of zero and a nil list.

The delim argument may be nil or the empty string specifying no delimiter characters. The resulting word list will be nil (if s is nil, or the empty string) or a single-item list with a copy of s.

See Also