We recently developed a TWiki plugin to support execution of Perl scripts that are written in TWiki topics. The scripts are executed on the server, and of course that means we have to do everything possible to ensure those scripts don't open security holes.For years Perl has had the Safe module, a clever package that provides a tightly constrained execution environment for Perl eval statements. Perl compiles all its code to a rich set of high level opcodes, which are then run on a virtual machine. By limiting the set of opcodes that are allowed to be run in the container, the Safe module can be used to create a very secure execution environment.For example, most people would consider the perl 'backtick' operator to be very dangerous, as it allows the caller arbitrary access to the shell. Backtick has a corresponding Perl opcode – called backtick – and to disable it, all we have to do is to remove it from the set of legal opcodes. The Perl developers have even gone so far as to classify the operators according to the usual safety concerns that a caller may have, making it relatively easy to decide which to allow, and which to exclude.Of course there's more to safety than that. We also have to be sure that the code being executed only has access to the namespaces we want it to have access to. The default condition for scripts run in a safe container is that they can only access the namespace of the container. We have to explicitly grant the container access to other namespaces when we create it.Of course there are potential risks with allowing any sort of script execution on your server, but in the case of a web server behind a corporate firewall, those risks are relatively small, and the 'Safe' module helps to make sure that such scripts are well controlled.The new TWiki plugin, called the PerlPlugin, is released under the GPL and is available to all WikiRing consultants for deployment on client sites.

With the recent announcement about the new Google Sites application, a number of former Jot Spot customers have decided to migrate to TWiki. WikiRing partner C-Dot Consultants has been engaged to help, and this post describes our experiences.

Jot Spot stores topic data in an XML database. Within that database, actual topic data is stored as "decorated HTML"; the basic topic content is HTML, within which Jot applications are embedded using Jot Spot's proprietary script, which uses XML tags.

Because of some fairly fundamental architectural differences between Jot Spot and TWiki it's not simple to automate the conversion of Jot Spot applications to TWiki. Fortunately our clients have not invested heavily in developing Jot applications, but have instead chosen to use the applications that Jot Spot provide by default. So the focus of our work has been to:

1. Convert existing Jot Spot topics to TWiki, with minimal formatting loss
2. Map a subset of Jot applications (most notably the Bug Reporter) to TWiki

Fortunately we were able to secure an XML dump of the Jot Spot database. This has alllowed us to perform the conversion without relying on the patchy availability of the Jot Spot site. Conversion of Jot topics has been achieved by leveraging a couple of existing technologies:

• The SAX XML parser from CPAN
• The open-source HTML to TML convertor we wrote for WYSIWYG editor integration into TWiki

SAX allows us to parse the Jot XML, pick out the form fields, and identify the subset of the topics suitable for passing on to the HTML to TML convertor. The HTML to TML convertor is already a proven technology, used every day with the Tiny MCE integration in TWiki, so it is robust and minimises formatting loss.

On the receiving side, we have customised the publicly available TWiki:Plugins.BugsContrib to support the data fields from the reporter in Jot Spot. We have had to develop a number of new reporting screens, something which has been made much simpler by the use of the type="query" search we contributed to TWiki 4.2.

It's cheering to note how similar the structure of a Jot Spot topic is to a TWiki topic. I guess you could call it convergent evolution!

See the Wikipedia article for the full gory details of what REST is. TWiki developers can think of a REST as a way of calling a single function in a a plugin, in a place where TWiki::Func is available. REST handlers are designed primarily to be called from XmlHttpRequest, but can also be useful in IFRAMEs.

A REST handler is invoked via a URL to the rest bin script. For example,

%SCRIPTURL%/rest/WysiwygPlugin/tml2html?topic=%WEB%.%TOPIC%&text=Some%20test%20text

would invoke the TML to HTML translator on TWiki.org (if it was available there).

REST handlers can be added piecemeal to address specific requirements. However there are other structured approaches currently in development that the interested reader might be well advised to investigate:

• Michael Daum developed the XmlRpcContrib, which supports XML-RPC calls
• Olivre Krüger has developed a REST interface to the TWiki::Func module. This interface uses JSON and XML to communicate the results of calling Func methods back to the Javascript (or so Oliver tells me; I haven't seen the code yet)
• Sven Dowideit is developing a REST plugin that uses the content-access syntax develop for search queries and IF statements in 4.2 to provide access to data in the TWiki database. This is most promising work, as it is a strong move in the direction of the TopicObjectModel.

So much for the future; what about the present? How do I go about writing a REST handler? Well, I can only describe how I do it; I'm sure there will be other, probably better, approaches.

As an example, let's look in detail at the REST function that implements the attachments list in the WysiwygPlugin. This Javascript function will, when given the name of a topic, return a list of the attachments on that topic. It uses:

• XmlHttpRequest from within Javascript to make the request
• TWiki::Func to fetch the data that satisfies the request
• HTTP status codes to transfer state and erros back to the client
• JSON to transfer the data back to the client

First let's look at the server side of the solution. A REST handler is registered in initPlugin thus:

sub initPlugin {
    TWiki::Func::registerRESTHandler('attachments', \&_restAttachments);
}

Now for the handler body:

sub _restAttachments {
    my ($session) = @_;
    my BlogEntry23 = TWiki::Func::getCgiQuery()->param('topic');
    my WRBlog;
    (WRBlog, BlogEntry23) = TWiki::Func::normalizeWebTopicName(undef, BlogEntry23);

This first section of the handler simply examines the compulsory topic parameter to the REST call to determine the topic we are interested in. Note that normalizeWebTopicName will also untaint the web and topic names. Now we have the topic we can check the access permissions:

    unless (TWiki::Func::checkAccessPermission(
        'VIEW', TWiki::Func::getWikiName(),
        undef, BlogEntry23, WRBlog)) {
        my $error = "Access denied";
        print CGI::header(-status => 401);
        print $error;
        print STDERR $error;
        return undef;
    }

This demonstrates how to handle errors. If access is denied, we print a CGI header to STDOUT with a 401 (access denied) HTTP status code. We also print the message, and duplicate that print to STDERR so the message also goes to the Apache log. Returning undef from a REST function just causes the rest script to terminate without generating any more output, and is the recommended way to terminate on an error.

    my ($meta, $text) = TWiki::Func::readTopic(WRBlog, BlogEntry23);
    # Create a JSON list of attachment data, sorted by name
    my @atts;
    foreach my $att (sort { $a->{name} cmp $b->{name} }
                               $meta->find('FILEATTACHMENT')) {
        push(@atts, '{'.join(',',
                         map {
                             "\"$_\":\"$att->{$_}\""
                         } keys %$att).'}');

    }
    return '['.join(',',@atts).']';
}

Having passed the access control check, we now generate a list of attachment data in JSON format. We could have used the CPAN:JSON module to do this, but the data is so trivial in this case that the overhead of the extra module would have been excessive.

When we return non-null data from a REST handler, the rest script automatically generates a 200 HTTP (OK) status response, indicating correct termination of the call.

Now, let's look at the Javascript. First we need to build the REST url:

function attachments() {
    // Work out the rest URL from the location
    var scripturl = getTWikiVar("SCRIPTURL");
    var suffix = getTWikiVar("SCRIPTSUFFIX");
    if (suffix == null) suffix = '';
    var path = getTWikiVar("WEB") + '.' + getTWikiVar("TOPIC");

In this code we have used a function called getTWikiVar to obtain the values of some standard TWiki variables. How this function is implemented is outside the scope of this article, but a typical technique is to pass such variable values in META tags, as described in a previous post.

We now have sufficient information to build the URL for the REST script:

    var url = scripturl + "/rest" + suffix + "/WysiwygPlugin/attachments";

See the documentation accompanying Plugins.EmptyPlugin for more information on constructing REST urls. Now we perform a standard XmlHttpRequest

    if (tinyMCE.isIE) {
        // branch for IE/Windows ActiveX version
        request = new ActiveXObject("Microsoft.XMLHTTP");
    } else {
        // branch for native XMLHttpRequest object
        request = new XMLHttpRequest();
    }
    request.open("POST", url, true);
    request.setRequestHeader(
        "Content-type", "application/x-www-form-urlencoded");

    // nocache helps us defeat client-side cacheing of the result
    var params = "nocache=" + encodeURIComponent((new Date()).getTime())
        + "&topic=" + encodeURIComponent(path);
    
    request.setRequestHeader("Content-length", params.length);
    request.setRequestHeader("Connection", "close");
    request.onreadystatechange = function() {
        attachmentListCallback(request);
    };
    request.send(params);

This request will be sent to the server, and when a response is ready the Javascript function attachmentListCallback will be called.

function attachmentListCallback(request) {
    if (request.readyState == 4) { // only if "OK"
        if (request.status == 200) {
            var atts = request.responseText;
            if (atts != null) {
                atts = eval(atts);
                // atts is now an array of attachment objects, just like the perl
            }
        } else {
            alert("There was a problem retrieving the attachments list: "
                  + request.statusText);
        }
    }
}

And we are done!

Transferring the values of TWiki variables into Javascript. It's no problem if you write your Javascript inline, but that will soon become impossible if security settings are enabled on the server. You could always write a REST handler to drag the values over from the server using a XmlHttpRequest – but that's like using a sledgehammer to swat flies.

What's really needed is a simple methodology for making the values of selected TWiki variables available to Javascript.

The problem is that while TWiki variables are expanded in templates, unless your Javascript is inline there is no way of getting that value into the Javascript. For example, let's take a simple example; we have an input, and we want to invoke a validator on it:

<script src="%PUBURL%/%SYSTEMWEB%/MyPlugin/validator.js"></script>
<input type="text" value="Urgh" onchange="javascript:validateInput()">

The validator is in a Javascript file called validator.js

function validateInput() {
   var url = '%SCRIPTURL{rest}%/MyPlugin/Validate';
   ... perform an XmlHttpRequest to get the value validated...
}

The problem is that there is no way to expand %SCRIPTURL{rest}% inside validator.js. Somehow we have to pass this value to the Javascript.

The easiest way to do this is in the DOM. And HTML generously gives us the META tag that we can use as we see fit. So in our plugin, we can write:

TWiki::Func::addToHEAD('MYPLUGIN', '<META name="MyPluginData" value="%SCRIPTURL{rest}%">');

Now we change validator.js as follows:

function validateInput() {
   var meta = document.getElementsByTagName('META');
   for (var i = 0; i < meta.length; i++)
      if (meta[i].name == 'MyPluginData') {
         scripturl = meta[i].value;
         break;
      }
   var url = scripturl + '/MyPlugin/Validate';
   ... perform an XmlHttpRequest to get the value validated...
}

Oviously if you are passing a number of values, it makes sense to do the search of the META tags once and build a hash of the values.

If you can'y write to the header for some reason, you can also use other (body) tags to perform a similar trick, and stop them being displayed using style='display:none'