Yesterday, 2008-10-27: 21:00 GMT, just a minute before the regular TWiki release meeting, the company TWIKI.NET announced unilaterally that the best for the TWiki.org project would be for them to take over governance. With it comes a complete lock down of the community site. From that minute on, all long-time contributors have lost access to their code. Counter-reaction: the community has left the building, leaving TWIKI.NET without a contributing community. Question: is it a sensible move for a venture capital firm that depends on a healthy Open Source community to lock it out?

Access to the site is only granted if contributors agree to a set of newly installed terms and conditions dictated by TWIKI.NET, a company founded by Peter Thoeny 12 months ago. His power to do so grows out of two sources: (a) he is the sole owner of the trademark on TWiki and (b) he is sponsoring the server hardware and thus had root access.

And now he has triggered the trademark gun and fired the TWiki community. He even repeatedly threatened people on the #twiki IRC channel that "[he has] been advised by one of [his] investors, Wilson Sonsini Goodrich and Rosati, that [they] need to protect [their] trademark". Clearly, their VC people have no picture of the situation other than their own return of investment. Sure, protecting a registered trademark is what it is all about. But threatening the community that has been working on TWiki on a volunteer basis for the recent 10 years that way is a bit strong. Too strong for the TWiki community.

If there was ever any hope to re-establish a relationship of trust and faith to create a win/win situation by combining community & commerce, this is totally gone now. Thoeny installed himself as BDFL (Benevolvent Dictator for Life) again, despite being rejected by a community vote during the TWiki Summit in Berlin last month.

During the TWiki Summit in Berlin 4+5 September 2008, it became clear that Thoeny has sold part of his trademark rights to his venture capital funded company. Part of that deal was that while he remains ownership on the trademark itself, TWIKI.NET gained the sole right to exploit the brand on a commercial basis. This created a completely new situation for the Open Source project and all of its already existing commercial eco-system. As a consequence, TWIKI.NET was asked to grant a perpetual license to the community to secure the legal situation for contributors and commercial stakeholders, a license that would only have formalized the way TWiki has been running for more than 10 years with Thoeny promising to "take care of the brand".

As faith in him as a leader diminished over the years, and the foreboding of a trademark problem increased, the community asked Thoeny to write down the rights he has granted orally before. Which he didn't. Instead he pulled the trademark trigger in a move he calls "relaunching the project" to "weed out" the good and the bad. Trust in Thoeny as a leader diminished last but not least when his role as a community leader became more and more mixed up with his interests as a CTO of TWIKI.NET, up to the point where he obviously showed more interest to cement a genuine marketing advantage for TWIKI.NET.

The rise of his newly created company continually eroded willingness to contribute to TWiki as an Open Source project. People were more and more irritated by the changed rules of the game. The community has been watching the actions of TWIKI.NET with a lot of interest, in the hope that they would add significant value to this very successful project. Unfortunately, they took an approach of recasting the success of the product, created with years of volunteer work, as their own success.

That's where Open Source shows its ugliest face. And there's definitely no beauty in this shock therapy, even though Tom Barton, CEO of TWIKI.NET says: "the beautiful thing about open source is you don't need to recognize the authority of TWiki.net". What an irony to close another very sad chapter. The last one for TWiki.org as we knew it before.

The appearance of TWIKI.NET on the scene forced a governance crisis TWiki was not able to overcome, despite the good progress that was made up to a couple of hours before. On the TWiki Summit in Berlin last month, a democratically elected Interim Board of Directors was founded whose sole agenda was to negotiate the conditions under which this governance crisis could have been overcome.

The plan was to create a TWiki Association consisting of a Board of Directors and a General Assembly following the example of KDE e.V. The board itself would have created so called Task Teams that manage the operational part of the project to a finer granularity.

The members of the Interim Board of Directors were in the process of creating the Articles of Association and were prepared for the next logical move in an ever growing project, organizing it similarly to other projects in the Open Source business. This formal body would also have been an entry path for sponsors and other organizations willing to partner with TWiki as a project. No such thing was available before. The only way outside parties could have made donations was to give them directly to Thoeny and thereby TWIKI.NET.

This was the case when Sun donated server hardware to power the TWiki.org community site. Sun sponsored TWiki as an Open Source project, not TWIKI.NET. However, there was no entity other than Thoeny and TWIKI.NET to handle these opportunities and resources. It now is clear that the access to these server resource has been used against the TWiki community itself by locking it out.

The democratically elected Interim Board of Directors of TWiki has been displaced by the trademark holder of TWiki as a final chord on the governance crisis. Now, Thoeny is sending around emails to high profile contributors individually to invite them to come back subordinate to the governance of TWIKI.NET. He obviously seems to be in hope that people will do so once the situation has settled. Quite far-fetched and not very likely to happen. Those same contributors who implemented the features he is praising aloud as the shiny new TWiki, are far too displeased by his hostile behavior to be willing to go back to business as usual.

TWIKI.NET is striving to repaint their move as a "new opportunity". What they don't see is that they have put their own business case into severe danger. They just lost the horse power for a product that they were selling. They have been signaling to the community that they don't have the manpower for certain developments and were seeking for help, even willing to pay work for hire. Another error. Adding money as an incentive to Open Source is changing the game completely. Before, people volunteered as part of an act of free speech. Add money to it and nobody will work for free anymore. This poisoned the dynamics.

The current situation is that all core developers have left the ship and joined a new undertaking with the working title NextWiki. This is a fork based on the current code in TWiki that will soon be released under a new name. The goals of NextWiki are clear. Basically, the plan is to found an Association as a formal body for the project, including the reorganization of its governance down to all operational questions, as was in progress for the TWiki project.

The result will be a much strengthened new player, much more agile as it just got rid of the reason for TWiki's ongoing paralysis.

There remains a message for TWiki's users: no worries, we continue working, faster and more productive than ever before, embedded in a volunteer-friendly environment. Sure, this fork now introduces a new choice that was not there before. Well, it was there before and it was introduced by TWIKI.NET, not those guys that "asked for a fork". TWiki users already had the choice between TWIKI.NET's product (a rebranded version of an old TWiki release, packaged as a VMware image), or Open Source TWiki, most recent stable version. This choice more or less remains available with the difference that you will get the real thing from a new site, reworked to be real Open Source, backed up by a large and highly motivated community as a guarantor for continuity and innovation.

Tags: community, fork, opensource, twiki

Get a decent T(wiki)-Shirt in the TrickyWikiShop. Have your favorite slogan on the back when you meet the TWikiCommunity on the upcoming TWiki Community Summit in Berlin on 4th and 5th September.

Update

Due to recent changes in the TWiki eco-system, this shop has stopped selling TWiki accessories. However, we will update the selection as soon as we've got a proper name for the NextWiki fork.

Tags: merchandise, twiki

A new plugin uses the Safe module in Perl to constrain perl scripts in TWiki topics so they are safe to execute on your server.

We recently developed a TWiki plugin to support execution of Perl scripts that are written in TWiki topics. The scripts are executed on the server, and of course that means we have to do everything possible to ensure those scripts don't open security holes.For years Perl has had the Safe module, a clever package that provides a tightly constrained execution environment for Perl eval statements. Perl compiles all its code to a rich set of high level opcodes, which are then run on a virtual machine. By limiting the set of opcodes that are allowed to be run in the container, the Safe module can be used to create a very secure execution environment.For example, most people would consider the perl 'backtick' operator to be very dangerous, as it allows the caller arbitrary access to the shell. Backtick has a corresponding Perl opcode – called backtick – and to disable it, all we have to do is to remove it from the set of legal opcodes. The Perl developers have even gone so far as to classify the operators according to the usual safety concerns that a caller may have, making it relatively easy to decide which to allow, and which to exclude.Of course there's more to safety than that. We also have to be sure that the code being executed only has access to the namespaces we want it to have access to. The default condition for scripts run in a safe container is that they can only access the namespace of the container. We have to explicitly grant the container access to other namespaces when we create it.Of course there are potential risks with allowing any sort of script execution on your server, but in the case of a web server behind a corporate firewall, those risks are relatively small, and the 'Safe' module helps to make sure that such scripts are well controlled.The new TWiki plugin, called the PerlPlugin, is released under the GPL and is available to all WikiRing consultants for deployment on client sites.

Tags: perl, plugin, security, twiki

How to convert Jot Spot data to TWiki

With the recent announcement about the new Google Sites application, a number of former Jot Spot customers have decided to migrate to TWiki. WikiRing partner C-Dot Consultants has been engaged to help, and this post describes our experiences.

Jot Spot stores topic data in an XML database. Within that database, actual topic data is stored as "decorated HTML"; the basic topic content is HTML, within which Jot applications are embedded using Jot Spot's proprietary script, which uses XML tags.

Because of some fairly fundamental architectural differences between Jot Spot and TWiki it's not simple to automate the conversion of Jot Spot applications to TWiki. Fortunately our clients have not invested heavily in developing Jot applications, but have instead chosen to use the applications that Jot Spot provide by default. So the focus of our work has been to:

1. Convert existing Jot Spot topics to TWiki, with minimal formatting loss
2. Map a subset of Jot applications (most notably the Bug Reporter) to TWiki

Fortunately we were able to secure an XML dump of the Jot Spot database. This has alllowed us to perform the conversion without relying on the patchy availability of the Jot Spot site. Conversion of Jot topics has been achieved by leveraging a couple of existing technologies:

• The SAX XML parser from CPAN
• The open-source HTML to TML convertor we wrote for WYSIWYG editor integration into TWiki

SAX allows us to parse the Jot XML, pick out the form fields, and identify the subset of the topics suitable for passing on to the HTML to TML convertor. The HTML to TML convertor is already a proven technology, used every day with the Tiny MCE integration in TWiki, so it is robust and minimises formatting loss.

On the receiving side, we have customised the publicly available TWiki:Plugins.BugsContrib to support the data fields from the reporter in Jot Spot. We have had to develop a number of new reporting screens, something which has been made much simpler by the use of the type="query" search we contributed to TWiki 4.2.

It's cheering to note how similar the structure of a Jot Spot topic is to a TWiki topic. I guess you could call it convergent evolution!

Tags: conversion, google, jotspot, twiki

A Long, Long Time Ago Rafael Alvarez contributed a REST architecture for TWiki that until relatively recently had been largely ignored. That's a shame, because TWiki has fallen behind the curve on effective interaction, partly because it is so difficult and inefficient to interact with TWiki from Javascript. More recently we have been able to re-architect big sections of the core to make REST more useful. Writing a REST script still isn't all that easy, however. This post is intended to try and make it easier.

See the Wikipedia article for the full gory details of what REST is. TWiki developers can think of a REST as a way of calling a single function in a a plugin, in a place where TWiki::Func is available. REST handlers are designed primarily to be called from XmlHttpRequest, but can also be useful in IFRAMEs.

A REST handler is invoked via a URL to the rest bin script. For example,

%SCRIPTURL%/rest/WysiwygPlugin/tml2html?topic=%WEB%.%TOPIC%&text=Some%20test%20text

would invoke the TML to HTML translator on TWiki.org (if it was available there).

REST handlers can be added piecemeal to address specific requirements. However there are other structured approaches currently in development that the interested reader might be well advised to investigate:

• Michael Daum developed the XmlRpcContrib, which supports XML-RPC calls
• Olivre Krüger has developed a REST interface to the TWiki::Func module. This interface uses JSON and XML to communicate the results of calling Func methods back to the Javascript (or so Oliver tells me; I haven't seen the code yet)
• Sven Dowideit is developing a REST plugin that uses the content-access syntax develop for search queries and IF statements in 4.2 to provide access to data in the TWiki database. This is most promising work, as it is a strong move in the direction of the TopicObjectModel.

So much for the future; what about the present? How do I go about writing a REST handler? Well, I can only describe how I do it; I'm sure there will be other, probably better, approaches.

As an example, let's look in detail at the REST function that implements the attachments list in the WysiwygPlugin. This Javascript function will, when given the name of a topic, return a list of the attachments on that topic. It uses:

• XmlHttpRequest from within Javascript to make the request
• TWiki::Func to fetch the data that satisfies the request
• HTTP status codes to transfer state and erros back to the client
• JSON to transfer the data back to the client

First let's look at the server side of the solution. A REST handler is registered in initPlugin thus:

sub initPlugin {
    TWiki::Func::registerRESTHandler('attachments', \&_restAttachments);
}

Now for the handler body:

sub _restAttachments {
    my ($session) = @_;
    my BlogEntry23 = TWiki::Func::getCgiQuery()->param('topic');
    my WRBlog;
    (WRBlog, BlogEntry23) = TWiki::Func::normalizeWebTopicName(undef, BlogEntry23);

This first section of the handler simply examines the compulsory topic parameter to the REST call to determine the topic we are interested in. Note that normalizeWebTopicName will also untaint the web and topic names. Now we have the topic we can check the access permissions:

    unless (TWiki::Func::checkAccessPermission(
        'VIEW', TWiki::Func::getWikiName(),
        undef, BlogEntry23, WRBlog)) {
        my $error = "Access denied";
        print CGI::header(-status => 401);
        print $error;
        print STDERR $error;
        return undef;
    }

This demonstrates how to handle errors. If access is denied, we print a CGI header to STDOUT with a 401 (access denied) HTTP status code. We also print the message, and duplicate that print to STDERR so the message also goes to the Apache log. Returning undef from a REST function just causes the rest script to terminate without generating any more output, and is the recommended way to terminate on an error.

    my ($meta, $text) = TWiki::Func::readTopic(WRBlog, BlogEntry23);
    # Create a JSON list of attachment data, sorted by name
    my @atts;
    foreach my $att (sort { $a->{name} cmp $b->{name} }
                               $meta->find('FILEATTACHMENT')) {
        push(@atts, '{'.join(',',
                         map {
                             "\"$_\":\"$att->{$_}\""
                         } keys %$att).'}');

    }
    return '['.join(',',@atts).']';
}

Having passed the access control check, we now generate a list of attachment data in JSON format. We could have used the CPAN:JSON module to do this, but the data is so trivial in this case that the overhead of the extra module would have been excessive.

When we return non-null data from a REST handler, the rest script automatically generates a 200 HTTP (OK) status response, indicating correct termination of the call.

Now, let's look at the Javascript. First we need to build the REST url:

function attachments() {
    // Work out the rest URL from the location
    var scripturl = getTWikiVar("SCRIPTURL");
    var suffix = getTWikiVar("SCRIPTSUFFIX");
    if (suffix == null) suffix = '';
    var path = getTWikiVar("WEB") + '.' + getTWikiVar("TOPIC");

In this code we have used a function called getTWikiVar to obtain the values of some standard TWiki variables. How this function is implemented is outside the scope of this article, but a typical technique is to pass such variable values in META tags, as described in a previous post.

We now have sufficient information to build the URL for the REST script:

    var url = scripturl + "/rest" + suffix + "/WysiwygPlugin/attachments";

See the documentation accompanying Plugins.EmptyPlugin for more information on constructing REST urls. Now we perform a standard XmlHttpRequest

    if (tinyMCE.isIE) {
        // branch for IE/Windows ActiveX version
        request = new ActiveXObject("Microsoft.XMLHTTP");
    } else {
        // branch for native XMLHttpRequest object
        request = new XMLHttpRequest();
    }
    request.open("POST", url, true);
    request.setRequestHeader(
        "Content-type", "application/x-www-form-urlencoded");

    // nocache helps us defeat client-side cacheing of the result
    var params = "nocache=" + encodeURIComponent((new Date()).getTime())
        + "&topic=" + encodeURIComponent(path);
    
    request.setRequestHeader("Content-length", params.length);
    request.setRequestHeader("Connection", "close");
    request.onreadystatechange = function() {
        attachmentListCallback(request);
    };
    request.send(params);

This request will be sent to the server, and when a response is ready the Javascript function attachmentListCallback will be called.

function attachmentListCallback(request) {
    if (request.readyState == 4) { // only if "OK"
        if (request.status == 200) {
            var atts = request.responseText;
            if (atts != null) {
                atts = eval(atts);
                // atts is now an array of attachment objects, just like the perl
            }
        } else {
            alert("There was a problem retrieving the attachments list: "
                  + request.statusText);
        }
    }
}

And we are done!

Tags: ajax, javascript, json, rest, twiki, wysiwyg, xmlrpc

Getting the values of TWiki variables into Javascript can be a trial. This article describes one simple technique for doing that.

Transferring the values of TWiki variables into Javascript. It's no problem if you write your Javascript inline, but that will soon become impossible if security settings are enabled on the server. You could always write a REST handler to drag the values over from the server using a XmlHttpRequest – but that's like using a sledgehammer to swat flies.

What's really needed is a simple methodology for making the values of selected TWiki variables available to Javascript.

The problem is that while TWiki variables are expanded in templates, unless your Javascript is inline there is no way of getting that value into the Javascript. For example, let's take a simple example; we have an input, and we want to invoke a validator on it:

<script src="%PUBURL%/%SYSTEMWEB%/MyPlugin/validator.js"></script>
<input type="text" value="Urgh" onchange="javascript:validateInput()">

The validator is in a Javascript file called validator.js

function validateInput() {
   var url = '%SCRIPTURL{rest}%/MyPlugin/Validate';
   ... perform an XmlHttpRequest to get the value validated...
}

The problem is that there is no way to expand %SCRIPTURL{rest}% inside validator.js. Somehow we have to pass this value to the Javascript.

The easiest way to do this is in the DOM. And HTML generously gives us the META tag that we can use as we see fit. So in our plugin, we can write:

TWiki::Func::addToHEAD('MYPLUGIN', '<META name="MyPluginData" value="%SCRIPTURL{rest}%">');

Now we change validator.js as follows:

function validateInput() {
   var meta = document.getElementsByTagName('META');
   for (var i = 0; i < meta.length; i++)
      if (meta[i].name == 'MyPluginData') {
         scripturl = meta[i].value;
         break;
      }
   var url = scripturl + '/MyPlugin/Validate';
   ... perform an XmlHttpRequest to get the value validated...
}

Oviously if you are passing a number of values, it makes sense to do the search of the META tags once and build a hash of the values.

If you can'y write to the header for some reason, you can also use other (body) tags to perform a similar trick, and stop them being displayed using style='display:none'

STC Spring Event highlights issues of Technical Communicators, and reveals an interesting new way of assessing "Web-2-ness"

Yesterday I attended the STC (Society for Technical Communication http://www.stc.org) spring event, where I was invited to give a talk on uses of Wikis and Blogs in industry. There was a great deal of interest in the way wikis can help enable collaborative authoring, especially in reducing the length of the revision cycle, and in engaging SMEs in the writing process.

I shared the platform with Ellis Pratt of Cherryleaf who was speaking about the impact of Web 2.0 on technical communicators; not a small subject, it emerges. Ellis drew a lot of examples of new and existing sites that either claim, or actually do, implement Web 2-ness. Ellis had an interesting way of breaking down Web 2 so that it is more comprehensible to the layman. He breaks down the degree to which sites/technologies contribute to Web 2 on the basis of three criteria:

• Collaboration
  • the degree to which the site/technology enables multiple parties to work together
• Conversation
  • whether the site/technology enables an exchange of views between users
• Aggregation
  • whether the site/technology enables new ways of collecting/mining other data

One interesting observation was the way that many products are now being independently documented on the web, as users, frustrated with the availability or quality of information, write and share their own.

Amusingly enough both Ellis and I had incorporated the KCU The Machine is Us/ing Us video in our presentations. Watch it, it's really good.

Altogether a very enjoyable day.

Tags: collaboration, user, web2, wiki

The current TWiki::Cache implementation has been proposed for integration to the next TWiki release (code name freetown).

The current code implements some storage backends now. Before we only had a Cache::Memcached based backend which is great on your own servers, but not applicable for little TWiki installations in hosted environments where you are not allowed to launch daemon processes.

What we have right now is

• Cache::Memcached
• Cache::FileCache
• Cache::SizeAwareFileCache
• Cache::MemoryCache
• Cache::SizeAwareMemoryCache

I did not make use of the shared memory variants as they are quite limited in their storage capacity. The Cache::FileCache is said to perform as fast anyway, but does not have those restrictions. For html page caching the first two variants are the most interesting. Basically you can forget Cache::SizeAwareFileCache, and it's only there for completeness. This backend simply performs too much maintenance on its own behalf to be usable for TWiki. The Cache::SizeAwareMemoryCache, however, is comparably fast purging out pages, as it does not suffer from the extensive disk IO that happens in the Cache::SizeAwareFileCache. Both memory variants are not able to share their cache among several processes, and thus are obviously out for html page caching. The current WikiRing blog runs on a Cache::FileCache now. It used the Cache::Memcached backend for quite some time. But as I am low on RAM but not on disk space on this server anyway (anybody out there that likes to sponsor a bigger server or more memory?) I switched to the file-based variant, which performs quite well.

I still owe you performance statistics, hard facts.

Tags: caching, performance, twiki

Dinky little quiz plugin adds value to TWiki-based training

I was asked on #twiki the other day if there was any support for generating online quizzes. I wrote something to do that years ago, so I got the garden spade out and went digging. The result is the TWiki:Plugins.QuizPlugin, an ultra-lightweight plugin that lets you develop simple quizzes. These are just the thing for embedding in online training, where you want to checkpoint a reader's learning in a non-threatening way.

At the moment the plugin doesn't capture any statistics about how well the quiz was answered, but it could be rather amusing to log that data. If your training course is working well, then the reader will answer the questions correctly. If it isn't, then it would be helpful to gather information about what parts of the message aren't getting through.

To avoid forcing people to log in to take the training, the data could be logged to the plugins work area. By saving that data as a CSV file it could easily be imported into a spreadsheet for analysis.

It will be interesting to hear feedback, and see if anyone is interested in taking it further.

Tags: collaboration, plugin, twiki, wikiapps

I was talking to James Matheson of Saikore a few months ago, and we thought it was about time that there was a 'Wiki-Wednesday' in Sydney, Australia. James has organised it while I was in Zurich.

An informal gathering of wiki enthusiasts will be held at the Greenwood Hotel at 6:30pm on Tuesday 27th February.

Join us at the Main Bar for a couple of drinks and some general conversation. There is no formal program, but the event is inspired by SocialText's Wiki Wednesday events, and you are encouraged to chat, learn about wikis, find jobs, talk deals and generally cavort.

The details are:

• Event: Wiki Tuesday
• Location : The Main Bar of the Greenwood hotel, North Sydney.
• Date: Tuesday 27th February, 2007
• Time: 6:30pm till late
• Refreshments: Food and drinks available at the bar

Sydney Wiki Tuesday

Tags: meetup, saikore, sydney, tuesday, wiki

WikiGardens.com is a new TWiki-based web site aimed at plant lovers. Developed with technical services provided by WikiRing, the site features a host of features designed to allow both professional and home gardeners to share information about plants, gardening, and all related matters.

Some of the initial features of the site include:

• A plant database with over 28,000 initial entries.
• Discussion forums about specific plants or general gardening topics.
• "Our Garden" showcase that allows gardeners to show off their creations pictures and blog-like journal entries so that can report on the garden throughout the season.
• Local gardening resource listing to help gardeners find local gardening groups and venders.

This site demonstrates the flexibility with which TWiki can accommodate both unstructured "wiki" input and structured data (such as the plant database). It also demonstrates how TWiki can provide all the functionality of several web applications (i.e. forum, gallery, wiki, etc) within an integrated framework.

Tags: gardening, twiki, wiki, wikiapps

It's a common scenario; a manager unplugs their laptop and goes travelling for a week. During their time away they want to work in the team wiki, but often don't have a suitable connection. It would be good if there was a way to let them roam, and re-synch their changes when they have a good connection again.

There have been a couple of attempts at building such a roaming technology into TWiki, but these have suffered the same flaw. They try to re-use third party synch tech, in the belief that it's easy to synch the database (the raw topic store). Unfortunately it's not so easy to synch the actual topics – histories, meta-data, attachments etc. Simply synching the topic files ends up with corrupt or lost histories, lost content, frustrated users. This article describes the ReadWriteOfflinePlugin which has been built by WikiRing engineers to address this problem.

The first problem is that synching topics is non-trivial. There are a couple of approaches you can take to this. The first is to treat a roaming machine as a client of the wiki server, and merge offline changes by rolling them forward into the server, then updating the client by copying what is on the server. This results in the roaming client being a mirror of the server. The second approach is to treat the roaming user as a peer of the server, acknowledging that the topics in the client may have different revision histories, and seeking just to merge the content of the topics. The first approach is appropriate in an environment where there is a single server sitting in the middle with lots of satellites. The second is more appropriate in a multisite environment, and allows the scenario where a laptop synchs webs from several servers, or even where multiple servers synch with eachother.

The ReadWriteOfflinePlugin uses the second approach, as it offers the most flexibility in how you configure your servers, and your roaming clients. The synching problem is actually quite similar to the problem faced when two users simultaneously edit the same topic. You have to choose how to merge the topics so that you lose as little information as possible. Thanks to previous work done by a WikiRing partner, TWiki supports asynchronous edits by multiple users, so we can leverage this tech. All we need to know is what version of the topic was last successfully synched, and we can merge together server and roamer changes to create a new, common topic version, without losing history information on either server or client.

Choosing the tech you use to synch the data is not trivial either. Most third party solutions use something like rsync, or a similar low-level copy program. These are generally fast, but often require special configuration on the server, such as opening ports.

However with a wiki, you have an advantage; you know that it's happy to serve HTTP, so the ReadWriteOfflinePlugin uses that protocol for the synch. It won't be as fast as rsync, but as long as you only synch the topics you know have changed, that shouldn't be too much of a problem. It requires no special setup, and as long as the plugin is installed on both server and client, it "just works". The synch uses special URLs defined by the plugin to maximize synch performance.

A final problem is deciding when to do the synch. Do you synch at every opportunity, or only when asked to? What triggers a synch?

At the moment ReadWriteOfflinePlugin has to be explicitly triggered, but plans are afoot to make a client detect automatically when a server it is synched to is available, and automatically synch when it sees changes. This requires sensing not just topic changes, but also sensing when the bandwidth of the connection is limited, because of course you don't want the synch to saturate your 100 baud dialup line!

An common question relates to security of the connection used to synch the server and client. A server really needs to support https: connections to make this secure. If it does, then ReadWriteOfflinePlugin can use it.

The ReadWriteOfflinePlugin is currently being tested by WikiRing partners and will shortly be available from the WikiRing Solutions Repository for downloading by subscribers.

Tags: ajax, database, repository, twiki

Sven Dowideit takes a closer look at a couple of existing TWiki skins and how they work.

Sadly, they are not much help for new skin developers and web designers, as they set no real consistency that can be built on. To try to resolve this, I have altered the default skin to offer better reusablility – initially as a base for the new TWiki CssZengardenSkin.

The documentation in TWikiTemplates defines the twiki.SKIN.tmpl file to be a convenience container, rather than a high level layout that can then be specialized by the HTML Page Template. The TMPL:P directives suggest to me another approach – one that gives HTML conversant designers a single entry point into the TWiki skins system.

After successfully creating the proof of concept CssZengarden Skin (and using it on my Blog/Website) a month ago, work stalled. At that point, the TWiki Skin comprised only of a single twiki.zengarden.tmpl file and some css – all other template definition being inherited from the TWiki default skin. What I hope to do is to define the overall look and feel in one file, and then inherit the twiki specific parts from the default skin – but this is turning out to be impossible without changing the way that the default TWiki skin is implemented.

Trouble spots

Reading the TWikiTemplates page with a programmers viewpoint, makes it clear why its not so simple.

1. The Master Template (twiki.SKIN.tmpl) defines some common named blocks
   • defines only a few TWiki specific sections, with few contextual cues for the non-TWikSkinMaster
   • the TWikiTemplates description suggests that there is no pre-defined Master template, only a tradition of using twiki.SKIN.tmpl. NatSkin and PatternSkin have dispensed with the twiki.SKIN.tmpl file, replacing it with body.SKIN.tmpl, page.SKIN.tmpl, constants.SKIN.tmpl etc.
2. Every top level tmpl file (HTML Page Template) defines its own final output
   • Which prevents you from knowing that your highlevel layout is used, and makes reuse harder
   • This is historical, originally, templates were simple HTML files, one for each cgi script.
3. Lack of predefined sections
   • Users don't know what sections you need to over-ride to achieve a local modification in all applicable skins
4. Unknown number of resultant outputs – I can't find a list of tmpl files that must exist for TWiki!
   • How can anyone know that all customized screens are done? How to test it?
5. very complex seeming Skin path for resolving what defines a template.

TWiki Template Directives

Resolving these issues requires an analysis of the tools (template directives) available to the skin developer.

1. %TMPL:INCLUDE{'othertemplate'}%
   • The simplest seeming template directive, but with a non-trivial twist. This directive attempts to import in-place the contents of another template by the specified name, using the Skin path. (will look for othertemplate.zengarden.tmpl, and if it does not exist, othertemplate.tmpl.)
   • All TMPL:INCLUDE statements are recursively evaluated until there are none remaining
2. %TMPL:DEF{'element'}% … %TMPL:END%
   • defines an named 'block' of text that is later used via the %TMPL:P% directive below.
   • if there are multiple blocks with the same name, the last one in the INCLUDE processed template wins
   • All %TMPL:DEF{'element'}%'s are parsed and loaded into TWiki before the next step
3. %TMPL:P{'element' parameter='...'}%
   • with reserved parameters context, then and else
   • finally, all TMPL:P's are replaced recursively with the TMPL:DEF's defined in the previous phase.
4. Literal text – any text not within a TMPL:DEF .. TMPL:END section makes up the resultant output seen by the user.

TWiki.TWikiTemplates guides the potential skin designer to create many different HTML Page templates each defining the Literal Text (is there a canonical list of required templates?), and then to push commonly used blocks up into the Master Template. This essentially ensures an inability of skins to interoperate, as the creation of named blocks is necessarily ad-hoc and specific to the skin.

There are many examples of named blocks that should be created that aren't, and many local to Page Template blocks, that only differ slightly from their equivalent blocks in other Page Templates. This approach makes it hard for new skin developers, who are forced to understand the entire system, and then to reproduce it. It also means that you cannot TMPL:INCLUDE HTML Page Templates, as this results in duplication of Literal Texts

A possible solution

If we treat the Master Template as a true master, with the HTML Page templates only defining named blocks that differ, we might improve re-use, and simplify the creation of new skins. This inversion is done, by moving all 'Literal Text' into the twiki.SKIN.tmpl, and TMPL:DEF-ing Page specific elements in the HTML Page template. This also has the benefit that there can be a standardized set of sections that become the default starting point for skin developers. Frustratingly, both NatSkin and PatternSkin go part of the way, defining page.nat.tmpl, default*.nat.tmpl etc, but not seemingly to re-use parts of the default skin, or to make it easier to integrate custom skin changes into multiple skins. Worse, they deviate totally from the advise given to other skin developers, in not defining a twiki.pattern.tmpl file at all.

Adding some code into TWiki::Templates, we can study what is currently happening.

I've attached a patch for TWiki::Templates, and a script to that can be used to create the reports below. They show:

1. the results after TMPL:INCLUDE processing is done, and record what templates the skin path code has used, and thus from where the named blocks come (templates_postinclude)
2. the results after template expansion is done (templates_cache)
3. analyze commonalities in the named blocks, and where they could best be re-used

Inspecting the output (from templates_postinclude) of all the skins installable from svn, shows that no skin author is re-using the building blocks from the default skins. The following is a count of TMPL:DEF, TMPL:P and TMPL:INCLUDE elements that are used to make the view template for each skin. Note how even though ClassicSkin and DefaultSkin look very similar, as ClassicSkin totally duplicates the files in the DefaultSkin, and how NatSkin and PatternSkins also, have nothing in common with the more basic skins.

svens-computer:~/src/MAIN/templates_postinclude sven$ ../template_stats.pl view__pattern__.tmpl 
=========
view__pattern__.tmpl 
P (91) : 
DEF (89) : 
INCLUDED (12) : 
   %{ INCLUDED(/Users/sven/src/MAIN/templates/view.pattern.tmpl) view, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/page.pattern.tmpl) page, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/constants.pattern.tmpl) constants, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/javascript.pattern.tmpl) javascript, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/styles.pattern.tmpl) styles, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/stylesdynamic.pattern.tmpl) stylesdynamic, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/body.pattern.tmpl) body, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewtopbar.pattern.tmpl) viewtopbar, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewtoolbar.pattern.tmpl) viewtoolbar, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewleftbar.pattern.tmpl) viewleftbar, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewrightbar.pattern.tmpl) viewrightbar, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewtopicactionbuttons.pattern.tmpl) viewtopicactionbuttons, pattern, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewbottombar.pattern.tmpl) viewbottombar, pattern, TWiki }% 
svens-computer:~/src/MAIN/templates_postinclude sven$ ../template_stats.pl view__zengarden__.tmpl 
=========
view__zengarden__.tmpl 
P (63) : 
DEF (55) : 
INCLUDED (3) : 
   %{ INCLUDED(/Users/sven/src/MAIN/templates/view.tmpl) view, zengarden, Sandbox }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/twiki.zengarden.tmpl) twiki, zengarden, Sandbox }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/twiki.tmpl) twiki.tmpl, zengarden, Sandbox }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/css.tmpl) css.tmpl, zengarden, Sandbox }% 
svens-computer:~/src/MAIN/templates_postinclude sven$ ../template_stats.pl view__classic__.tmpl 
=========
view__classic__.tmpl 
P (46) : 
DEF (43) : 
INCLUDED (2) : 
   %{ INCLUDED(/Users/sven/src/MAIN/templates/view.classic.tmpl) view, classic, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/twiki.classic.tmpl) twiki, classic, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/css.tmpl) css.tmpl, classic, TWiki }% 
svens-computer:~/src/MAIN/templates_postinclude sven$ ../template_stats.pl view__default__.tmpl 
=========
view__default__.tmpl 
P (44) : 
DEF (40) : 
INCLUDED (2) : 
   %{ INCLUDED(/Users/sven/src/MAIN/templates/view.tmpl) view, default, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/twiki.tmpl) twiki, default, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/css.tmpl) css.tmpl, default, TWiki }% 
svens-computer:~/src/MAIN/templates_postinclude sven$ ../template_stats.pl view__nat__.tmpl 
=========
view__nat__.tmpl 
P (17) : 
DEF (40) : 
INCLUDED (8) : 
   %{ INCLUDED(/Users/sven/src/MAIN/templates/view.nat.tmpl) view, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/page.nat.tmpl) page, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/strings.nat.tmpl) strings, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/defaultbody.nat.tmpl) defaultbody, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/defaultsidebar.nat.tmpl) defaultsidebar, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/defaulttopbar.nat.tmpl) defaulttopbar, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/defaultjavascript.nat.tmpl) defaultjavascript, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/viewtopicactions.nat.tmpl) viewtopicactions, nat, TWiki }%
   %{ INCLUDED(/Users/sven/src/MAIN/templates/searchbox.nat.tmpl) searchbox, nat, TWiki }% 

To me, this is totally unexpected, because in my conversations with people about defining tmpl's we've discussed inheritance, over-riding, and specialization – none of which seem to be taking place. I had hoped that common issues, like the html, head, body tags, copywrite, would be dealt with in the default skin, and then shared out. Nope – just looking to see where the <html> tag is defined :

svens-computer:~/src/MAIN/templates sven$ grep -i '<html' *
actionnotify.tmpl:<html>
createtopic.tmpl:<HTML>
edit.wikiwyg.tmpl:<html>
page.nat.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">
page.pattern.tmpl:<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">
twiki.classic.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%"> %TMPL:END%
twiki.dandruff.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
twiki.quickmenu.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">%TMPL:END%
twiki.see.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> %TMPL:END%
twiki.seneca.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
twiki.simpleblue.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">%TMPL:END%
twiki.tiger.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> %TMPL:END%
twiki.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%"> %TMPL:END%
view.cc-movabletype.tmpl:<html xmlns="http://www.w3.org/1999/xhtml">
view.dallas.tmpl:<HTML>
view.nav.tmpl:<html>
view.plain.nat.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">
view.print.nat.tmpl:<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="%LANG%" lang="%LANG%">
view.slidy.tmpl:<html>
view.tiger.tmpl:<html>

Implementing the 'Possible Solution'

To show what I'm talking about, I'm refactoring the default skin so that twiki.tmpl contains most of the HTML based definition, and each of the <script >.tmpl files contain only changes specific to those scripts.

What I changed:

• The <script>.tmpl files contain currently all contain some trivial variation of the following. These I moved into twiki.tmpl

  %TMPL:P{"htmldoctype"}%
  %TMPL:P{"head"}%
  %TMPL:P{"bodystart"}%
  %TMPL:P{"main"}%
  %TMPL:P{"bodyend"}%
  

• There seems to be no clear definition of what belongs in TMPL:DEF{content} vs TMPL:DEF{main}
  • I'm going to refactor based on the idea that the TMPL:DEF{heading} is not part of the TMPL:DEF{content} – I'm moving it into the definition of main
• TMPL:DEF{message} is used in the oops templates in a similar way to TMPL:DEF{content}
  • replace message with content
• TMPL:DEF{topicinfo} is a bad mix of topicactions and htmlfooter – containing copyright etc
  • will need to separate this out
• There's a lot of definition of bodystart, and almost all identical

Progress, without changing Zen

I now have a second revision of the CssZengardenContrib running on home.org.au – with very few changes in the twiki.zengarden.tmpl. Everything else is inherited from the default TWiki skin's templates – for which there is a patch file attached.. This means that we have a fully functional skin (last time, only view script worked well), with mostly tweaking and fixes to go.

The following code changes were made –

• TWiki::Template patch
• Template defaultskin patch
• template_stats.pl

More Zengarden demo urls using CssZenGardenContrib at my personal blog site (note the Next design links to browse):

• css Zen Garden default style - 'Tranquille' by Dave Shea – http://www.mezzoblue.com/
• css Zen Garden submission 065 - 'New Groove' by Martin Neumann, http://www.rushmedia.de/
• css Zen Garden submission 068 - 'Ballade' by Charlotte Lambert, http://charlotte.flibuste.net/
• css Zen Garden submission 069 - 'Bonsai Sky' by Mike Davidson – http://www.mdavidson.com/
• css Zen Garden submission 161 - 'Zenfandel' by Nicholas Rougeux, http://www.c82.net/

Tags: css, skin, twiki, zengarden

Today most sites use content engines that produce highly dynamic content, by assembling different objects to finally generate html and send over to your browser. This makes caching on the server side a must. As wikis emerge from simple one-page services and leap into the regions of content management systems, caching is increasingly attractive here too. Last, but not least, there are many more lurkers and page hoppers than contributors to a wiki, and there's no need to recompute the same pages again and again.

Client-side as opposed to server-side caches try to cache as close to the browser as possible, to reduce bandwidth congestion early. Such a cache gets an url, and returns a full html page. The only additional knowledge it has is the time the page is supposed to expire. There are also server-side caches that work along the same lines — called reverse caches.

The central task of a cache is to maintain its integrity (don't return outdated information). Caches tend to sacrifice a bit of integrity to get any caching effects at all. Now, the problem is that the cache and the content source don't talk to each other. If the content source has an update it should notify the cache about it, so that the cache can actively invalidate some of its store. To my knowledge there's no third party cache that implements such an API. Nevertheless, there are third party cache solutions that have very good APIs to integrate them into your own software. Two of them are Memcached and Varnish. While Memcached has bindings for several (scripting) languages, Varnish comes with its own scripting language. Memcached is more of a multi-purpose cache for any kind of storable objects; Varnish is a highly optimized server-side cache and restricted to url based caching only.

Frankly, notifying the cache so that it performs invalidation is no big conceptual problem. The hard bit is when and which parts of the cache shall be invalidated. The task of “dependency tracking” is to record which objects were needed to compute one another. Based on this information, dependencies are “fired” to recursively invalidate cache information. As you can imagine, one page can depend on a lot of other information spread all over the place. For example, the same page may look differently for each user due to user preferences, url parameters or session values. A wiki page may depend on a WikiWord not being defined yet, as it renders a link to it differently when it exists or not. While these case can be detected automatically there are also cases when the engine can't. Imagine a RandomPlugin that generates random blind text. A page using such a feature is basically not cacheable, as content materializes out of nowhere. More realistic examples are plugins that integrate external data sources. The knowledge about the content source is part of an external system. So it is responsible to establish dependencies to the objects that are constructed on its base.

Dependency tracking will be nearly impossible in other cases where there's no a priori knowledge at hand: a page showing query results. The page is assembled using its search results, and thus depends on the found items. Either this page can become invalid because the query does not match an item anymore, or a new item comes into existence that now matches the query. This is bad news as search queries are the most expensive and most valuable operation in any content engine. However, things are not that bad for caching as data does not change so frequently, and we can present the same results as long as we know that no data has changed. Think of it as a 1:n dependency instead of a couple of 1:1 dependencies between different objects.

Yet another type of dependency is content that changes over time, without any external input, e.g. a clock. You really don't want to cache anything here, the same way as it is pointless to cache random numbers on a page using your shiny new RandomPlugin. From the angle of a cache this type of content is pure “dirt”. In fact, while the rest of the page is quite static for a while, some areas in it may be filled with up-to-date information on every request. A way to deal with this is so-called “dirty areas” … which is nothing offensive but just a way to prevent the cache from being trashed with information it can't cache at all. In a way, cached pages that have dirty areas in it are rather similar to very specific templates.

Alright, so far I mused about caching in a more general way motivating the typical problems and outlined what can be done about it. Let me introduce you to the TWiki::Cache that has been implemented recently in one of my next postings.

Tags: caching, content, performance, twiki

Recent work on content repositories

Recently I've been looking closely at content repositories. "Content Repository" is a generic term adopted in the CMS community to describe a database that stores web content; fully rendered web pages, raw blog posts, templates etc. Content repositories are generic databases which trade off efficiency for flexibility. There are quite a few alternative 'off the shelf' repositories out there, mostly either tightly bound into applications, or very loosely bound (usually serving XML). All of them have something you might want; none of them have everything.

What I really want is:

1. A simple repository specification that can be rapidly and efficiently implemented to provide local repository services on both client and server
2. The flexibility to link these services in the background using offline update agents
3. Independence from any specific language or server platform
4. Something that is reasonably efficient for traditional CMS applications

But what I really want is a platform on which to build new web applications really fast without too many compromises.

Some time ago the Java community identified the need for some standards in this area, and I couldn't agree more. The approach taken by the Java folks was to define an API, now enshrined in a standard, the JSR 170. There is even a CPAN module for perlites wanting to access the API, and a strong reference implementation (Apache Jackrabbit). Several commercial vendors sell JSR 170 implementations, and there's a dynamic community involved in the next revision of the spec. A number of open-source projects have layered JSR 170 compliant interfaces over their existing content repositories.

After playing with Jackrabbit for a while I found myself asking, why is it so darned complicated? The reference implementation is very heavy, but it has to be honest to the spec. The API is defined in two levels, effectively "read only repository" and "read-wite repository". Level 1 is extremely rich with features. To me it looks very much like an API designed by committee, where no-one got their idea voted down. Having said that, it's really an excellent API. So, how about picking apart the API and seeing what parts an implementation really needs to provide? I'm not proposing anything more than a complementary API that is:

1. Able to leverage all the wonderful work the java folks have done,
2. Language agnostic (not tied to Java, or any other specific language),
3. Simpler – a subset of JSR 170,
4. More amenable to implementation using commonly available tools and libraries,
5. Much less tied to XML/XPath

So, I have embarked on this voyage of discovery. I have an initial implemention of a content repository coded using perl, which leverages CPAN heavily. I've tried to stay true to the JSR 170 definition, except where language specifics dictate otherwise (e.g. Java uses method signatures that are not available in other languages). The goal is to produce a reference implementation of this cut-down API (working name WCR). If we feel it's powerful enough, then we'll move on to a high performance implementation in a proper language, and a web interface. But the key is the API, not the implementation. The idea is that an application using WCR can use a standard JSR 170 implementation, assuming a compatible interface exists.

The resultant content repository won't be tied to any specific purpose. It's possible that the first application of it will be a wiki, though.

Tags: database, repository

The relationship between Perl and CPAN

A long-running argument has popped up again in the TWiki development community recently. Some developers, call them "half fullers", believe that the huge volume of code in CPAN is there to be used, and that developers shouldn't try to duplicate what someone else has already done. A lot of CPAN modules are pretty rubbish, but there are some incredible contributions there. What's more, someone else is maintaining them! What more could you ask?

The "half emptiers", on the other hand, see CPAN as, at best, a necessary evil. The argument is that CPAN modules are large (due to their desire to be generic) and often slow as a result. Worse, CPAN modules are hard to install for a novice or unprivileged user. Therefore in the name of "customer focus" CPAN should be avoided, and application-specific functionality coded instead. If that's not possible, the CPAN modules should be bundled with the application.

This argument seems to me to be a question that strikes at the very heart of Larry Wall's original concept for perl. Larry's goal was a language that is not the fastest, not the most elegant, not the easiest to maintain, but is the most fun to code in and is the quickest way to get the job done. CPAN is a critically important part of this; it provides pre-built and tested modules of common (and uncommon) functionality that you can just pick off the shelf and run with. OK, so they are not the best implementations in the world. But they are free, are already there, and do stuff you can't be bothered to do, or just aren't smart enough.

To me, the very idea of using perl as a programming language, and then not leveraging CPAN, is oxymoronic. If you want performance, then what on earth are you doing using an interpreted language? If you want installation simplicity, go and write an installer! It's hardly rocket science.

Of course, if you want to reinvent the wheel, that's fine with me. Just don't expect me to do it, and don't act all surprised when I swap out your homebrew code for a tested, maintained equivalent from CPAN.