Teapot 7

October 4, 1997
by Stephen Taylor

An idea waiting to happen

I was bored with my job - bored with scripting yet another button to jump to yet another screen, and there were no interesting projects on the horizon. In the interest of self preservation, I started looking around for something challenging to write. At first, I only knew two things about this project: It had to be done entirely in Lingo, and it had to be something which could make me money - pure Lingo, because I wanted to be sure I could go cross platform, and I didn't have a Mac C compiler handy, and money because I like the stuff.

After a few days of idle speculation, the idea hit me like a lightning bolt - write a Director to Java translator. There was so much going for it - Director is such a good tool for simple multimedia projects, but it's either unknown or despised in the general programming community. The whole world has gone Java crazy. Using Director to pump out Java adds up perfectly... But I shelved the idea after a few days of note taking. There were far too many sandtraps - for a start, there wasn't even a formal specification for Lingo's syntax, something that's easy to find for most languages. The problems of exporting assets using only Lingo looked unsolveable (Sure, I could have done it in C - but I made myself three promises at University - never to learn COBOL, never to do any SQL work, and never to do any Windows programming. I've already well and truly broken the SQL promise, so I thought I'd better not sail too close to the wind with the Xtra). After a couple of hours of paging through the manual I realised that I was effectively asking myself to re-implement all of Lingo in Java - it's too big, and it's behaviour is too poorly specified, as a quick look at the unLingo Dictionary will show. Finally, I knew that Macromedia would certainly be working on some sort of Java export mechanism if they had any sense at all... and as recent announcements show, they do.

So I put my notes into yet another folder of projects-never-to-be-done and forgot them. A few months later, when Dija, the first Director to Java exporter was announced, I was furious. How dare they read my mind! It's bad enough with the CIA and the aliens... I made two resolutions - i) never again to take off my aluminium foil brainwave-blocker cap and ii) to get back to looking for a Director project.

The earlier work I'd done on parsing Lingo suggested something: I'd already used automatic documentation generators on the NeXT platform, and I'd played with the JavaDoc program which comes with Java. Why not do the same for Director? That was the genesis of Teapot-7 Documentor.

In the beginning

What could I do to make the information embedded in a Lingo program more accessible? Just putting all of the scripts into HTML form and creating an index would be a start - and better than nothing, as you wouldn't need a copy of Director running just to see what scripts you had handy. That would be ignoring the power of hypertext though - Lingo programs could become so much more transparent if you could just click on the name of the handler that was being called and instantly jump to the source code for that handler, to see what it really did.

Given that, it only made sense to do the same for global variable, so that clicking on any use of a global could give you information about which scripts it's declared in. Given that I was doing handlers and globals, I couldn't stop without documenting properties, where they were declared and where they were used, could I? Given that I was gathering all of this information, I might as well provide central lists of all the handlers, all the scripts, all the globals and all of the properties - and if I couldn't decide whether to list them alphabetically or in order of occurrence, why not provide both? Each feature suggested - even demanded - the next, so the Xtra began to take shape under its own power. Perhaps because of this, the project sprawled: I printed out the source code before I started writing this, and there are 37 pages of six point type hidden inside the Xtra. I'm pretty sure that if I'd planned every feature from the very start it would fit into 20 pages. That can be the problem when you build something new though - you don't know what you've built until the roof goes on.

Making OOP easier

One of my housemates at the time was a Smalltalk programmer, and I was having a talk with him about how reluctant a lot of people on the DIRECT-L mailing list were to embrace Object-oriented design. Eventually I said in their defense that after all, Director users didn't have access to the sophisticated object browsers that came with the Smalltalk development environment - and something clicked, and another feature was born. The Object Browser section is my favourite part of the Documentor - it takes an object and lays out its ancestry, showing which ancestor each handler and property originated in - and in which ancestors handlers have been overridden. This is where all the information gathered from the scripts is shown in it's most compressed form.

Literate Programming

One other feature had a less than obvious origin: I've always been interested in the idea of Literate Programming - the idea, developed by Donald Knuth of writing one integrated text which is mechanically separated into program and documentation - thus making programs more self documenting, and helping to keep documentation from being out of date. I realised that since you can put whatever you want into your comments without bothering the Lingo interpreter, it would be possible to embed instructions to the Documentor inside comments. A normal Lingo comment starts with "--". A comment starting with "--^" is still a valid comment, but is recognised by the Documentor Xtra as being special. When handlers are listed, these special comments are shown along with them, while the other, less important comments are left inside the handler where they belong. This way, the index of your handlers can go a long way towards giving an overview of your program.

The final significant feature had nothing to do with documentation as such, but I thought it served a useful purpose. One of the things about Director which has always infuriated me is the interaction of two Lingo features: i) local variables come into existence automatically, the first time you use them. ii) A handler only knows about global variables if you declare the global in the handler, or in the script containing that handler. Working together, these features make a particular type of bug prevalent in Director programs, in which you forget to declare a global variable in one of your handlers, and a local variable of the same name is magically created for you - along with a hard to find bug! The 'errors' section of Documentor list all local variables which have the same name as a global declared elsewhere in the program. You can look through this section and see if you've made this common and annoying error.

Motivation

Part of the motivation for writing the Documentor was the desire to make life easier for Director users, but it also - more cynically - occurred to me that in these days of Total Quality Management and other management fads, some of the bigger corporate clients might see quite a lot of perceived extra value in a set of HTML documentation which took only $65 and a mouse-click to create. Shame on me for thinking such things... but it just might be worth remembering.

There were any number of technical oddities that came up while writing the Xtra. At the start it wasn't even going to be an Xtra - it was going to be a standalone Director program which would dive into a movie, pluck out the script text it wanted, and jump back to the home movie. It was only while reading the XtraNet documentation that I realised that Xtras could be written entirely in Lingo, and hurriedly changed my plans. Parsing the text of the Lingo scripts proved to be a major headache, as it has to be done character by character, constantly looking for the end of each word or symbol. Using "item x of ..." is no use here, as the user is free to run words and symbols together and the Lingo interpreter can sort it out. And if it can, I have to.

I was surprised to find that Macromedia hadn't published the BNF for Lingo, so I had to work out even simple things like the allowed characters for a Lingo variable by trial and error. (What's BNF? Backus Naur Form notation, which is used to specify the legal constructions in a programming language, from what a legal variable is, right up to the highest level syntax.) I cheated a little, and used Director's auto formatting to help me parse the code - only to find that under some circumstance (multi line uses of the "tell" command) Director's auto formatting is completely broken. Thanks to the Australian programmer (Chris, I think?) who alerted me to this. As a consequence of this, the first patch to the Xtra came out about two days after the program's first release.

When I ported the Xtra to Dir 6. there were more funnies - a brand new bug (feature?) in D6 meant that my old way of closing MIAWs (I think it was "forget window the movie") no longer gave sane results, and needed a workaround. Other nasties emerged over time: Through laziness and familiarity I'd used the old FileIO Xobject for all my file I/O - conveniently forgetting that Xobjects aren't supported under Windows NT. I hurriedly shifted over to the FileIO Xtra and everything was fine - until I found that my character parsing table was missing a lot of accented characters, and that the Documentor would get unhappy when it found them. Since a lot of the interest in the Documentor has come from Germany and Switzerland, that came as a rude shock. As far as I can tell, everything is stable now - knock on wood.

Whats coming up

What didn't make it into the Documentor? A few bits and pieces. I had originally intended to allow it to do syntax based colouring of scripts - after all, I was already doing some of that in the HTML I generated, and it would be a handy feature. I decided not to though, as I was developing in Director 5 (and hadn't even seen D6 at the time), and programmatic script colouring is only possible in Director 6. I'd also found out that Matthew Cave was working on a script colouring Xtra, so I didn't see any point in going head to head with him. His Xtra is cheap and it works. Rush out and buy it. I had also originally intended to do more asset reporting - thumbnail sheets of graphics, lists of sound assets, markers, etc., but I decided against it, partly because I wasn't convinced it would be useful, and partly because I didn't want my Xtra to be dependent on other people's image-capture Xtras. If you think this would be useful though, email me -I'm always happy to listen.

What's still to come? There are (at the moment) two features still to come in the next revision: Colours for text, links, background and frames will become configurable options for registered users. This should be handy if you need to do a page in someone's corporate colours, or if you just don't like my taste in colours. (Zac - I cringed inside when I found you'd modified the colours of the whole objectShop site by hand. Can you ever forgive me?) I'm also aiming to incorporate some level of reporting on behaviours, but haven't yet decided what form it will take. As ever, suggestions are welcome.

It was a fun project, and satisfying to finish, though it was also rather humbling to realise how many loose ends I'd left untied. One of the benefits of writing the Documentor is the contact I've had with some very bright and very helpful people. I have to mention Nick (from Australia) and Toon (from the Netherlands) for error reporting and helpful suggestions above and beyond the call of duty.

Steve Taylor
Teapot-7 Software Development

Steve is an Australian and 34 yrs old. He's been programming for fourteen years now, doing a bit of everything - five years writing games in assembler for ancient game machines like the Sinclair Spectrum, assorted Unix/C jobs, including a stint on the hip but commercially unfeasible NeXTStep, and about three years of Director. Currently Steve is working in Smalltalk on a money market trading system. Major ambitions are to get back to writing games, only this time in Director. While living in Melbourne with his girlfriend, a house full of unread books and two cats, rumor has it that one of them either worships Steve as a living God or complains about his God complex.