Perl or How I learned to stop fearing and love CGIs

September 20, 1998
by Alan Levine

Our opinions are largely framed from our experiences, but often they are not universal truths. Huh? Well, one of the issues I've bantered back and forth with my esteemed colleague Zac, is the merit of CGI and especially Perl for doing the grunt work of interactive requests from web pages.

Since 1994, we've been using quite a few perl scripts on the machine that hosts DirectorWeb, running search requests, processing form submissions to databases, updating web pages every day, running web discussion boards, etc. Our CGIs are not dependent on browser flavor or version. Most of the scripts have been written by some clever students I've hired, who have learned perl on the job, and I've tinkered here and there. They are most likely not the most elegant programming ever done, but they do the job. Perhaps it's because we have a reliable unix web server, a fast disk, tons of RAM, a very fast network connection, or just incredible kharma. Thus, for me, perl & CGI rocks.

Judging from some email exchanges, I deduce Zac has had some bad voodoo experiences with perl, perhaps from oddball server configurations (hint: "Not Terrific") or perhaps he had to deal with restricted access to the server. It happens. For him, CGI is a "crappy kludge". So he finds other workable solutions.

And that's all that matters-- getting it to work, and work well, on the front end.

Even in the first beta of shockwave we had the ability to send a request to a server CGI script from Lingo, have it do something, and return a result. Early on, this opened up the possibilities for Shockwave as an industrial strength application tool rather than a spinner of eye candy dancing doo-dads.

Among the zillions of frequently asked questions on the director-related listservs, are many requests for help in getting shockwave to talk to CGI. In this article, I'll describe my approach and what I do to prepare a CGI request, and provide some examples of what we've managed to do with shockwave and CGI.

Since I'm not a perl wizard or even a worthy apprentice, I cannot give much advice about writing perl scripts. The key is being able to specify what inputs and outputs you want to happen when your shockwave app contacts your web server and hand it off to a perl pro. Personally, I do not find perl as arcane as the comments I read on the listservs. It's just another scripting language! If you an write logical constructs and function calls in Lingo, Perl presents just a different syntax and set of commands. For most of the stuff you would do from shockwave, in Perl you mainly manipulate string and numerical input, open, read, and write to files on a web server, and format the output.

So Zac, sit back, and relax. Here's my case for Shockwave/CGI.

"Why CGI?"

THINK ABOUT CGI SCRIPTS as some sort of Xtra or remote network programmed object that you can send some info and get something back in return. Done properly, like a Director Xtra or a well written Behavior, you never even have to worry about what goes on under the hood. (Hey, this is sounding "OOPy"!)

When you make any call to a CGI script, whether from shockwave or an HTML web form, the conversation goes like this:

Web Browser Client says, "Calling CGI script at this URL! Here is a variable and its value, plus another variable and its value, and ..."
CGI Script says, "Well I can sense some info about the client (browser, computer platform, etc), but I am expecting this variable with a value, that variable with that value, and... Now I will test their values, calculate some new values, perhaps write data to some file. Now, the script says to return some feedback in the form of either a string of text or a string of HTML."

Client waits patiently until it gets a response.

I've come across a handful of situations where a CGI script provides a work-around for things you cannot do in Shockwave.

Writing data to a server. The netLingo of Shockwave is like "FileIO" without the "O". With the getNetText(url) command, you can do the "I", but you never have the means to directly write to a file on a network server. A common need has been for game writers who want to keep track of high scores. Darrel Plant has written a tool for this, but HIGHSCOR is limited to a dedicated Mac or PC running a full-time net process, and a full-time unix server can probably run rings around that. Maybe not.
One of my earliest efforts was creating a shockwave front end to run a survey; data needed to be written for each vote cast (sent via a CGI to update a server file) and the results fetched to display results. The code (both Lingo and Perl) is very crude, but it's still operational at:

http://www.mcli.dist. maricopa.edu/alan/nojava/poll/
Print this! Have you clicked the browser print button for a shockwave page? You get a blank square. Some folks have gone the route of installing a printing Xtra, but this seems like ultra risky ground to assume the user can get it installed in the correct obscure folder. (We even doubt their ability/willingness to get the plug-in!) Until Macromedia comes up with a method for embedding the Xtras at shockwave run-time, it's "No Printsville". But with a CGI script, you can generate HTML content in a second browser window that the users can then print using the browser's functionality. You might use this to print a list of those game high scores, or perhaps to print the results of an online quiz. For quite a few years I've requested some shockwave net Lingo for
```
      printHTML("string", "target")
```
that would send the string to a targeted frame or window. But I have no influence on the inner engineering sanctum. If you are daring thrill seeker, you can try to do this via externalEvent and JavaScript/VBScript, but this is the hemmoraging edge.
Reliably targeting HTML frames. A common use for Shockwave is as a site navigation control that sits in one frame of a web page and shuttles different HTML file content to other frames (this is now more of a job for Flash). With the flurry of plug-in and browser releases last summer, it was a real crap shot knowing if this would ever work in Internet Explorer. You might get the intended content in a new window or even as a second launched instance of IE. Rather than face even more hair loss, I have sometimes used shockwave/CGI to handle the targeting of frames.
Shockwave generated e-mail. If you are risky you can try doing a gotoNetPage with a "mailto:" URL, but this is downtown Risksville. But you can send the intended email destination and content to a CGI script, which can route it through a server's sendmail facility. I've written a few of these for clients, using a format where they can write a server-side template text file with place holders for the variables (username, email, game score, etc) that come from shockwave. This way, they can edit the content of the email message without digging into the perl scripts. Also, Detlef Beyer has written a great (and free) package of a perl script and a Director 6 demo for accomplishing the same feat.

There are most likely other areas that would call for shockwave needing to talk to CGI (i.e. communicating with other processes/systems on a server), but these are the areas I've had experiences.

"Approaches for Achieving Shockwave / CGI Nirvana"

LET'S SAY YOU ARE FORTUNATE to have a stable web server that successfuly runs CGI scripts from ordinary web forms. Gross assumption? Let's go farther and say that you also have at your beck and call a versatile programmer who can write Perl scripts for you. Have we jumped off the cliff of reality?

Let's say we are creating a shockwave shopping app, where at some point I will enter my name, select something to purchase, and naively provide my credit card number. I will be thus sending three pieces of information. The server script needs to register my order in a server file and return a web page with a receipt for my order.

The starting point for you, the Lingo Jock, is knowing the limits of shockwave talking to CGI. First, there is the limitation of shockwave being capable of sending CGI calls via the GET method and not by POST. What's the diff? In a GET method, you send the content (pairs of variables and values from the names of HTML form elements and values of form fields or menu selections) as one long continuous string appended to the URL for the CGI script:


<cgi-url>?var1=var1_value&var2=var2_value¬
  &var3=var3_value

or something for my little shopping program like


http://www.biff.com/cgi-bin/order_proc.pl?name=¬
  Alex+Zavatone&item=inflatable+toy&¬
  cardnum=123456789

Note how there are three variables and values strung together by "=", "&" and "+" characters.

When you call a CGI script by the POST method, all of the information is sent in a more private manner, and is not sent with the URL (Honestly I am not sure exactly what happens, but I loose no sleep). You can send more data via POST (i.e. long text field input)... but you cannot do a POST method from shockwave. Don't ask me why, call Macromedia. Those nifty folks at Human Code (makers of the XtraNet) have beenworking on a new Xtra that allows POST method calls. Cool, but that's more edgy stuff.

So here's the snuff on CGI from shockwave; there is a limit on how much data can be sent, appended to the script's URL. I've understood from those that know more than me that shockwave from NetScape can append up to a 4k string and Interent Explorer is limited to 2k of text. I've also heard rumblings of a 1k limit. (Help me, someone). The point is you cannot easily send very long text strings (well, actually you can through some perl hijinks of breaking up the call into several smaller strings, writing content to temp files, and having some flag to signal when the last chunk has been sent).

From the shockwave side, you have to construct a long string that makes up the string sent to the CGI script. More on that soon.

The other consideration is knowing what kind of data you want returned. If you want to send some data to a CGI and have it generate a web page in response, you will want to call it from Lingo using


  gotoNetPage cgi-url?variable_name_value_string

Of course, this will zap you right out of the shockwave page, so you often will use the Lingo option to target the result to either a named HTML frame or the name for a second browser window:


gotoNetPage cgi-url?variable_name_value_string ¬
  , target_name

However, for many other actions, you want the CGI script to perform some function (i.e. calculating results based upon input, or writing data to a web server file), so you may want the CGI script to return nothing, or perhaps an "OK" string, or some other text response. In these cases, the route is to call the CGI script via:


  getNetText cgi-url?variable_name_value_string

and checking the results using the value of the netTextResult() function (and do the right stuff covered elsewhere on properly checking netDone, or at least using the D6 Behavior).

So your primary step is knowing up-front:

what data will be sent from shockwave
what you want the server script to do with that data (append to files, generate e-mail, perform some data manipulation)
what the returned data should be (web page HTML or text string)

For my fake example, I would tell my programmer what server file needs to be appended with the data and what the format is (i.e. tab delimited, one order per line), and provide an HTML file showing what the returned page should look like. Before doing anything in shockwave, I create a plain HTML web page form so I can make sure the CGI script does what it should. And this will also let my programmer know what the names of the input variables are (just the names of each form element):

name:<INPUT TYPE="text" NAME="name" SIZE=40><br>

item to buy:
<SELECT NAME="item">
<OPTION>Motor Oil

<OPTION>Inflatable Toy
<OPTION>Spaghetti O's
<OPTION>Hair Tonic
</SELECT>
<p>

credit card number

Or this is how it looks (go ahead and click, there is no such script!):

I can create whatever interface I like in shockwave, as long as I end of with my three pieces of data. Once I'm ready to call the CGI from Lingo, I need to do some massaging to assemble an appropriate CGI string. Let's say in Lingo I have the following variables


buyers_name     -- a string entered in an 
                -- editable field member
shopping_list   -- a string representing 
                -- an item chosen from
                -- perhaps a menu
creditCardNum   -- a string entered in 
                -- an editable field member

Now because there is a limiting factor on that string I need to send to the CGI script, sometimes I may break my Lingo habits and use cryptically short variable names for the CGI program (sometimes just one letter variables, "x", "n", etc). With our example, however, we are only sending three variables and none will be unduly long.

To construct our string we'll need to concatenate a string with the CGI URL, a "?" and the variable name=value pairs, each separated by a "&".

One of the most frequent mistakes I hear of is that people will write their call:


gotoNetPage "http://www.biff.com/cgi-bin/order_proc.pl?¬
  name=buyers_name&item=shopping_list¬
  &card_num=creditCardNum"

What's the problem? We are sending the names of each Lingo variable and not the value. What I need to do is a little more string operating in Lingo:


set cgiString = "http://www.biff.com/cgi-bin/order_proc.pl?"
put "name=" & buyers_name after cgiString
put "&item=" & shopping_list after cgiString
put "card_num=" & creditCardNum after cgiString  
gotoNetPage cgiString

There's one more hitch. Note that a CGI string does not contain and blank spaces. Plus, there are a few other characters that are not allowed (quotes, question marks, ampersands) but have hexadecimal substitutes. So you have to run your string variables through a "cleansing" function before sending them to a CGI. The following code was posted to Direct-L by Peter Fierlinger and I've used it several times:


  
on urlEncode inputString
  -- encodes the string into 
  -- appropriate characters for CGI
  set encodedString to EMPTY
    
  -- just in case we get a non-string
  -- input, we better convert it
  if not StringP( inputString ) then set ¬
    inputString = string( inputString )
    
  repeat with i = 1 to the length of inputString
    set theChar = char i of inputString
      
    -- check for char=acters to parse
    if chartoNum( theChar ) < 33 OR chartoNum¬
     (theChar) > 127 OR ("+=&%") contains ¬
     theChar then      
      put Dec2Hex(chartoNum(theChar)) ¬
       after encodedString
    else
      put theChar after encodedString
    end if   
  end repeat
    
  return encodedString
end
  
on dec2Hex decNum
-- decimal to hexidecimal conversion
  set digit1 = decNum/16
  set digit2 = decNum - (digit1 * 16)
    
  case digit1 of
    10: set digit1 to "A"
    11: set digit1 to "B"
    12: set digit1 to "C"
    13: set digit1 to "D"
    14: set digit1 to "E"
    15: set digit1 to "F"
  end case

  case digit2 of
    10: set digit2 to "A"
    11: set digit2 to "B"
    12: set digit2 to "C"
    13: set digit2 to "D"
    14:set digit2 to "E"
    15: set digit2 to "F"
  end case
    
  return "%" & digit1 & digit2
 end

So I modify my little script fragment above to read:


set cgiString = "http://www.biff.com/cgi-bin/¬ 
  order_proc.pl?"
put "name=" & urlEncode(buyers_name ) ¬
  after cgiString
put "&item=" & urlEncode( shopping_list )¬
  after cgiString
put "card_num=" & urlEncode( creditCardNum ) ¬
  after cgiString
gotoNetPage cgiString

"How about Some Fresh Cache?"

WHEN I WAS TOYING AROUND with my shockwave poll, I ran into a problem where if I made a second call to a CGI from shockwave with the same paramters, rather than sending a new call to the server, it would just retrieve the first call's results from the browser cache and not communicate the server.

There are some cryptic methods in the Director 6 plug-ins to force a server call, but I've found a reliable alternative that has worked since Director 4. The key is making sure you always have a unique CGI string sent from shockwave so it won't find that URL in the cache. I do this by appending another variable to my cgiString, one that is never used by my CGI script:


put "randy=" & the ticks after cgiString
gotoNetPage cgiString

It's pretty safe to assume that "randy" will have the same value, and thus the cgiString variable will always be unique. Voila! Fresh calls, no cache calls.

"Wanna See Some Perl?"

YOU ASKED FOR IT. I worked long on a shockwave application that at the end, the user takes a test, and if they scored well, they would see a Diploma on their shockwave screen that showed their name, the date, and their score. We needed a way to print the diploma, so the only way was to spawn an external browser window and create an HTML formatted page that had the same graphics of the diploma as well as inserting the user name and score as sent from shockwave. To print this, I needed some shockwave to CGI.

All my shockwave code had to do was to build a CGI string that contained the URLencoded string for the name and score.


on printDiploma 
  global gUserName 
  -- users name as entered from 
  -- entry screen global gUserScore 
  -- score on last atempt at test 
  set cgiString = "http://www.mcli.dist.maricopa.edu ¬
   /cgi-bin/nru_diploma.pl?" 
  put "name=" & urlEncode( gUserName ) after cgiString 
  put "&score=" & gUserScore after cgiString 
  gotoNetPage cgiString, "diplo"
end

This perl CGI was called via gotoNetPage so it would pop up in a new browser window.

#!/usr/local/bin/perl
# nru_diploma.pl

# Load CGI.pm library to handle input variables
use CGI qw(:standard :cgi-lib);

# convert the CGI variables to perl variables
&ReadParse(*in);

# call the library function to generate the
# proper return header (let's the
# browser know content is HTML
print header;

# call unix function for system date
$date_string = 'date +"%B %e, %Y"';

# Now just print out the HTML page and insert the two
# variable we got from the CGi call

print <<EOF;
<HTML>
<HEAD>

<TITLE>NRU Diploma</TITLE>
</HEAD>
<BODY
bgcolor=#FFFFFF>
<center>
<img src="http://www.mcli.dist.maricopa.edu/proj/nru/nru_vl/images/diploma_top.gif"
alt="Negative Reinforcement University grants a degree to"
width="500"
height="121"><br>

# here is where we print the name
<font size=+4>$in{'name'}</font><br>

<img src="http://www.mcli.dist.maricopa.edu/proj/nru/nru_vl/images/diploma_mid.gif"
alt="Having scored on the final exam" width="500"
eight="19"><br>

# and here is where we print the score
<font size=+2>$in{'score'} / 10</font><br>
<img src="http://www.mcli.dist.maricopa.edu/proj/nru/nru_vl/images/diploma_bot.gif"
alt="on this date"
width="500"
height="108"><br>

# and this is where the dat goes
<font
size=+2>$dateString</font><br>
<img src="http://www.mcli.dist.maricopa.edu/proj/nru/nru_vl/images/diploma_seal.gif"
alt="NRU Seal"
width="500"
height="87">
<p>
<hr size=1 noshade>
<font size=2 color=#666666>negative reinforcement university<br>
http://www.mcli.dist.maricopa.edu/proj/nru/
</font>

</BODY>
</HTML>
EOF

"So Now What? "

This article may not give you a 1-2-3 formula for using CGI from shockwave. I've tried to make the case that CGI is reliable (much more so than shockwave-browser communication) for tasks that shockwave cannot do, especially in storing data on a web server. It works best when the amount of data sent to the server is fairly limited. If you can be clear describing the nature of the input, functionality, and expected output, a perl coder can write your scripts pretty easily.

Alan Levine is an Instructional Technologist at the Maricopa Center for Learning & Instruction at the Maricopa Community Colleges. Among other random things, he tries to maintain the DirectorWeb, does side work as dommy media, and not often enough, mountainbikes in the desert.