Zope, the Python-based Application Server of Tomorrow

Stephan Richter<srichter@cbu.edu> (and Maik Roeder)
http://www.zope.org/

Abstract
Only in the last two years, Web Application Servers like BEA WebLogic, Persistent PowerTier, Lotus Domino and Netscape Application Server became common development environments for the Internet. Is Zope the Python-based Application Server just another Web Application Server? Far off, the unique mix of key concepts, flexible architecture, which allows for simple Plug & Play, and the impressive amount of components, as well as the fact that Zope is Open Source make the difference.

Zope historically

Let us go back and talk a little bit about the origin and development of Zope. Digital Creations, which currently publish Zope, developed originally three products, called Bobo, Principia and Aqueduct, all developed in Python using some C extensions for better performance. In 1996, shortly after the International Python Conference (IPC), Jim Foulton hacked the Zope Core in the airplane on his way home. In 1998, Hadar Pedhazur, a well-known Venture Capitalist, convinced Digital Creations to open up their commercial products and publish them as Opene Source under the name Zope. Zope stands for the "Z Object Publishing Environment". The first Zope release was 1.7 in December 1998. Paul Everitt, former CEO of Digital Creations, and all the other people at Digital Creations converted from a product company into a successful consultant firm. Alone the story how Digital Creations went from a proprietary, product- to a service-based company is very interesting, but would be out of the scope of this article.

"Teamwork" in Zope

Zope does not save its content (read files) in the file-system like most other Web technologies, such as it is common for Active Server Pages (ASP), Hypertext Preprocessor (PHP), Java Server Pages (JSP, based on Servlets) and pure CGI scripts, but in a object database called ZODB. The ZODB is always communicating with the Zope Core, which regulates the information processing and enforces the security model. Therefore, the Zope Core is responsible for the communication with external data sources, such as relational databases and the interpretation of data in combination with other protocols, like XML-RPC or FTP. The ZServer is Zope's own Web server, which prepares the incoming HTTP requests before forwarding it to the Zope Core.

Zope Architecture

Figure 1: The Zope architecture

The classes of the Zope Core also provide a very nice API, so that Zope can be easily extended using ZClasses and Python Products. ZClasses are exclusively managed using the Web based Management Interface (which is discussed later in the article) and the user can very quickly define its own objects including everything one needs, like attributes, methods and inherited classes as well as defining the security parameters of the object instances. Since ZClasses can be created quickly, they are perfect for Rapid Prototyping. A Web developer, who only uses ZClasses, can create complete Web sites like http://zdp.zope.org without the need of any real programming language. Zope's Document Template Markup Language (DTML) is usually enough to create simple Web Sites and the developer only needs to enter the Python level to implement complicated things.

Extending Zope using Products

Of course, ZClasses have their limits and are not for every project useful. In many cases Python Products are used, that are developed in Python as the name suggests. There are only very few situations in which C extensions are required, which can then be integrated using a Python interface. For example, you need only one base class and you are ready to develop your own Database Adapter, define a new Authentication source, write a DTML extension, implement a new protocol or just simply make a customized Zope data object. It is also very straightforward to integrate other Python modules like the famous Python Imaging Library (PIL). There is in fact already a Poll product for Zope, which uses PIL to generate statistical graphs. We use Python Products in our backend coding, to utilize many Python modules like PIL or Mailman, so that a user of the Web Sites is able to register for a mailing list. Using the COMObject Product one can even search through Microsoft Office Documents, for which again you will only need Zope and Python. The only disadvantage of Zope is the API is generally not well documented, but the Zope community (Zope Documentation Project) and Digital Creations are working hard to provide better documentation. There is now already a Zope book under the Open Content License, which is the official Zope documentation.

Zope and relational databases

Zope can connect to many relational databases using a couple of mouse clicks. Some of these databases are directly supported by Digital Creations, such as the Oracle, Sybase, Gadfly and ODBC Database Adapter. A specialty is the in Python implemented database Gadfly, which is great for learning and Rapid Prototyping, but should not be used on a production server. Through the community there are third-party Database Adapters for PostGreSQL, MySQL, SAP DB, Informix, Microsoft Access and DBMaker.

Zope's database communication was designed to make the switch from one database to another very simple by exchanging the database connection object. Let's say that we have an Oracle DB connection object called 'myDBConnection' and several ZSQL Methods (another Python Product), which are making use of the connection. A ZSQL Method contains SQL statements that are executed using the DB connection. Now one wants to change to another database. This is simply done by deleting the old connection and to add a new one for any of the supported databases. As long as the new connection object has the same object id as the old one (it is assumed that the SQL code is compatible with the new database), you can switch your complete site to another database in 5 Minutes.

Another feature that should be mentioned in combination with relational databases, are Pluggable Brains, which allow for a fast integration of relational-to-object (R2O) mapping. In fact, R2O mapping is usually assumed to be the most time consuming step during a development cycle. But with Pluggable Brains one defines a Python class that is associated to a specific ZSQL Method. For every line that is returned from the database, an instance of this class is initialized. Therefore the total result object is a list of these class instances. Now, a user does not have to deal with simple lists of attributes, but is able to use the advantages of object orientation. In principle the user simply defines and implements a class and is already done, since Zope takes care of the R2O mapping.

Furthermore, Zope is able to communicate via common Internet protocols like FTP, XML-RPC and WebDAV. Let's take for example an FTP capable editor such as Emacs; we are now able to edit objects like DTML Methods and Documents using this editor, instead of using the Content Management Interface. This feature is very useful for editing large pieces of HTML. The FTP access is also very useful for graphic designers, which would like to upload several graphics at a time. Instead of tediously uploading one file at a time using the browser, all the files can be uploaded at once using an FTP client; one mouse click and they are appearing in the respective Zope folder.

And what about the security?

Another great advantage of Zope, is the integrated security management, which controls the user's rights and what the user is allowed to do, and is divided into three sections: Permissions, Roles and User. Permissions are actions like add, edit, delete and view. The developer can create permissions for any object and assign methods to these permissions. Every role is connected to a list of permissions. These mappings are done using the Role Manager in which permissions to manage or view any object are assigned to roles. For example a user has only access to the parts of the Content Management Interface he is allowed to see/manage. Finally, users can have many roles. Using this model, every possible security setting is simply and quickly done. Recently the first implementations of user groups appeared, but are not supported by all authentication products.

Zope offers different kinds of user authentication mechanisms. On one hand, there is the standard User Folder, which simply saves the user in the ZODB; on the other hand you have various authentication products, which can save users in a relational database or get user information from LDAP, SMB or even /etc/passwd. Thanks to these Products and the Management Interface, it usually does not take longer than an hour to setup the security model of a larger site.

Solution for Object Management

The Zope Management Interface was mentioned several times in this article, but what exactly is it? The Zope Content Management Interface is a Web-enabled system that allows the developer to manage Zope Web sites, including all objects, HTML pages, security settings and ZClasses. Python Products are the only part of Zope that cannot be changed using the Zope Management Interface. Because of ZODB's natural support for transactions, it is possible to create private versions of objects. This way changes can be made to a live site without interrupting its functionality on the outside. Only after all changes were made and the correct functioning of the new code is guaranteed, the version is committed and the site is updated. As long as the ZODB was not packed, it is also possible to Undo a transaction. But what would be a Content Management System without a search feature? The user interface offers an excellent object search that allows to specify many parameters, which make the search as exact as possible.

Another nice feature of the Management Interface is the possibility to export any object in the native Pickle or XML format. This allows to quickly move objects from one ZODB to another. Since the ZODB is platform independent, it is possible to move it to another Zope installation by copying one file. This can be extremely useful, if a company wants to move its servers from Sun Solaris to Linux.

Currently a non-web-based Zope Management Interface is developed for the Boa Constructor Python IDE, called ZOA. It will allow to execute management tasks directly from Boa. Boa Constructor itself is based on wxPython, so that all platforms will be able to enjoy this new IDE.

Features

Now that we introduced all these features, we have to ask the question for what do we need all that? To answer this question, we should recall some of the common Internet project requirements. One requirement, we will all agree on is TIME! Additionally, we may have a small budget as well as the need to incorporate external relational databases and the need to use existing authentication mechanisms. Of course, our application server should additionally provide all the features other application servers are offering. and we already have a small problem. But for most of these requirements Zope is the perfect tool.

There are many out-of-the-box products, which allow the developer to easily cover most of the site's functionality, so that the developer can concentrate mainly on the site-specific features. For example, you got many products that provide complete solutions for calendaring, news articles, message boards, polls, guest books, chat rooms and bug collectors. Even if you end up not using any of these products, it is always good to see other people's code to get ideas for ones own development. Since all products are Open Source, this is even wanted.

As already mentioned, usually a lot of time is required to implement security and login information for Web-based projects. Since Zope saves its content in the ZODB, it implements its own security mechanism, which is specifically designed for Web requirements. Therefore it is possible to create a security model with a couple of mouse clicks, instead of sacrificing several days of development time.

Many clients ask Web consultants whether it is possible to integrate their current database system into the new Web site. Usually that is possible, but requires much time, since many components have to be written. Zope offers all general communication tools, which are required to solve the issue, in order to provide the developer a stable and tested environment that offers many specific components. Recently, Stephan had to integrate the login mechanism of another site in his own site. It was only possible to send and receive certain XML strings with a very specific syntax - but it was possible to integrate completely into their system in only eight hours. At the moment, the system uses a Python module that parses the XML, but in the next version every thing will be solved using XML-RPC. This shows the great flexibility, which comes with Zope and Python, and time can be saved.

Some of the recent new Products include the Content Management Framework 1.0, a tool to quickly create community-based Web sites, Squishdot 1.1, a Zope-based SlashCode clone, and Proiektor, a groupware application with an integrated project management system. There are many more cool products; you can find them all on http://www.zope.org/.

As one can see, it is easy to save development time using Zope, due to the countless features and many diverse products. One thing we all value: everything is handled very cleanly. The implementation of a site does not appear to be a hack, which was often the case with CGI scripts. One must not even be efficient using a programming language and can still implement many things; if required, Python can be picked up very quickly, if you want to write your own Python Products. There will be soon the possibility to use Perl in Zope as well.

Installation

After you have already read so far, you might want to see a small example for a better understanding. First, you should go to the Zope Homepage http://www.zope.org and download the latest Zope version for your platform and install it. Zope runs almost on any platform on which Python can run on. Download the latest Zope version. Don't be afraid, it will not take hours, since the file should not be larger than 4 MB.

There are two Zope versions with Python precompiled for Linux and Windows - using these files there should be absolutely no problem. We will briefly describe the installation of version 2.3.1 under Linux. The complete installation instructions can be found in the /doc/INSTALL.txt file. Now we will untar the downloaded file in /usr/local using the command: tar xofvz Zope-2.3.1-linux2-x86.tgz. Then we use cd Zope-2.3.1-linux2-x86 to enter the new directory, and call the command ./install with which Zope is installed. Please take note of the username and password at the end, since we will need it alter on. It maybe required to use the recursive chown command to make the entire directory structure consistently owned a by one user. Finally, you start Zope using ./start. It takes a short moment till Zope is ready. Then you enter the address http://localhost:8080/ in your browser and the Zope welcome text will appear.

'Hello World!' with Zope

Let us begin with the famous "Hello World!" example. After you installed Zope, you can use the superuser account (the ones you just wrote down) to log in. Next, you click on acl_users in the left frame of the screen and add another user having the Owner and Manager roles. For security reasons, the superuser cannot own any objects, therefore you will have to close the browser or use the new logout feature! After the browser was restarted, you should use the new username and password to log in into the system. Now we create a new folder (equivalent to a directory) by choosing FOLDER from the drop-down list in the upper part of the right frame. In the next screen you enter the ID Test and submit the request using the ADD button. Now we enter the folder TEST and add a DTML Document named hello and add the following content:

<dtml-var standard_html_header> <h1>Hello World!</h1> <dtml-var standard_html_footer>

Now we can view the new object by calling the URL: http://localhost:8080/Test/hello.

When you edited the content of the DTML Document you may have asked yourself, what the dtml-var is used for. This is a command of the Document Template Markup Language (DTML) which is used by Zope to publish object information. dtml-var's responsibility is to display the value/content of an object or variable. In our case the content of the DTML Methods standard_html_header and standard_html_footer is added at the locations.

The Document Template Markup Language, short DTML

DTML was originally designed to present the object's data and the results of executed business logic in Zope. Today however, it is used for many other tasks and an often large part of the core business logic is implemented using the DTML. DTML is an extension to Zope, which allows to embed programmatic parts to your HTML code, similar to Cold Fusion's CFML or even PHP. There are only a few important tags like DTML-VAR, DTML-IF, DTML-IN, DTML-LET, DTML-CALL, DTML-TRY, DTML-RAISE, DTML-TREE, DTML-SENDMAIL and DTML-COMMENT, some of which will be introduced here in some more detail. Some may say that the DTML- as a prefix is too long; in this case there is a product which changes the prefixes from 'DTML-' to '%' or '?'. Furthermore it is very simple to implement a new custom tag.

The most important tag in DTML-VAR. It allows to directly insert information (in form of a string) into your HTML code. This method allows not only for inserting data, but also to factor your site into many components, such as headers, footers, menu and main content for example. One can easily see that reusing HTML/DTML code is very trivial this way. To support more flexible ways of representing the data, the tag also supports a long list of attribute. A few commonly-used attributes are listed in the table of Figure 3 (you can also see some examples of the usage of the tag there as well):

Attribute Description Example/Usage
missing Provide a value to be used if the variable is missing (meaning it does not exist in the current namespace). <dtml-var tasks missing="No tasks for today.">
fmt Specify a data format for your object's data. The format can be Special, C-Style, strftime or Custom. Based on the type of the resulting output, the format attribute expects some different fmt values. Special Format:

<dtml-var "2" fmt=dollars-and-cents> Will display: $2.00

C-Style Format: <dtml-var "120.00" fmt="%E"> Will display: 1.200000E+02

null Specify a string to be substituted for null values..

Will put NA if the price variable is None (or null).

capitalize Convert the first letter of the inserted text to upper case.

<dtml-var "'once upon a time…'" capitalize< Will display: Once upon a time

html_quote

Convert characters that have special meanings in HTML, such as <, >.

<dtml-var "'<A> <B> <C>'" html_quote>

Will display: <A> <B> <C>. If you do not use this attribute, nothing will be displayed.

Figure 3: Commonly used attributes for the DTML-VAR tag

Since this tag is used so often, there is also a shortcut for it called the entity syntax: &dtml.attribute-varName; This form is often used when generating URLs or other parts of HTML tags, so that the code stays clean. An example looks like this: &dtml.missing-firstName; Using the 'missing' attribute, Zope will not report an error and abort when 'firstName' is not found, but ignore it and insert simply nothing.

Another very useful tag for reporting is DTML-IF, since it lets us evaluate conditions (as the name suggests). In this case some basic Python knowledge (even though not necessary!) may be of advantage, since the condition is in Python and also the behavior of the DTML-IF tag and its complementary tags DTML-ELIF, DTML-ELSE and DTML-UNLESS reflect the Python behaviors. Since a simple example will do more than a thousand words, here a small snippet, demonstrating the behavior:

<dtml-if "grade >= 90" > Your final grade was an A. <dtml-elif "grade >= 80" > Your final grade was a B. <dtml-elif "grade >= 70" > Your final grade was a C. <dtml-else> Sorry, you failed. </dtml-if>

I should note that the grade was defined somewhere as an integer between 100 and 0. Again, the code in the quotes of the tags, is a simple Python expression which returns true or false.

The last tag, that is falling in the top three is the DTML-IN tag, which simply lets you iterate through a list of any type of objects (basically a typical scripting loop). For example, if you would like to display the grades of all the students in a class, you could use the following snippet:

<ul>
<dtml-in students>
<li><dtml-var name>: <dtml-var grade></li>
</dtml-in>
</ul>

In this case 'students' is a list of Student objects which have the attributes 'name' and 'grade'. Furthermore, the DTML-IN tag supports a great amount of attributes, which allow you to display lists in batches and therefore make the list output more user-friendly and useful. Some of the tags are listed in the following figure:

Attribute Description Example/Usage
Sort Causes a sequence of objects to be sorted by the specified field, before insertion. <dtml-in AddressBook sort="name">
<li>Contact: &dtml-name;, &dtml-phone;</li>

Will display entries sorted by the name.

Size The batch size. <dtml-in AddressBook size=10>
<li>Contact: &dtml-Name;, &dtml-phone;</li>
</dtml-in>

Will display 10 entries at one time.

Orphan The desired minimum batch size. <dtml-in AddressBook size=10 orphan=5>
<li>Contact: &dtml-Name;, &dtml-phone;</li>
</dtml-in>

(If your address book contains 14 items, all 14 items will be displayed on one page. If you address book contains 15 items, the first 10 will be displayed on the first page, the remaining 5 will be on the next page.)

Previous Iterative Insertion will not be performed. The text following the in tag will be inserted and variables associated with information about a previous batch will be available.

This attribute was created only to allow you to easily link the current batch display with the previous one (if one exists). This is why, if this attribute is specified, the actual items of the sequence are not inserted.

<dtml-in AddressBook size=10 orphan=5 previous>
<a href="&dtml-URL;&dtml-sequence-query; query_start=&dtml-previous-sequence-startnumber;">
(Previous <dtml-var previous-sequence-size> results)
</a>
</dtml-in>
Next Iterative Insertion will not be performed. The text following the in tag will be inserted and variables associated with information about the next batch will be available.

Similar example description as above.

<dtml-in AddressBook size=10 orphan=5 next>
<a href="&dtml-URL;&dtml-sequence-query; query_start=&dtml-next-sequence-startnumber;">
(Next <&dtml-next-sequence-size;> results)
</a>
</dtml-in>

Figure 4: Commonly used attributes for DTML-IN

More on all the DTML tags can be found in Chapter 4 of the online Open-Content Zope Book at http://www.zope.org/Members/michel/ZB/DTML.dtml.

Acquisition, an Important Concept

But where are these two DTML Methods defined? You did certainly not create them in the TEST Folder. In reality, they are in the root folder, which Zope sees from the TEST Folder. This behavior is called Acquisition. This allows the system to look up in the tree to find objects and attributes, if they were not defined locally. This is a unique feature of Zope, since it allows making a setting only once and the entire sub-tree will "inherit" this attribute. These attributes and objects can change and be "overloaded" and are updated everywhere immediately.

Therefore you can see the DTML Document hello from a new folder TEST2 (that lies inside the TEST folder). The URL would look something like that: http://localhost:8080/Test/Test2/hello, but the output remains the same. Acquisition first looks into TEST2, then in TEST for HELLO. Even though this example may seem trivial, acquisition is a very powerful mechanism about which deserves its own article...

Fig. 2: The representation of a typical Zope Object Tree.

Zope for Enterprises

Zope also responds to some of the issues that raised again and again. In the corporate world it is necessary to be able to distribute ones data and processes over several machines at many different locations. For this Zope uses the Zope Enterprise Option which allows you to generate a network of Zope Client Storages which get their data from many Zope Storage Servers. This allows Zope to:

  1. Scale well, since now the resource requirements are distributed over several machines (by the way, the scaling is almost linearly, like other solutions)
  2. Load balance and to be high highly available, since you can have several Zope processes and the data storage is separated from the process itself, which adds data integrity.
  3. Distribute the website in multiple locations and configurations.

This way one can customize the storages, processes and configurations of their individual Zope installations as they wish.

Zope's Future is the Community

Even though Zope comes from the US, it is widely accepted as a Web application server in Europe. One example is http://www.linux-community.de, which uses Zope's full potential and was developed by Tom Schwaller, a Zope enthusiast. In July 2000, Zope was present at Linuxtag in Stuttgart, having a booth and a "ZopeCon" which had many events, such as very interesting presentations and tutorials given by Zope developers to which many Zope users from all over Europe came. The great feedback from the ZopeCon gave us the idea to create an initiative called EuroZope, a community that would represent Zope at all large exhibitions and events in Europe. In August there was another meeting with Paul Everitt, the former CEO of Digital Creations, in Paris, which also was a big success. Since then, EuroZope organized several other meetings and conferences and created now this magazine in collaboration with Linux Enterprise. Pictures and reports about the events are found on the EuroZope site under http://zdp.zope.org/projects/eurozope.

Zope grows quickly. It has an Open Source License and was cleanly developed. Because of that - and with the support of the enthusiastic Zope community - it surely has a good chance to become a widespread development platform for many Intranet and Internet sites.


About the author: Stephan Richter finished his 8th semester at Christian Brothers University in Physics and Chemistry. He is using Zope since August 1999 and worked for Digital Creations last summer writing Documentation. He is also one of the founders of the EuroZope Community and will be at Linuxtag 2001, at which EuroZope will have its 2nd annual ZopeCon.