Application Development¶

Webware provides Python classes for generating dynamic content from a web-based, server-side application. It is a significantly more powerful alternative to CGI scripts for application-oriented development.

In this chapter we explain some more fundamental concepts of Webware for Python and describe best practices for developing a web application using Webware.

Core Concepts¶

The core concepts of Webware for Python are the Application, Servlet, Request, Response and Transaction, for which there are one or more Python classes.

The application resides on the server-side and manages incoming requests in order to deliver them to servlets which then produce responses that get sent back to the client. A transaction is a simple container object that holds references to all of these objects and is accessible to all of them.

Content is normally served in HTML format over an HTTP connection. However, applications can provide other forms of content and the framework is designed to allow new classes for supporting protocols other than HTTP.

In order to connect the web server and the application, Webware for Python 3 uses the Web Server Gateway Interface (WSGI). The Webware Application instance serves as the WSGI callable. The web server calls the Application, passing a dictionary containing CGI-style environment variables for every request. The Application then then processes the request and sends the response back to the web server, for which WSGI provides to different mechanisms. By default, Webware applications use the write() callable mechanism, because this is more suitable for the way Webware for Python applications create responses, by writing to an output stream. However, since not all WSGI servers support this mechanism, it is also possible to use the more usual WSGI mechanism of passing the response as an iterable. You will need to switch the Application.config setting WSGIWrite to False in order to use this mechanism.

Many different WSGI servers are available that can be used with Webware for Python 3. By default, Webware uses the waitress WSGI server as its development server. If you have installed Webware with the “development” or “test” extra, as recommended in the chapter on Installation, the waitress server should already be installed together with Webware for Python and will be used when running the webware serve command.

In production, you may want to use a web server with better performance. In the chapter on Deployment we describe how you can configure a web server like Apache to serve static files directly, while passing dynamic contents to the Webware application via WSGI, using the mod_wsgi module.

The whole process of serving a page with Webware for Python then looks like this:

A user requests a web page by typing a URL or submitting a form.
The user’s browser sends the request to the remote Apache web server.
The Apache web server passes the request to a mod_wsgi daemon process.
The mod_wsgi daemon process collects information about the request and sends it to the Webware Application using the WSGI protocol.
The Webware Application dispatch the raw request.
The application instantiates an HTTPRequest object and asks the appropriate Servlet (as determined by examining the URL) to process it.
The servlet generates content into a given HTTPResponse object, whose content is then sent back via WSGI to the mod_wsgi daemon process.
The mod_wsgi daemon process sends the content through the web server and ultimately to the user’s web browser.

Setting up your application¶

The first task in developing an application is to set up the file structure in which you will be working.

It is possible to put your application in a subdirectory at the path where the webware package is installed and change Configs/Application.config to add another context. But do not do this. Your application will be entwined with the Webware installation, making it difficult to upgrade Webware, and difficult to identify your own files from Webware files.

Creating a working directory¶

Instead you should use the webware make command to create an application working directory. You should run it like:

webware make -c Context -l Lib WorkDir

This will create a directory WorkDir that will contain a directory structure for your application. The options are:

-c Context:: Use Context as the name for the application default context. A subdirectory with the same name will be created in the work dir (you can change that with the -d option). If you do not use the -c option, the context name will be MyContext. You may simply want to call it App or Context, particularly if you are using only one context. If you want to add more than one context, you need to create a subdirectory and a corresponding Contexts dictionary entry in the Application.config file manually.
-l Lib:: Create a Lib directory in the work dir which will be added to the Python module search path. You can use the -l option multiple times; and you can also add already existent library directories outside of the work dir. If you want to add the work dir itself to the Python path, pass -l .. In that case, you can import from any Python package placed directly in the working, including the Webware contexts. Note that the webware package will always be added to the Python module search path, so that you can and should import any Webware modules and sub packages directly from the top level.
WorkDir:: The files will be put here. Name if after your application, place it where it is convenient for you. It makes sense to put the working directory together with the virtual environment where you installed Webware for Python inside the same distinct base directory. Install any other requirements either into the virtual environment or provide them in one of the directories specified with the -l option.

You can see all available options if you run webware make --help.

When you run the webware make command with the options describe above, the following directory structure will be created inside the WorkDir directory:

Cache/          Context/        ErrorMsgs/      Logs/           Sessions/
Configs/        error404.html   Lib/            Scripts/        Static/

Here’s what the files and directories are for:

Cache:: A directory containing cache files. You won’t need to look in here.
Configs:: Configuration files for the application. These files are copied from the Configs subdirectory in the webware package, but are specific to this application.
Context:: The directory for your default context. This is where you put your servlets. You can change its name and location with the -c and -d options. You can also change this subsequently in the Application.config file in the Configs directory, where you can also configure more than one context. You may also want to remove the other standard contexts that come with Webware from the config file.
error404.html:: The static HTML page to be displayed when a page is not found. You can remove this to display a standard error message, modify the page according to your preferences, or use a custom error servlet instead by setting ErrorPage in the Application.config file appropriately.
ErrorMsgs:: HTML pages for any errors that occur. These can pile up and take up considerable size (even just during development), so you’ll want to purge these every so often.
Lib:: An example for an application-specific library package that can be created -l option (in this case, -l Lib).
Logs:: Logs of accesses.
Scripts:: This directory contains a default WSGI script named WSGIScript.py that can be used to start the development server or connect the Webware application with another WSGI server.
Sessions:: Users sessions. These should be cleaned out automatically, you won’t have to look in this directory.
Static:: This directory can be used as a container for all your static files that are used by your application, but should be served directly via the web server.

Using a Version Control system for Your Application¶

A version control system is a useful tool for managing your application. Currently, Git is the most popular one. These systems handle versioning, but they also make it possible for other people to see snapshots of your progress, for multiple developers to collaborate and work on an application simultaneously, and they create a sort of implicit file share for your project. Even if you are the only developer on an application, a version control system can be very helpful.

The working directory is a good place to start for creating a versioned project. Assuming you’re using Git, you can get started by creating a repository and importing your project into the repository simply by running:

cd WorkDir
git init
git add .
git commit -m "initial import"

Note that a hidden .gitignore file with reasonable defaults has already been created for you in the working directory. It tells Git to ignore files with certain extensions (such as .log or .pyc files), and all the files in certain directories (Cache, ErrorMsgs, Logs, and Sessions).

Structuring your Code¶

Once you’ve got the basic files and directories in place, you’re ready to go in and write some code. Don’t let this document get in the way of developing the application how you choose, but here are some common patterns that have proven useful for Webware applications.

SitePage¶

Subclass a SitePage from Page for your application. This subclass will change some methods and add some new methods. It serves as the basis and as a template for all the pages that follow. If you have added a Lib subdirectory to your working directory as explained above, place the SitePage.py file containing the SitePage class into that directory.

Some code you may wish to include in your SitePage:

Authentication and security
Accessing common objects (e.g., a user object, or a document object)
Page header and footer
Common layout commands, like writeHeader
Database access

You may also want to add other frequently used functions into the SitePage module and then do from SitePage import * in each servlet. You can also put these functions in a dedicated SiteFuncs module, or distribute them in different modules, and import them explicitly, for better code readability and to avoid cluttering your namespace.

Whether you want to use functions or methods is up to you – in many cases methods can be more easily extended or customized later, but sometimes method use can become excessive and create unnecessary dependencies in your code.

A basic framework for your SitePage might be:

from Page import Page


class SitePage(Page):

    def respond(self, trans):
        if self.securePage():
            if not self.session().value('username', False):
                self.respondLogIn()
                return

    def securePage(self):
        """Override this method in your servlets to return True if the
        page should only be accessible to logged-in users -- by default
        pages are publicly viewable"""
        return False

    def respondLogin(self):
        # Here we should deal with logging in...
        pass

Obviously there are a lot of details to add in on your own which are specific to your application and the security and user model you are using.

Configuration¶

There are several configuration parameters through which you can alter how Webware behaves. They are described below, including their default values. Note that you can override the defaults by placing config files in the Configs/ directory. A config file simply contains Python code assigning the settings you wish to override. For example:

SessionStore = 'Memory'
ShowDebugInfoOnErrors =  True

See the chapter on Configuration for more information on settings.

Contexts¶

Webware divides the world into contexts, each of which is a directory with its own files and servlets. Webware will only serve files out of its list of known contexts.

Some of the contexts you will find out of the box are Examples, Documentation and Admin. When viewing either an example or admin page, you will see a sidebar that links to all the contexts.

Another way to look at contexts is a means for “directory partitioning”. If you have two distinct web applications (for example, PythonTutor and DayTrader), you will likely put each of these in their own context. In this configuration, both web applications would be served by the same Application instance. Note that there may be also reasons to run multiple Application instances for serving your web applications. For instance, this would allow you to start and stop them independently, run them under different users to give them different permissions, or partition resources like number of threads individually among the web applications.

Instead of adding your own contexts you may wish to use the webware make command, which will partition your application from the Webware installation.

To add a new context, add to the Contexts dictionary of Application.config. The key is the name of the context as it appears in the URL and the value is the path (absolute or relative to the application working directory). Often the name of the context and the name of the directory will be the same:

'DayTrader': '/All/Web/Apps/DayTrader',

The URL to access DayTrader would then be something like: http://localhost:8080/DayTrader/

The special name default is reserved to specify what context is served when none is specified (as in http://localhost:8080/). Upon installation, this is the Examples context, which is convenient during development since it provides links to all the other contexts.

Note that a context can contain an __init__.py which will be executed when the context is loaded at Application startup. You can put any kind of initialization code you deem appropriate there.

Plug-ins¶

A plug-in is a software component that is loaded by Webware in order to provide additional functionality without necessarily having to modify Webware’s source.

The most infamous plug-in is PSP (Python Server Pages) which ships with Webware.

Plug-ins often provide additional servlet factories, servlet subclasses, examples and documentation. Ultimately, it is the plug-in author’s choice as to what to provide and in what manner.

Technically, plug-ins are Python packages that follow a few simple conventions in order to work with Webware. See the chapter on Plug-ins for information about writing your own.

Sessions¶

Webware provides a Session utility class for storing data on the server side that relates to an individual user’s session with your site. The SessionStore setting determines where the data is stored and can currently be set to Dynamic, File, Memcached, Memory, Redis or Shelve.

Storing to the Dynamic session store is the fastest solution and is the default. This session storage method keeps the most recently used sessions in memory, and moves older sessions to disk periodically. All sessions will be moved to disk when the server is stopped. Note that this storage mechanism cannot be used in a multi-process environment, i.e. when you’re running multiple Applications instances in different processes in production.

There are two settings in Application.config relating to this Dynamic session store. MaxDynamicMemorySessions specifies the maximum number of sessions that can be in memory at any one time. DynamicSessionTimeout specifies after what period of time sessions will be moved from memory to file. (Note: this setting is unrelated to the SessionTimeout setting below. Sessions which are moved to disk by the Dynamic Session store are not deleted). Alternatively to the Dynamic store, you can try out the Shelve session store which stores the sessions in a database file using the Python shelve module.

If you are using more than one Application instance for load-balancing, the Memcached store will be interesting for you. Using the python-memcached interface, it is able to connect to a Memcached system and store all the session data there. This allows user requests to be freely moved from one server to another while keeping their sessions, because they are all connected to the same memcache. Alternatively, using the redis-py client, the application can also store sessions in a Redis database.

All on-disk session information is located in the Sessions subdirectory of the application working directory.

Also, the SessionTimeout setting lets you set the number of minutes of inactivity before a user’s session becomes invalid and is deleted. The default is 60. The Session Timeout value can also be changed dynamically on a per session basis.

Actions¶

Suppose you have a web page with a form and one or more buttons. Normally, when the form is submitted, a method such as Servlet’s respondToPost() or Page’s writeBody(), will be invoked. However, you may find it more useful to bind the button to a specific method of your servlet such as new(), remove() etc. to implement the command, and reserve writeBody() for displaying the page and the form that invokes these methods. Note that your “command methods” can then invoke writeBody() after performing their task.

The action feature of Page let’s you do this. The process goes like this:

Add buttons to your HTML form of type submit and name _action_. For example:

<input name="_action_" type="submit" value="New">
<input name="_action_" type="submit" value="Delete">

Alternately, name the submit button _action_methodName. For example:

<input name="_action_New" type="submit" value="Create New Item">

Add an actions() method to your class to state which actions are valid. (If Webware didn’t force you to do this, someone could potentially submit data that would cause any method of your servlet to be run). For example:
```
def actions(self):
    return SuperClass.actions(self) + ['New', 'Delete']
```
Now you implement your action methods.

The ListBox example shows the use of actions (in Examples/ListBox.py).

Note that if you proceed as in 1., you can add a methodNameForAction() method to your class transforming the value from the submit button (its label) to a valid method name. This will be needed, for instance, if there is a blank in the label on the button. However, usually it’s simpler to proceed as in 2. in such cases.

Naming Conventions¶

Cookies and form values that are named with surrounding underscores (such as _sid_ and _action_) are generally reserved by Webware and various plugins and extensions for their own internal purposes. If you refrain from using surrounding underscores in your own names, then (a) you won’t accidentally clobber an already existing internal name and (b) when new names are introduced by future versions of Webware, they won’t break your application.

Errors and Uncaught Exceptions¶

One of the conveniences provided by Webware is the handling of uncaught exceptions. The response to an uncaught exception is:

Log the time, error, script name and traceback to standard output.
Display a web page containing an apologetic message to the user.
Save a technical web page with debugging information so that developers can look at it after-the-fact. These HTML-based error messages are stored one-per-file, if the SaveErrorMessages setting is true (the default). They are stored in the directory named by the ErrorMessagesDir (defaults to "ErrorMsgs").
Add an entry to the error log, found by default in Logs/Errors.csv.
E-mail the error message if the EmailErrors setting is true, using the settings ErrorEmailServer and ErrorEmailHeaders. See Configuration for more information. You should definitely set these options when deploying a web site.

Archived error messages can be browsed through the administration page.

Error handling behavior can be configured as described in Configuration.

Activity Log¶

Three options let you control:

Whether or not to log activity (LogActivity, defaults to 0, i.e. off)
The name of the file to store the log (ActivityLogFilename, defaults to Activity.csv)
The fields to store in the log (ActivityLogColumns) </ul>

See the chapter on Configuration for more information.

Administration¶

Webware has a built-in administration page that you can access via the Admin context. You can see a list of all contexts in the sidebar of any Example or Admin page.

The admin pages allows you to view Webware’s configuration, logs, and servlet cache, and perform actions such as clearing the cache or reloading selected modules.

More sensitive pages that give control over the application require a user name and password, the username is admin, and you can set the password with the AdminPassword setting in the Application.config file.

The administration scripts provide further examples of writing pages with Webware, so you may wish to examine their source in the Admin context.

Debugging¶

Development Mode¶

When creating the Application instance, it takes a development flag as argument that defines whether it should run in “development mode” or “production mode”. By default, if no such flag is passed, Webware checks whether the environment varibale WEBWARE_DEVELOPMENT is set and not empty. When you run the development server using the webware serve command, the flag is automatically set, so you are running in “development mode”, unless you add the --prod option on the command line. The development flag is also available with the name Development in the Application.config file and used to make some reasonable case distinctions depending on whether the application is running in development mode. For instance, debugging information is only shown in development mode.

print¶

The most common technique is the infamous print statement which has been replaced with a print() function in Python 3. The results of print() calls go to the console where the WSGI server was started (not to the HTML page as would happen with CGI). If you specify AppLogFilename in Application.config, this will cause the standard output and error to be redirected to this file.

For convenient debugging, the default Application.config file already uses the following conditional setting:

AppLogFilename = None if Development else 'Application.log'

This will prevent standard output and error from being redirected to the log file in development mode, which makes it easier to find debugging output, and also makes it possible to use pdb (see below).

Prefixing the debugging output with a special tag (such as >>) is useful because it stands out on the console and you can search for the tag in source code to remove the print statements after they are no longer useful. For example:

print('>> fields =', self.request().fields())

Raising Exceptions¶

Uncaught exceptions are trapped at the application level where a useful error page is saved with information such as the traceback, environment, fields, etc. In development mode, you will see this error page directly. In production, you can examine the saved page, and you can also configure the application to automatically e-mail you this information.

When an application isn’t behaving correctly, raising an exception can be useful because of the additional information that comes with it. Exceptions can be coupled with messages, thereby turning them into more powerful versions of the print() call. For example:

raise Exception(f'self = {self}')

While this is totally useful during development, giving away too much internal information is also a security risk, so you should make sure that the application is configured properly and no such debugging output is ever shown in production.

Reloading the Development Server¶

When a servlet’s source code changes, it is reloaded. However, ancestor classes of servlets, library modules and configuration files are not. You may wish to enable the auto-reloading feature when running the development server, by adding the -r or --reload option to the webware serve command in order to mitigate this problem.

In any case, when having problems, consider restarting the development server (or the WSGI server you are running in production).

Another option is to use the AppControl page of the Admin context to clear the servlet instance and class cache (see Administration).

Assertions¶

Assertions are used to ensure that the internal conditions of the application are as expected. An assertion is equivalent to an if statement coupled with an exception. For example:

assert shoppingCart.total() >= 0, \
    f'shopping cart total is {shoppingCart.total()}'

Debugging using PDB¶

To use Python’s built-in debugger pdb, see the tip above about setting AppLogFilename for convenient debugging.

To have Webware automatically put you into pdb when an exception occurs, set this in your Application.config file:

EnterDebuggerOnException = Development

A quick and easy way to debug a particular section of code is to add these lines at that point in the code:

import pdb
pdb.set_trace()

Debugging in an IDE¶

You can also use PyCharm or other Python IDEs to debug a Webware application. To do this, first configure the IDE to use the virtual environment where you installed Webware for Python.

Then, create the following script serve.py on the top level of your application working directory:

#!/usr/bin/python3

from webware.Scripts.WebwareCLI import main

main(['serve'])

Now run this file in your IDE in debug mode. For instance, in PyCharm, right-click on serve.py and select “Debug ‘serve’”.

Some IDEs like PyCharm can also debug remote processes. This could be useful to debug a test or production server.

Bootstrap Webware from Command line¶

You may be in a situation where you want to execute some part of your Webware applicaton from the command line, for example to implement a cron job or maintenance script. In these situations you probably don’t want to instantiate a full-fledged Application – some of the downsides are that doing so would cause standard output and standard error to be redirected to the log file, and that it sets up the session sweeper, task manager, etc. But you may still need access to plugins such as MiscUtils, MiddleKit, which you may not be able to import directly.

Here is a lightweight approach which allows you to bootstrap Webware and plugins:

import webware
app = webware.mockAppWithPlugins()

# now plugins are available...
import MiscUtils
import MiddleKit

How do I Develop an App?¶

The answer to that question might not seem clear after being deluged with all the details. Here’s a summary:

Make sure you can run the development server. See the Quickstart for more information.
Go through the Beginner Tutorial.
Read the source to the examples (in the Examples subdirectory), then modify one of them to get your toes wet.
Create your own new example from scratch. Ninety-nine percent of the time you will be subclassing the Page class.
Familiarize yourself with the class docs in order to take advantage of classes like Page, HTTPRequest, HTTPResponse and Session.
With this additional knowledge, create more sophisticated pages.
If you need to secure your pages using a login screen, you’ll want to look at the SecurePage, LoginPage, and SecureCountVisits examples in Examples. You’ll need to modify them to suit your particular needs.