Silva: Setting up a ZEO Cluster for ZEO 2, Zope 2.6.1 and DBTab.

Setting up a ZEO Cluster for ZEO 2, Zope 2.6.1 and DBTab.

Meta:

Valid for:  Silva 0.9.x
Author:     Jan-Wijbrand Kolman
Email:      jw@infrae.com
CVS:        $Revision: 1.3 $

Introduction:

These notes were collected during a ZEO2 Cluster setup proces to test Silva in a clustered environment. Most information can be found at the www.zope.org website, although either slightly outdated or targeting a specific platform or usage.

The document assumes knowledge of and experience with setting up Zope from sources, using the "INSTANCE_HOME" [Ax] technique, using sessions, etc. It also assumes a UNIX-like OS for the installation. The instructions '$ ' for the command line prompt and commands or code snippets which really should be typed on one line, will be split with a ''.

I hope the notes do provide enough instructions to setup a ZEO2 cluster, with DBTab and shared sessions and a highlevel overview and sufficient pointers to more information. Feedback and/or corrections are highly appreciated.

Problem description, solution:

A ZEO cluster provides building blocks to setup a scalable and highly available platform for CPU intensive webapplications.

Typically a ZEO cluster consists of 1 or more clients (using a "ClientStorage" - one will often find the "CS" abbreviation) and 1 server (the "ZEO StorageServer" - or ZSS). Each client, most probably running on a seperate network node, serves the actual webapplications.

The server provides access to one or more object databases and is responsible for synchronizing the clients throughout database changes. Each client keeps a local cache in RAM for objects and a separate "storage layer" cache, to speed up object read actions. Objects in these caches get invalidation messages from the ZSS whenever an object changed on an other node.

To evenly spread the application load over the different clients a 'front-end', or 'load balancing' server is put in place. End users connnect to the balancer, which transparently maps the requests to one of the clients. Typically the client and server nodes are kept on a private network, where the balancer acts as a "bridge" between the public and this private network.

Most web applications - and Silva is certainly one of them - keep track of the end users "state" within the application. Webapplications may do so by "tagging" the end user web browser with a cookie. This cookie merely provides an identifying number to the webapplication. With this ID the application keeps a 'session' object per end user. This session object is a generic data container to store end user states within the application.

Normally (in a non-clustered situation) this session object is not persisted into the ZODB (usually a FileStorage - the "Data.fs" file), but into a volatile storage type (TemporaryStorage), storing the objects in RAM.

This is possible since session objects are generaly speaking small and only exist for a limited lifespan (typically 20 minutes). Yet another reason is to prevent the main ZODB from excessive growth, since for each change in object (and in case of session objects, this effectively is for each HTTP request), data is appended to the ZODB.

In a clustered situation however, if the balancer maps a request to, say, client node A in the network, this will set a session object in its RAM cache. A consecutive request might be mapped to a client B, which does not have a session object in RAM identified by the end users cookie and a new empty sessions object is created. This will obviously break the application state tracking.

A solution is to actually persist the session objects in a central database on the ZSS. To prevent the mentioned excesive growth, a seperate database is setup which does not append, but just replaces the data on each (session) object change.

This separate database can be a TemporaryStorage, which means that the session objects are kept in a volatile RAM base storage on the ZSS itself. Since all clients connect to this storage, they are able to share the session objects. This setup also requires less system administration.

An alternative approach (similar as described in the earlier version of this document) is to store these session objects in a BDBStorage - which is the successor of the bsddb3Storage - a Berkeley DB backed storage type. This approach is not (yet) described in this document.

[DBTab details]

Software versions used:

Python 2.1.3

http://www.python.org
Zope 2.6.1

http://www.zope.org/Products/Zope/2.6.1
ZEO version 2.0.2

http://www.zope.org/Products/ZODB3.1 ZEO is now part of the so called "Standalone ZODB" distribution.
DBTab 1.1

http://hathaway.freezope.org/Software/DBTab

Software setup (in high level instrcutions):

Asumming the following directory layout:

../src (compiled Zope2.6.1 sources) ../client ('INSTANCE_HOME' for CS) ../server (location for ZSS)

To use ZEO copy the ZEO directory from the ZODB3.1.1 distribution to src/lib/python (or any other means to get ZEO on the python search path).

Place the 'StorageConfig.py' and 'CustomTemporaryStorage.py' python modules in the ../server directory. The 'StorageConfig' instructs the ZSS what storages should be served. The 'CustomTemporaryStorage' is a simple subclass of the standard Zope TemporaryStorage class.

The ZSS should be able to start now [B3]. The ZSS will start in "debug" mode and provide many details in the logfile if the '-D' option is supplied (in the startup script). In case the ZSS does not (correctly) start, this output might provide hints. A common problem is configuring the 'PYTHONPATH'.

Install the DBTab Product in ../client/Products directory and place the external method to delete the temp_folder [B6] in ../client/Extensions. As instructed by the DBTab documentation, copy (or symlink) the 'custom_zodb.py' from the DBTab Product directory to the ../client directory. We will not alter this file.

Copy the 'dbtab.conf.in' from the DBTab Product directory to the ../client directory and rename it to 'dbtab.conf'. This is the actual configuration file which will instruct the CS to connect to the ZSS [B4] and what storages to mount from the ZSS.

Now, the CS should be able to start. Again, when supplied with the '-D' option, detailed debugging information is provided in the logfile. This should show the CS actually connecting to the ZSS. The ZSS logs will also show this information.

Now two more steps are required both using the Zope Management Interface (ZMI). First add an "External Method". This requires three options to to be filled in (and a title which is optional): 'id', 'module name' and 'function name'. This external method will point to the 'delete_temp_folder.py' file. The filename is the 'module name'. The name of the method defined in there, is the 'function name'.

Once added, we can use the 'test tab' of this object to remove the existing 'temp_folder' (mind the special icon of this object). This folder cannot be deleted with the normal Zope 'delete' button, since it is "protected".

Now we can add a "DBTab mount point'. The form provided shows all available mountable storages. The '/temp_folder' as defined in the 'dbtab.conf' should be available. Select it and create the mount point.

After a CS restart, there should be a 'temp_folder' normal looking folder icon. Inside this folder there should be a 'session_data' object. This folder is actually a mount point to the TemporaryStorage on the ZSS.

A: References:

[Ax] Advanced Production Site Setup with ZEO:: http://www.zope.org/Members/dshaw/AdvancedSiteSetup

[Ax] Central Zope code base, multiple instances:: http://www.zope.org/Members/4am/instancehome

[Ax] ZEO setup:: http://www.zope.org/Members/kedai/UseZeoZope

[Ax] ZODB:: http://www.zope.org/Products/ZODB3.1

[Ax] DBTab product:: http://hathaway.freezope.org/Software/DBTab

B: Code:

[B1] StorageConfig.py - for storages to server to clients:

# begin import os from ZODB import DB from ZODB.FileStorage import FileStorage from Products.Transience.Transience import TransientObjectContainer from OFS.Folder import Folder # Getting TemporaryStorage to work with ZEO2 from CustomTemporaryStorage import CustomTemporaryStorage

mainStorage = FileStorage('/path/to/Data.fs',) sessionStorage = CustomTemporaryStorage() # end

[B2] CustomTemporaryStorage.py - subclass of TemporaryStorage:

# begin from Products.TemporaryFolder.TemporaryStorage import TemporaryStorage

class CustomTemporaryStorage(TemporaryStorage):: def supportsTransactionalUndo(self): return 0

# end

[B3] ZSS start script:

This bash script start the ZEO client. Should obviously be adjusted to local ZEO and INSTANCE_HOME setup.

#! /bin/bash export SRC=../src export INSTANCE_HOME=$PWD export STUPID_LOG_FILE=$PWD/var/ZEO_EVENTS.log export PYTHONPATH=$PWD:$SRC/lib/python:$PYTHONPATH

exec python2.1 $SRC/lib/python/ZEO/start.py: -t 10 -p portnumber-to-serve-on -D -S main=StorageConfig:mainStorage -S sessiondata=StorageConfig:sessionStorage &

# end

[B4] dbtab.conf for CS:

The 'dbtab.conf.in' as distributed with the DBTab Prodcust contains a lot of documentation on required and optional parameters.

# begin [Storage: Main] type=ClientStorage addr=zeo://url.to.zss:port client=zss-filesystemcache storage=main

[Database: Main] cache_size=10000 mount_paths=/

[Storage: SharedSessions] type=ClientStorage addr=zeo://url.to.zss:port storage=sessiondata

[Database: SharedSessions] #connection_class=LowConflictConnection # is the needed? cache_size=10000 mount_paths=/temp_folder open_at_startup=1 auto_create=1 # end

[B5] CS start script:

This bash script start the ZEO client. Should obviously be adjusted to local ZEO and INSTANCE_HOME setup, the use of FastCGI, ProxyPass, etc. For more info and details, read [zopesrc]/doc/ENVIRONMENT.txt

#!/bin/bash export FORCE_PRODUCT_LOAD=1 export INSTANCE_HOME=$PWD export CLIENT_HOME=/var/zeo2 # for storing caches and logs export EVENT_LOG_FILE=$CLIENT_HOME/zope.log

exec python2.1 ../src/z2.py: -X -w http-portnumber -F /path/to/fcgi/socket/file "$@" &

[B6] Delete temp_folder external method:

Place this external method in the Extensions directory of the clients INSTANCE_HOME.

# begin def delete_temp_folder(self):

self.getPhysicalRoot()._delObject('temp_folder') return 'Removed temp_folder successfully.'

# end

Infrae

Setting up a ZEO Cluster for ZEO 2, Zope 2.6.1 and DBTab.

Meta:

Introduction:

Problem description, solution:

Software versions used:

Software setup (in high level instrcutions):

A: References:

B: Code: