Customising Roundup

Welcome

This document used to be much larger and include a lot of reference material. That has been moved to the reference document.

The documentation is slowly being reorganized using the Diataxis framework. Help with the reorganization is welcome.

What You Can Do

Before you get too far, it’s probably worth having a quick read of the Roundup design documentation.

Customisation of Roundup can take one of six forms:

  1. tracker configuration changes

  2. database, or tracker schema changes

  3. “definition” class database content changes

  4. behavioural changes through detectors, extensions and interfaces.py

  5. security / access controls

  6. change the web interface

The third case is special because it takes two distinctly different forms depending upon whether the tracker has been initialised or not. The other two may be done at any time, before or after tracker initialisation. Yes, this includes adding or removing properties from classes.

Examples

Changing what’s stored in the database

The following examples illustrate ways to change the information stored in the database.

Adding a new field to the classic schema

This example shows how to add a simple field (a due date) to the default classic schema. It does not add any additional behaviour, such as enforcing the due date, or causing automatic actions to fire if the due date passes.

You add new fields by editing the schema.py file in you tracker’s home. Schema changes are automatically applied to the database on the next tracker access (note that roundup-server would need to be restarted as it caches the schema).

  1. Modify the schema.py:

    issue = IssueClass(db, "issue",
               assignedto=Link("user"), keyword=Multilink("keyword"),
               priority=Link("priority"), status=Link("status"),
               due_date=Date())

  2. Add an edit field to the issue.item.html template:

    <tr>
 <th>Due Date</th>
 <td tal:content="structure context/due_date/field" />
</tr>

    If you want to show only the date part of due_date then do this instead:

    <tr>
 <th>Due Date</th>
 <td tal:content="structure python:context.due_date.field(format='%Y-%m-%d')" />
</tr>

  3. Add the property to the issue.index.html page:

    (in the heading row)
  <th tal:condition="request/show/due_date">Due Date</th>
(in the data row)
  <td tal:condition="request/show/due_date"
      tal:content="i/due_date" />

    If you want format control of the display of the due date you can enter the following in the data row to show only the actual due date:

    <td tal:condition="request/show/due_date"
    tal:content="python:i.due_date.pretty('%Y-%m-%d')">&nbsp;</td>

  4. Add the property to the issue.search.html page:

    <tr tal:define="name string:due_date">
  <th i18n:translate="">Due Date:</th>
  <td metal:use-macro="search_input"></td>
  <td metal:use-macro="column_input"></td>
  <td metal:use-macro="sort_input"></td>
  <td metal:use-macro="group_input"></td>
</tr>

  5. If you wish for the due date to appear in the standard views listed in the sidebar of the web interface then you’ll need to add “due_date” to the columns and columns_showall lists in your page.html:

    columns string:id,activity,due_date,title,creator,status;
columns_showall string:id,activity,due_date,title,creator,assignedto,status;

Adding a new constrained field to the classic schema

This example shows how to add a new constrained property (i.e. a selection of distinct values) to your tracker.

Introduction

To make the classic schema of Roundup useful as a TODO tracking system for a group of systems administrators, it needs an extra data field per issue: a category.

This would let sysadmins quickly list all TODOs in their particular area of interest without having to do complex queries, and without relying on the spelling capabilities of other sysadmins (a losing proposition at best).

Adding a field to the database

This is the easiest part of the change. The category would just be a plain string, nothing fancy. To change what is in the database you need to add some lines to the schema.py file of your tracker instance. Under the comment:

# add any additional database schema configuration here

add:

category = Class(db, "category", name=String())
category.setkey("name")

Here we are setting up a chunk of the database which we are calling “category”. It contains a string, which we are refering to as “name” for lack of a more imaginative title. (Since “name” is one of the properties that Roundup looks for on items if you do not set a key for them, it’s probably a good idea to stick with it for new classes if at all appropriate.) Then we are setting the key of this chunk of the database to be that “name”. This is equivalent to an index for database types. This also means that there can only be one category with a given name.

Adding the above lines allows us to create categories, but they’re not tied to the issues that we are going to be creating. It’s just a list of categories off on its own, which isn’t much use. We need to link it in with the issues. To do that, find the lines in schema.py which set up the “issue” class, and then add a link to the category:

issue = IssueClass(db, "issue", ... ,
    category=Multilink("category"), ... )

The Multilink() means that each issue can have many categories. If you were adding something with a one-to-one relationship to issues (such as the “assignedto” property), use Link() instead.

That is all you need to do to change the schema. The rest of the effort is fiddling around so you can actually use the new category.

Populating the new category class

If you haven’t initialised the database with the “roundup-admin initialise” command, then you can add the following to the tracker initial_data.py under the comment:

# add any additional database creation steps here - but only if you
# haven't initialised the database with the admin "initialise" command

Add:

category = db.getclass('category')
category.create(name="scipy")
category.create(name="chaco")
category.create(name="weave")

If the database has already been initalised, then you need to use the roundup-admin tool:

% roundup-admin -i <tracker home>
Roundup <version> ready for input.
Type "help" for help.
roundup> create category name=scipy
1
roundup> create category name=chaco
2
roundup> create category name=weave
3
roundup> exit...
There are unsaved changes. Commit them (y/N)? y
Setting up security on the new objects

By default only the admin user can look at and change objects. This doesn’t suit us, as we want any user to be able to create new categories as required, and obviously everyone needs to be able to view the categories of issues for it to be useful.

We therefore need to change the security of the category objects. This is also done in schema.py.

There are currently two loops which set up permissions and then assign them to various roles. Simply add the new “category” to both lists:

# Assign the access and edit permissions for issue, file and message
# to regular users now
for cl in 'issue', 'file', 'msg', 'category':
    p = db.security.getPermission('View', cl)
    db.security.addPermissionToRole('User', 'View', cl)
    db.security.addPermissionToRole('User', 'Edit', cl)
    db.security.addPermissionToRole('User', 'Create', cl)

These lines assign the “View” and “Edit” Permissions to the “User” role, so that normal users can view and edit “category” objects.

This is all the work that needs to be done for the database. It will store categories, and let users view and edit them. Now on to the interface stuff.

Changing the web left hand frame

We need to give the users the ability to create new categories, and the place to put the link to this functionality is in the left hand function bar, under the “Issues” area. The file that defines how this area looks is html/page.html, which is what we are going to be editing next.

If you look at this file you can see that it contains a lot of “classblock” sections which are chunks of HTML that will be included or excluded in the output depending on whether the condition in the classblock is met. We are going to add the category code at the end of the classblock for the issue class:

<p class="classblock"
   tal:condition="python:request.user.hasPermission('View', 'category')">
 <b>Categories</b><br>
 <a tal:condition="python:request.user.hasPermission('Edit', 'category')"
    href="category?@template=item">New Category<br></a>
</p>

The first two lines is the classblock definition, which sets up a condition that only users who have “View” permission for the “category” object will have this section included in their output. Next comes a plain “Categories” header in bold. Everyone who can view categories will get that.

Next comes the link to the editing area of categories. This link will only appear if the condition - that the user has “Edit” permissions for the “category” objects - is matched. If they do have permission then they will get a link to another page which will let the user add new categories.

Note that if you have permission to view but not to edit categories, then all you will see is a “Categories” header with nothing underneath it. This is obviously not very good interface design, but will do for now. I just claim that it is so I can add more links in this section later on. However, to fix the problem you could change the condition in the classblock statement, so that only users with “Edit” permission would see the “Categories” stuff.

Setting up a page to edit categories

We defined code in the previous section which let users with the appropriate permissions see a link to a page which would let them edit conditions. Now we have to write that page.

The link was for the item template of the category object. This translates into Roundup looking for a file called category.item.html in the html tracker directory. This is the file that we are going to write now.

First, we add an info tag in a comment which doesn’t affect the outcome of the code at all, but is useful for debugging. If you load a page in a browser and look at the page source, you can see which sections come from which files by looking for these comments:

<!-- category.item -->

Next we need to add in the METAL macro stuff so we get the normal page trappings:

<tal:block metal:use-macro="templates/page/macros/icing">
 <title metal:fill-slot="head_title">Category editing</title>
 <td class="page-header-top" metal:fill-slot="body_title">
  <h2>Category editing</h2>
 </td>
 <td class="content" metal:fill-slot="content">

Next we need to setup up a standard HTML form, which is the whole purpose of this file. We link to some handy javascript which sends the form through only once. This is to stop users hitting the send button multiple times when they are impatient and thus having the form sent multiple times:

<form method="POST" onSubmit="return submit_once()"
      enctype="multipart/form-data">

Next we define some code which sets up the minimum list of fields that we require the user to enter. There will be only one field - “name” - so they better put something in it, otherwise the whole form is pointless:

<input type="hidden" name="@required" value="name">

To get everything to line up properly we will put everything in a table, and put a nice big header on it so the user has an idea what is happening:

<table class="form">
 <tr><th class="header" colspan="2">Category</th></tr>

Next, we need the field into which the user is going to enter the new category. The context.name.field(size=60) bit tells Roundup to generate a normal HTML field of size 60, and the contents of that field will be the “name” variable of the current context (namely “category”). The upshot of this is that when the user types something in to the form, a new category will be created with that name:

<tr>
 <th>Name</th>
 <td tal:content="structure python:context.name.field(size=60)">
 name</td>
</tr>

Then a submit button so that the user can submit the new category:

<tr>
 <td>&nbsp;</td>
 <td colspan="3" tal:content="structure context/submit">
  submit button will go here
 </td>
</tr>

The context/submit bit generates the submit button but also generates the @action and @csrf hidden fields. The @action field is used to tell Roundup how to process the form. The @csrf field provides a unique single use token to defend against CSRF attacks. (More about anti-csrf measures can be found in upgrading.txt.)

Finally we finish off the tags we used at the start to do the METAL stuff:

 </td>
</tal:block>

So putting it all together, and closing the table and form we get:

<!-- category.item -->
<tal:block metal:use-macro="templates/page/macros/icing">
 <title metal:fill-slot="head_title">Category editing</title>
 <td class="page-header-top" metal:fill-slot="body_title">
  <h2>Category editing</h2>
 </td>
 <td class="content" metal:fill-slot="content">
  <form method="POST" onSubmit="return submit_once()"
        enctype="multipart/form-data">

   <table class="form">
    <tr><th class="header" colspan="2">Category</th></tr>

    <tr>
     <th>Name</th>
     <td tal:content="structure python:context.name.field(size=60)">
     name</td>
    </tr>

    <tr>
     <td>
       &nbsp;
       <input type="hidden" name="@required" value="name">
     </td>
     <td colspan="3" tal:content="structure context/submit">
      submit button will go here
     </td>
    </tr>
   </table>
  </form>
 </td>
</tal:block>

This is quite a lot to just ask the user one simple question, but there is a lot of setup for basically one line (the form line) to do its work. To add another field to “category” would involve one more line (well, maybe a few extra to get the formatting correct).

Adding the category to the issue

We now have the ability to create issues to our heart’s content, but that is pointless unless we can assign categories to issues. Just like the html/category.item.html file was used to define how to add a new category, the html/issue.item.html is used to define how a new issue is created.

Just like category.issue.html, this file defines a form which has a table to lay things out. It doesn’t matter where in the table we add new stuff, it is entirely up to your sense of aesthetics:

<th>Category</th>
<td>
 <span tal:replace="structure context/category/field" />
 <span tal:replace="structure python:db.category.classhelp('name',
             property='category', width='200')" />
</td>

First, we define a nice header so that the user knows what the next section is, then the middle line does what we are most interested in. This context/category/field gets replaced by a field which contains the category in the current context (the current context being the new issue).

The classhelp lines generate a link (labelled “list”) to a popup window which contains the list of currently known categories.

Searching on categories

Now we can add categories, and create issues with categories. The next obvious thing that we would like to be able to do, would be to search for issues based on their category, so that, for example, anyone working on the web server could look at all issues in the category “Web”.

If you look for “Search Issues” in the html/page.html file, you will find that it looks something like <a href="issue?@template=search">Search Issues</a>. This shows us that when you click on “Search Issues” it will be looking for a issue.search.html file to display. So that is the file that we will change.

If you look at this file it should begin to seem familiar, although it does use some new macros. You can add the new category search code anywhere you like within that form:

<tr tal:define="name string:category;
                db_klass string:category;
                db_content string:name;">
  <th>Priority:</th>
  <td metal:use-macro="search_select"></td>
  <td metal:use-macro="column_input"></td>
  <td metal:use-macro="sort_input"></td>
  <td metal:use-macro="group_input"></td>
</tr>

The definitions in the <tr> opening tag are used by the macros:

  • search_select expands to a drop-down box with all categories using db_klass and db_content.

  • column_input expands to a checkbox for selecting what columns should be displayed.

  • sort_input expands to a radio button for selecting what property should be sorted on.

  • group_input expands to a radio button for selecting what property should be grouped on.

The category search code above would expand to the following:

<tr>
  <th>Category:</th>
  <td>
    <select name="category">
      <option value="">don't care</option>
      <option value="">------------</option>
      <option value="1">scipy</option>
      <option value="2">chaco</option>
      <option value="3">weave</option>
    </select>
  </td>
  <td><input type="checkbox" name=":columns" value="category"></td>
  <td><input type="radio" name=":sort0" value="category"></td>
  <td><input type="radio" name=":group0" value="category"></td>
</tr>
Adding category to the default view

We can now add categories, add issues with categories, and search for issues based on categories. This is everything that we need to do; however, there is some more icing that we would like. I think the category of an issue is important enough that it should be displayed by default when listing all the issues.

Unfortunately, this is a bit less obvious than the previous steps. The code defining how the issues look is in html/issue.index.html. This is a large table with a form down at the bottom for redisplaying and so forth.

Firstly we need to add an appropriate header to the start of the table:

<th tal:condition="request/show/category">Category</th>

The condition part of this statement is to avoid displaying the Category column if the user has selected not to see it.

The rest of the table is a loop which will go through every issue that matches the display criteria. The loop variable is “i” - which means that every issue gets assigned to “i” in turn.

The new part of code to display the category will look like this:

<td tal:condition="request/show/category"
    tal:content="i/category"></td>

The condition is the same as above: only display the condition when the user hasn’t asked for it to be hidden. The next part is to set the content of the cell to be the category part of “i” - the current issue.

Finally we have to edit html/page.html again. This time, we need to tell it that when the user clicks on “Unassigned Issues” or “All Issues”, the category column should be included in the resulting list. If you scroll down the page file, you can see the links with lots of options. The option that we are interested in is the :columns= one which tells Roundup which fields of the issue to display. Simply add “category” to that list and it all should work.

Adding a time log to your issues

We want to log the dates and amount of time spent working on issues, and be able to give a summary of the total time spent on a particular issue.

  1. Add a new class to your tracker schema.py:

    # storage for time logging
timelog = Class(db, "timelog", period=Interval())

    Note that we automatically get the date of the time log entry creation through the standard property “creation”.

    You will need to grant “Creation” permission to the users who are allowed to add timelog entries. You may do this with:

    db.security.addPermissionToRole('User', 'Create', 'timelog')
db.security.addPermissionToRole('User', 'View', 'timelog')

    If users are also able to edit timelog entries, then also include:

    db.security.addPermissionToRole('User', 'Edit', 'timelog')

  1. Link to the new class from your issue class (again, in schema.py):

    issue = IssueClass(db, "issue",
                assignedto=Link("user"), keyword=Multilink("keyword"),
                priority=Link("priority"), status=Link("status"),
                times=Multilink("timelog"))

    the “times” property is the new link to the “timelog” class.

  2. We’ll need to let people add in times to the issue, so in the web interface we’ll have a new entry field. This is a special field because unlike the other fields in the issue.item template, it affects a different item (a timelog item) and not the template’s item (an issue). We have a special syntax for form fields that affect items other than the template default item (see the cgi documentation on special form variables). In particular, we add a field to capture a new timelog item’s period:

    <tr>
 <th>Time Log</th>
 <td colspan=3><input type="text" name="timelog-1@period" />
  (enter as '3y 1m 4d 2:40:02' or parts thereof)
 </td>
</tr>

    and another hidden field that links that new timelog item (new because it’s marked as having id “-1”) to the issue item. It looks like this:

    <input type="hidden" name="@link@times" value="timelog-1" />

    On submission, the “-1” timelog item will be created and assigned a real item id. The “times” property of the issue will have the new id added to it.

    The full entry will now look like this:

    <tr>
 <th>Time Log</th>
 <td colspan=3><input type="text" name="timelog-1@period" />
  (enter as '3y 1m 4d 2:40:02' or parts thereof)
  <input type="hidden" name="@link@times" value="timelog-1" />
 </td>
</tr>

  1. We want to display a total of the timelog times that have been accumulated for an issue. To do this, we’ll need to actually write some Python code, since it’s beyond the scope of PageTemplates to perform such calculations. We do this by adding a module timespent.py to the extensions directory in our tracker. The contents of this file is as follows:

    from roundup import date

def totalTimeSpent(times):
    ''' Call me with a list of timelog items (which have an
        Interval "period" property)
    '''
    total = date.Interval('0d')
    for time in times:
        total += time.period._value
    return total

def init(instance):
    instance.registerUtil('totalTimeSpent', totalTimeSpent)

    We will now be able to access the totalTimeSpent function via the utils variable in our templates, as shown in the next step.

  2. Display the timelog for an issue:

    <table class="otherinfo" tal:condition="context/times">
 <tr><th colspan="3" class="header">Time Log
  <tal:block
       tal:replace="python:utils.totalTimeSpent(context.times)" />
 </th></tr>
 <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
 <tr tal:repeat="time context/times">
  <td tal:content="time/creation"></td>
  <td tal:content="time/period"></td>
  <td tal:content="time/creator"></td>
 </tr>
</table>

    I put this just above the Messages log in my issue display. Note our use of the totalTimeSpent method which will total up the times for the issue and return a new Interval. That will be automatically displayed in the template as text like “+ 1y 2:40” (1 year, 2 hours and 40 minutes).

  3. If you’re using a persistent web server - roundup-server or mod_wsgi for example - then you’ll need to restart that to pick up the code changes. When that’s done, you’ll be able to use the new time logging interface.

An extension of this modification attaches the timelog entries to any change message entered at the time of the timelog entry:

  1. Add a link to the timelog to the msg class in schema.py:

    msg = FileClass(db, “msg”,

    author=Link(“user”, do_journal=’no’), recipients=Multilink(“user”, do_journal=’no’), date=Date(), summary=String(), files=Multilink(“file”), messageid=String(), inreplyto=String(), times=Multilink(“timelog”))

  2. Add a new hidden field that links that new timelog item (new because it’s marked as having id “-1”) to the new message. The link is placed in issue.item.html in the same section that handles the timelog entry.

    It looks like this after this addition:

    <tr>
 <th>Time Log</th>
 <td colspan=3><input type="text" name="timelog-1@period" />
  (enter as '3y 1m 4d 2:40:02' or parts thereof)
  <input type="hidden" name="@link@times" value="timelog-1" />
  <input type="hidden" name="msg-1@link@times" value="timelog-1" />
 </td>
</tr>

    The “times” property of the message will have the new id added to it.

  3. Add the timelog listing from step 5. to the msg.item.html template so that the timelog entry appears on the message view page. Note that the call to totalTimeSpent is not used here since there will only be one single timelog entry for each message.

    I placed it after the Date entry like this:

    <tr>
 <th i18n:translate="">Date:</th>
 <td tal:content="context/date"></td>
</tr>
</table>

<table class="otherinfo" tal:condition="context/times">
 <tr><th colspan="3" class="header">Time Log</th></tr>
 <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
 <tr tal:repeat="time context/times">
  <td tal:content="time/creation"></td>
  <td tal:content="time/period"></td>
  <td tal:content="time/creator"></td>
 </tr>
</table>

<table class="messages">

The wiki includes an auditor that extracts specially formatted timelog entries from emails sent to the tracker.

Tracking different types of issues

Sometimes you will want to track different types of issues - developer, customer support, systems, sales leads, etc. A single Roundup tracker is able to support multiple types of issues. This example demonstrates adding a system support issue class to a tracker.

  1. Figure out what information you’re going to want to capture. OK, so this is obvious, but sometimes it’s better to actually sit down for a while and think about the schema you’re going to implement.

  2. Add the new issue class to your tracker’s schema.py. Just after the “issue” class definition, add:

    # list our systems
system = Class(db, "system", name=String(), order=Number())
system.setkey("name")

# store issues related to those systems
support = IssueClass(db, "support",
                assignedto=Link("user"), keyword=Multilink("keyword"),
                status=Link("status"), deadline=Date(),
                affects=Multilink("system"))

  3. Copy the existing issue.* (item, search and index) templates in the tracker’s html to support.*. Edit them so they use the properties defined in the support class. Be sure to check for hidden form variables like “required” to make sure they have the correct set of required properties.

  4. Edit the modules in the detectors, adding lines to their init functions where appropriate. Look for audit and react registrations on the issue class, and duplicate them for support.

  5. Create a new sidebar box for the new support class. Duplicate the existing issues one, changing the issue class name to support.

  6. Re-start your tracker and start using the new support class.

Optionally, you might want to restrict the users able to access this new class to just the users with a new “SysAdmin” Role. To do this, we add some security declarations:

db.security.addPermissionToRole('SysAdmin', 'View', 'support')
db.security.addPermissionToRole('SysAdmin', 'Create', 'support')
db.security.addPermissionToRole('SysAdmin', 'Edit', 'support')

You would then (as an “admin” user) edit the details of the appropriate users, and add “SysAdmin” to their Roles list.

Alternatively, you might want to change the Edit/View permissions granted for the issue class so that it’s only available to users with the “System” or “Developer” Role, and then the new class you’re adding is available to all with the “User” Role.

Using External User Databases

Using an external password validation source

Note

You will need to either have an “admin” user in your external password source or have one of your regular users have the Admin Role assigned. If you need to assign the Role after making the changes below, you may use the roundup-admin program to edit a user’s details.

We have a centrally-managed password changing system for our users. This results in a UN*X passwd-style file that we use for verification of users. Entries in the file consist of name:password where the password is encrypted using the standard UN*X crypt() function (see the crypt module in your Python distribution). An example entry would be:

admin:aamrgyQfDFSHw

Each user of Roundup must still have their information stored in the Roundup database - we just use the passwd file to check their password. To do this, we need to override the standard verifyPassword method defined in roundup.cgi.actions.LoginAction and register the new class. The following is added as externalpassword.py in the tracker extensions directory:

import os, crypt
from roundup.cgi.actions import LoginAction

class ExternalPasswordLoginAction(LoginAction):
    def verifyPassword(self, userid, password):
        '''Look through the file, line by line, looking for a
        name that matches.
        '''
        # get the user's username
        username = self.db.user.get(userid, 'username')

        # the passwords are stored in the "passwd.txt" file in the
        # tracker home
        file = os.path.join(self.db.config.TRACKER_HOME, 'passwd.txt')

        # see if we can find a match
        for ent in [line.strip().split(':') for line in
                                            open(file).readlines()]:
            if ent[0] == username:
                return crypt.crypt(password, ent[1][:2]) == ent[1]

        # user doesn't exist in the file
        return 0

def init(instance):
    instance.registerAction('login', ExternalPasswordLoginAction)

You should also remove the redundant password fields from the user.item template.

Using a UN*X passwd file as the user database

On some systems the primary store of users is the UN*X passwd file. It holds information on users such as their username, real name, password and primary user group.

Roundup can use this store as its primary source of user information, but it needs additional information too - email address(es), Roundup Roles, vacation flags, Roundup hyperdb item ids, etc. Also, “retired” users must still exist in the user database, unlike some passwd files in which the users are removed when they no longer have access to a system.

To make use of the passwd file, we therefore synchronise between the two user stores. We also use the passwd file to validate the user logins, as described in the previous example, using an external password validation source. We keep the user lists in sync using a fairly simple script that runs once a day, or several times an hour if more immediate access is needed. In short, it:

  1. parses the passwd file, finding usernames, passwords and real names,

  2. compares that list to the current Roundup user list:

    1. entries no longer in the passwd file are retired

    2. entries with mismatching real names are updated

    3. entries only exist in the passwd file are created

  3. send an email to administrators to let them know what’s been done.

The retiring and updating are simple operations, requiring only a call to retire() or set(). The creation operation requires more information though - the user’s email address and their Roundup Roles. We’re going to assume that the user’s email address is the same as their login name, so we just append the domain name to that. The Roles are determined using the passwd group identifier - mapping their UN*X group to an appropriate set of Roles.

The script to perform all this, broken up into its main components, is as follows. Firstly, we import the necessary modules and open the tracker we’re to work on:

import sys, os, smtplib
from roundup import instance, date

# open the tracker
tracker_home = sys.argv[1]
tracker = instance.open(tracker_home)

Next we read in the passwd file from the tracker home:

# read in the users from the "passwd.txt" file
file = os.path.join(tracker_home, 'passwd.txt')
users = [x.strip().split(':') for x in open(file).readlines()]

Handle special users (those to ignore in the file, and those who don’t appear in the file):

# users to not keep ever, pre-load with the users I know aren't
# "real" users
ignore = ['ekmmon', 'bfast', 'csrmail']

# users to keep - pre-load with the roundup-specific users
keep = ['comment_pool', 'network_pool', 'admin', 'dev-team',
        'cs_pool', 'anonymous', 'system_pool', 'automated']

Now we map the UN*X group numbers to the Roles that users should have:

roles = {
 '501': 'User,Tech',  # tech
 '502': 'User',       # finance
 '503': 'User,CSR',   # customer service reps
 '504': 'User',       # sales
 '505': 'User',       # marketing
}

Now we do all the work. Note that the body of the script (where we have the tracker database open) is wrapped in a try / finally clause, so that we always close the database cleanly when we’re finished. So, we now do all the work:

# open the database
db = tracker.open('admin')
try:
    # store away messages to send to the tracker admins
    msg = []

    # loop over the users list read in from the passwd file
    for user,passw,uid,gid,real,home,shell in users:
        if user in ignore:
            # this user shouldn't appear in our tracker
            continue
        keep.append(user)
        try:
            # see if the user exists in the tracker
            uid = db.user.lookup(user)

            # yes, they do - now check the real name for correctness
            if real != db.user.get(uid, 'realname'):
                db.user.set(uid, realname=real)
                msg.append('FIX %s - %s'%(user, real))
        except KeyError:
            # nope, the user doesn't exist
            db.user.create(username=user, realname=real,
                address='%s@ekit-inc.com'%user, roles=roles[gid])
            msg.append('ADD %s - %s (%s)'%(user, real, roles[gid]))

    # now check that all the users in the tracker are also in our
    # "keep" list - retire those who aren't
    for uid in db.user.list():
        user = db.user.get(uid, 'username')
        if user not in keep:
            db.user.retire(uid)
            msg.append('RET %s'%user)

    # if we did work, then send email to the tracker admins
    if msg:
        # create the email
        msg = '''Subject: %s user database maintenance

        %s
        '''%(db.config.TRACKER_NAME, '\n'.join(msg))

        # send the email
        smtp = smtplib.SMTP(db.config.MAILHOST)
        addr = db.config.ADMIN_EMAIL
        smtp.sendmail(addr, addr, msg)

    # now we're done - commit the changes
    db.commit()
finally:
    # always close the database cleanly
    db.close()

And that’s it!

Using an LDAP database for user information

A script that reads users from an LDAP store using https://pypi.org/project/python-ldap/ and then compares the list to the users in the Roundup user database would be pretty easy to write. You’d then have it run once an hour / day (or on demand if you can work that into your LDAP store workflow). See the example Using a UN*X passwd file as the user database for more information about doing this.

To authenticate off the LDAP store (rather than using the passwords in the Roundup user database) you’d use the same python-ldap module inside an extension to the cgi interface. You’d do this by overriding the method called verifyPassword on the LoginAction class in your tracker’s extensions directory (see using an external password validation source). The method is implemented by default as:

def verifyPassword(self, userid, password):
    ''' Verify the password that the user has supplied
    '''
    stored = self.db.user.get(self.userid, 'password')
    if password == stored:
        return 1
    if not password and not stored:
        return 1
    return 0

So you could reimplement this as something like:

def verifyPassword(self, userid, password):
    ''' Verify the password that the user has supplied
    '''
    # look up some unique LDAP information about the user
    username = self.db.user.get(self.userid, 'username')
    # now verify the password supplied against the LDAP store

Other External Databases

See examples for Shibboleth and info about using OAUTH in the Roundup Wiki.

Changes to Tracker Behaviour

Preventing SPAM

The following detector code may be installed in your tracker’s detectors directory. It will block any messages being created that have HTML attachments (a very common vector for spam and phishing) and any messages that have more than 2 HTTP URLs in them. Just copy the following into detectors/anti_spam.py in your tracker:

from roundup.exceptions import Reject

def reject_html(db, cl, nodeid, newvalues):
    if newvalues['type'] == 'text/html':
    raise Reject('not allowed')

def reject_manylinks(db, cl, nodeid, newvalues):
    content = newvalues['content']
    if content.count('http://') > 2:
    raise Reject('not allowed')

def init(db):
    db.file.audit('create', reject_html)
    db.msg.audit('create', reject_manylinks)

You may also wish to block image attachments if your tracker does not need that ability:

if newvalues['type'].startswith('image/'):
    raise Reject('not allowed')

Stop “nosy” messages going to people on vacation

When users go on vacation and set up vacation email bouncing, you’ll start to see a lot of messages come back through Roundup “Fred is on vacation”. Not very useful, and relatively easy to stop.

  1. add a “vacation” flag to your users:

    user = Class(db, "user",
           username=String(),   password=Password(),
           address=String(),    realname=String(),
           phone=String(),      organisation=String(),
           alternate_addresses=String(),
           roles=String(), queries=Multilink("query"),
           vacation=Boolean())

  2. So that users may edit the vacation flags, add something like the following to your user.item template:

    <tr>
 <th>On Vacation</th>
 <td tal:content="structure context/vacation/field">vacation</td>
</tr>

  3. edit your detector nosyreactor.py so that the nosyreaction() consists of:

    def nosyreaction(db, cl, nodeid, oldvalues):
    users = db.user
    messages = db.msg
    # send a copy of all new messages to the nosy list
    for msgid in determineNewMessages(cl, nodeid, oldvalues):
        try:
            # figure the recipient ids
            sendto = []
            seen_message = {}
            recipients = messages.get(msgid, 'recipients')
            for recipid in messages.get(msgid, 'recipients'):
                seen_message[recipid] = 1

            # figure the author's id, and indicate they've received
            # the message
            authid = messages.get(msgid, 'author')

            # possibly send the message to the author, as long as
            # they aren't anonymous
            if (db.config.MESSAGES_TO_AUTHOR == 'yes' and
                    users.get(authid, 'username') != 'anonymous'):
                sendto.append(authid)
            seen_message[authid] = 1

            # now figure the nosy people who weren't recipients
            nosy = cl.get(nodeid, 'nosy')
            for nosyid in nosy:
                # Don't send nosy mail to the anonymous user (that
                # user shouldn't appear in the nosy list, but just
                # in case they do...)
                if users.get(nosyid, 'username') == 'anonymous':
                    continue
                # make sure they haven't seen the message already
                if nosyid not in seen_message:
                    # send it to them
                    sendto.append(nosyid)
                    recipients.append(nosyid)

            # generate a change note
            if oldvalues:
                note = cl.generateChangeNote(nodeid, oldvalues)
            else:
                note = cl.generateCreateNote(nodeid)

            # we have new recipients
            if sendto:
                # filter out the people on vacation
                sendto = [i for i in sendto
                          if not users.get(i, 'vacation', 0)]

                # map userids to addresses
                sendto = [users.get(i, 'address') for i in sendto]

                # update the message's recipients list
                messages.set(msgid, recipients=recipients)

                # send the message
                cl.send_message(nodeid, msgid, note, sendto)
        except roundupdb.MessageSendError as message:
            raise roundupdb.DetectorError(message)

    Note that this is the standard nosy reaction code, with the small addition of:

    # filter out the people on vacation
sendto = [i for i in sendto if not users.get(i, 'vacation', 0)]

    which filters out the users that have the vacation flag set to true.

Adding in state transition control

Sometimes tracker admins want to control the states to which users may move issues. You can do this by following these steps:

  1. make “status” a required variable. This is achieved by adding the following to the top of the form in the issue.item.html template:

    <input type="hidden" name="@required" value="status">

    This will force users to select a status.

  2. add a Multilink property to the status class:

    stat = Class(db, "status", ... , transitions=Multilink('status'),
             ...)

    and then edit the statuses already created, either:

    1. through the web using the class list -> status class editor, or

    2. using the roundup-admin “set” command.

  3. add an auditor module checktransition.py in your tracker’s detectors directory, for example:

    def checktransition(db, cl, nodeid, newvalues):
    ''' Check that the desired transition is valid for the "status"
        property.
    '''
    if 'status' not in newvalues:
        return
    current = cl.get(nodeid, 'status')
    new = newvalues['status']
    if new == current:
        return
    ok = db.status.get(current, 'transitions')
    if new not in ok:
        raise ValueError('Status not allowed to move from "%s" to "%s"'%(
            db.status.get(current, 'name'), db.status.get(new, 'name')))

def init(db):
    db.issue.audit('set', checktransition)

  4. in the issue.item.html template, change the status editing bit from:

    <th>Status</th>
<td tal:content="structure context/status/menu">status</td>

    to:

    <th>Status</th>
<td>
 <select tal:condition="context/id" name="status">
  <tal:block tal:define="ok context/status/transitions"
             tal:repeat="state db/status/list">
   <option tal:condition="python:state.id in ok"
           tal:attributes="
                value state/id;
                selected python:state.id == context.status.id"
           tal:content="state/name"></option>
  </tal:block>
 </select>
 <tal:block tal:condition="not:context/id"
            tal:replace="structure context/status/menu" />
</td>

    which displays only the allowed status to transition to.

Blocking issues that depend on other issues

We needed the ability to mark certain issues as “blockers” - that is, they can’t be resolved until another issue (the blocker) they rely on is resolved. To achieve this:

  1. Create a new property on the issue class: blockers=Multilink("issue"). To do this, edit the definition of this class in your tracker’s schema.py file. Change this:

    issue = IssueClass(db, "issue",
                assignedto=Link("user"), keyword=Multilink("keyword"),
                priority=Link("priority"), status=Link("status"))

    to this, adding the blockers entry:

    issue = IssueClass(db, "issue",
                blockers=Multilink("issue"),
                assignedto=Link("user"), keyword=Multilink("keyword"),
                priority=Link("priority"), status=Link("status"))

  2. Add the new blockers property to the issue.item.html edit page, using something like:

    <th>Waiting On</th>
<td>
 <span tal:replace="structure python:context.blockers.field(showid=1,
                              size=20)" />
 <span tal:replace="structure python:db.issue.classhelp('id,title',
                              property='blockers')" />
 <span tal:condition="context/blockers"
       tal:repeat="blk context/blockers">
  <br>View: <a tal:attributes="href string:issue${blk/id}"
               tal:content="blk/id"></a>
 </span>
</td>

    You’ll need to fiddle with your item page layout to find an appropriate place to put it - I’ll leave that fun part up to you. Just make sure it appears in the first table, possibly somewhere near the “superseders” field.

  3. Create a new detector module (see below) which enforces the rules:

    • issues may not be resolved if they have blockers

    • when a blocker is resolved, it’s removed from issues it blocks

    The contents of the detector should be something like this:

    def blockresolution(db, cl, nodeid, newvalues):
    ''' If the issue has blockers, don't allow it to be resolved.
    '''
    if nodeid is None:
        blockers = []
    else:
        blockers = cl.get(nodeid, 'blockers')
    blockers = newvalues.get('blockers', blockers)

    # don't do anything if there's no blockers or the status hasn't
    # changed
    if not blockers or 'status' not in newvalues:
        return

    # get the resolved state ID
    resolved_id = db.status.lookup('resolved')

    # format the info
    u = db.config.TRACKER_WEB
    s = ', '.join(['<a href="%sissue%s">%s</a>'%(
                    u,id,id) for id in blockers])
    if len(blockers) == 1:
        s = 'issue %s is'%s
    else:
        s = 'issues %s are'%s

    # ok, see if we're trying to resolve
    if newvalues['status'] == resolved_id:
        raise ValueError("This issue can't be resolved until %s resolved."%s)


def resolveblockers(db, cl, nodeid, oldvalues):
    ''' When we resolve an issue that's a blocker, remove it from the
        blockers list of the issue(s) it blocks.
    '''
    newstatus = cl.get(nodeid,'status')

    # no change?
    if oldvalues.get('status', None) == newstatus:
        return

    resolved_id = db.status.lookup('resolved')

    # interesting?
    if newstatus != resolved_id:
        return

    # yes - find all the blocked issues, if any, and remove me from
    # their blockers list
    issues = cl.find(blockers=nodeid)
    for issueid in issues:
        blockers = cl.get(issueid, 'blockers')
        if nodeid in blockers:
            blockers.remove(nodeid)
            cl.set(issueid, blockers=blockers)

def init(db):
    # might, in an obscure situation, happen in a create
    db.issue.audit('create', blockresolution)
    db.issue.audit('set', blockresolution)

    # can only happen on a set
    db.issue.react('set', resolveblockers)

    Put the above code in a file called “blockers.py” in your tracker’s “detectors” directory.

  4. Finally, and this is an optional step, modify the tracker web page URLs so they filter out issues with any blockers. You do this by adding an additional filter on “blockers” for the value “-1”. For example, the existing “Show All” link in the “page” template (in the tracker’s “html” directory) looks like this:

    <a href="#"
   tal:attributes="href python:request.indexargs_url('issue', {
  '@sort': '-activity',
  '@group': 'priority',
  '@filter': 'status',
  '@columns': columns_showall,
  '@search_text': '',
  'status': status_notresolved,
  '@dispname': i18n.gettext('Show All'),
 })"
   i18n:translate="">Show All</a><br>

    modify it to add the “blockers” info to the URL (note, both the “@filter” and “blockers” values must be specified):

    <a href="#"
   tal:attributes="href python:request.indexargs_url('issue', {
  '@sort': '-activity',
  '@group': 'priority',
  '@filter': 'status,blockers',
  '@columns': columns_showall,
  '@search_text': '',
  'status': status_notresolved,
  'blockers': '-1',
  '@dispname': i18n.gettext('Show All'),
 })"
   i18n:translate="">Show All</a><br>

    The above examples are line-wrapped on the trailing & and should be unwrapped.

That’s it. You should now be able to set blockers on your issues. Note that if you want to know whether an issue has any other issues dependent on it (i.e. it’s in their blockers list) you can look at the journal history at the bottom of the issue page - look for a “link” event to another issue’s “blockers” property.

Add users to the nosy list based on the keyword

Let’s say we need the ability to automatically add users to the nosy list based on the occurance of a keyword. Every user should be allowed to edit their own list of keywords for which they want to be added to the nosy list.

Below, we’ll show that this change can be done with minimal understanding of the Roundup system, using only copy and paste.

This requires three changes to the tracker: a change in the database to allow per-user recording of the lists of keywords for which he wants to be put on the nosy list, a change in the user view allowing them to edit this list of keywords, and addition of an auditor which updates the nosy list when a keyword is set.

Adding the nosy keyword list

The change to make in the database, is that for any user there should be a list of keywords for which he wants to be put on the nosy list. Adding a Multilink of keyword seems to fullfill this. As such, all that has to be done is to add a new field to the definition of user within the file schema.py. We will call this new field nosy_keywords, and the updated definition of user will be:

user = Class(db, "user",
                username=String(),   password=Password(),
                address=String(),    realname=String(),
                phone=String(),      organisation=String(),
                alternate_addresses=String(),
                queries=Multilink('query'), roles=String(),
                timezone=String(),
                nosy_keywords=Multilink('keyword'))
Changing the user view to allow changing the nosy keyword list

We want any user to be able to change the list of keywords for which he will by default be added to the nosy list. We choose to add this to the user view, as is generated by the file html/user.item.html. We can easily see that the keyword field in the issue view has very similar editing requirements as our nosy keywords, both being lists of keywords. As such, we look for Keywords in issue.item.html, and extract the associated parts from there. We add this to user.item.html at the bottom of the list of viewed items (i.e. just below the ‘Alternate E-mail addresses’ in the classic template):

<tr>
 <th>Nosy Keywords</th>
 <td>
 <span tal:replace="structure context/nosy_keywords/field" />
 <span tal:replace="structure python:db.keyword.classhelp(property='nosy_keywords')" />
 </td>
</tr>
Addition of an auditor to update the nosy list

The more difficult part is the logic to add the users to the nosy list when required. We choose to perform this action whenever the keywords on an item are set (this includes the creation of items). Here we choose to start out with a copy of the detectors/nosyreaction.py detector, which we copy to the file detectors/nosy_keyword_reaction.py. This looks like a good start as it also adds users to the nosy list. A look through the code reveals that the nosyreaction function actually sends the e-mail. We don’t need this. Therefore, we can change the init function to:

def init(db):
    db.issue.audit('create', update_kw_nosy)
    db.issue.audit('set', update_kw_nosy)

After that, we rename the updatenosy function to update_kw_nosy. The first two blocks of code in that function relate to setting current to a combination of the old and new nosy lists. This functionality is left in the new auditor. The following block of code, which handled adding the assignedto user(s) to the nosy list in updatenosy, should be replaced by a block of code to add the interested users to the nosy list. We choose here to loop over all new keywords, than looping over all users, and assign the user to the nosy list when the keyword occurs in the user’s nosy_keywords. The next part in updatenosy – adding the author and/or recipients of a message to the nosy list – is obviously not relevant here and is thus deleted from the new auditor. The last part, copying the new nosy list to newvalues, can stay as is. This results in the following function:

def update_kw_nosy(db, cl, nodeid, newvalues):
    '''Update the nosy list for changes to the keywords
    '''
    # nodeid will be None if this is a new node
    current = {}
    if nodeid is None:
        ok = ('new', 'yes')
    else:
        ok = ('yes',)
        # old node, get the current values from the node if they haven't
        # changed
        if 'nosy' not in newvalues:
            nosy = cl.get(nodeid, 'nosy')
            for value in nosy:
                if value not in current:
                    current[value] = 1

    # if the nosy list changed in this transaction, init from the new value
    if 'nosy' in newvalues:
        nosy = newvalues.get('nosy', [])
        for value in nosy:
            if not db.hasnode('user', value):
                continue
            if value not in current:
                current[value] = 1

    # add users with keyword in nosy_keywords to the nosy list
    if 'keyword' in newvalues and newvalues['keyword'] is not None:
        keyword_ids = newvalues['keyword']
        for keyword in keyword_ids:
            # loop over all users,
            # and assign user to nosy when keyword in nosy_keywords
            for user_id in db.user.list():
                nosy_kw = db.user.get(user_id, "nosy_keywords")
                found = 0
                for kw in nosy_kw:
                    if kw == keyword:
                        found = 1
                if found:
                    current[user_id] = 1

    # that's it, save off the new nosy list
    newvalues['nosy'] = list(current.keys())

These two function are the only ones needed in the file.

TODO: update this example to use the find() Class method.

Caveats

A few problems with the design here can be noted:

Multiple additions

When a user, after automatic selection, is manually removed from the nosy list, he is added to the nosy list again when the keyword list of the issue is updated. A better design might be to only check which keywords are new compared to the old list of keywords, and only add users when they have indicated interest on a new keyword.

The code could also be changed to only trigger on the create() event, rather than also on the set() event, thus only setting the nosy list when the issue is created.

Scalability

In the auditor, there is a loop over all users. For a site with only few users this will pose no serious problem; however, with many users this will be a serious performance bottleneck. A way out would be to link from the keywords to the users who selected these keywords as nosy keywords. This will eliminate the loop over all users. See the rev_multilink attribute to make this easier.

Restricting updates that arrive by email

Roundup supports multiple update methods:

  1. command line

  2. plain email

  3. pgp signed email

  4. web access

in some cases you may need to prevent changes to properties by some of these methods. For example you can set up issues that are viewable only by people on the nosy list. So you must prevent unauthenticated changes to the nosy list.

Since plain email can be easily forged, it does not provide sufficient authentication in this senario.

To prevent this we can add a detector that audits the source of the transaction and rejects the update if it changes the nosy list.

Create the detector (auditor) module and add it to the detectors directory of your tracker:

from roundup import roundupdb, hyperdb

from roundup.mailgw import Unauthorized

def restrict_nosy_changes(db, cl, nodeid, newvalues):
    '''Do not permit changes to nosy via email.'''

    if 'nosy' not in newvalues:
        # the nosy field has not changed so no need to check.
        return

    if db.tx_Source in ['web', 'rest', 'xmlrpc', 'email-sig-openpgp', 'cli' ]:
        # if the source of the transaction is from an authenticated
        # source or a privileged process allow the transaction.
        # Other possible sources: 'email'
        return

    # otherwise raise an error
    raise Unauthorized( \
        'Changes to nosy property not allowed via %s for this issue.'%\
        tx_Source)

def init(db):
   ''' Install restrict_nosy_changes to run after other auditors.

       Allow initial creation email to set nosy.
       So don't execute: db.issue.audit('create', requestedbyauditor)

       Set priority to 110 to run this auditor after other auditors
       that can cause nosy to change.
   '''
   db.issue.audit('set', restrict_nosy_changes, 110)

This detector (auditor) will prevent updates to the nosy field if it arrives by email. Since it runs after other auditors (due to the priority of 110), it will also prevent changes to the nosy field that are done by other auditors if triggered by an email.

Note that db.tx_Source was not present in roundup versions before 1.4.22, so you must be running a newer version to use this detector. Read the CHANGES.txt document in the roundup source code for further details on tx_Source.

Changes to Security and Permissions

Restricting the list of users that are assignable to a task

  1. In your tracker’s schema.py, create a new Role, say “Developer”:

    db.security.addRole(name='Developer', description='A developer')

  2. Just after that, create a new Permission, say “Fixer”, specific to “issue”:

    p = db.security.addPermission(name='Fixer', klass='issue',
    description='User is allowed to be assigned to fix issues')

  3. Then assign the new Permission to your “Developer” Role:

    db.security.addPermissionToRole('Developer', p)

  4. In the issue item edit page (html/issue.item.html in your tracker directory), use the new Permission in restricting the “assignedto” list:

    <select name="assignedto">
 <option value="-1">- no selection -</option>
 <tal:block tal:repeat="user db/user/list">
 <option tal:condition="python:user.hasPermission(
                            'Fixer', context._classname)"
         tal:attributes="
            value user/id;
            selected python:user.id == context.assignedto"
         tal:content="user/realname"></option>
 </tal:block>
</select>

For extra security, you may wish to setup an auditor to enforce the Permission requirement (install this as assignedtoFixer.py in your tracker detectors directory):

def assignedtoMustBeFixer(db, cl, nodeid, newvalues):
    ''' Ensure the assignedto value in newvalues is used with the
        Fixer Permission
    '''
    if 'assignedto' not in newvalues:
        # don't care
        return

    # get the userid
    userid = newvalues['assignedto']
    if not db.security.hasPermission('Fixer', userid, cl.classname):
        raise ValueError('You do not have permission to edit %s'%cl.classname)

def init(db):
    db.issue.audit('set', assignedtoMustBeFixer)
    db.issue.audit('create', assignedtoMustBeFixer)

So now, if an edit action attempts to set “assignedto” to a user that doesn’t have the “Fixer” Permission, the error will be raised.

Users may only edit their issues

In this case, users registering themselves are granted Provisional access, meaning they have access to edit the issues they submit, but not others. We create a new Role called “Provisional User” which is granted to newly-registered users, and has limited access. One of the Permissions they have is the new “Edit Own” on issues (regular users have “Edit”.)

First up, we create the new Role and Permission structure in schema.py:

#
# New users not approved by the admin
#
db.security.addRole(name='Provisional User',
    description='New user registered via web or email')

# These users need to be able to view and create issues but only edit
# and view their own
db.security.addPermissionToRole('Provisional User', 'Create', 'issue')
def own_issue(db, userid, itemid):
    '''Determine whether the userid matches the creator of the issue.'''
    return userid == db.issue.get(itemid, 'creator')
p = db.security.addPermission(name='Edit', klass='issue',
    check=own_issue, description='Can only edit own issues')
db.security.addPermissionToRole('Provisional User', p)
p = db.security.addPermission(name='View', klass='issue',
    check=own_issue, description='Can only view own issues')
db.security.addPermissionToRole('Provisional User', p)
# This allows the interface to get the names of the properties
# in the issue. Used for selecting sorting and grouping
# on the index page.
p = db.security.addPermission(name='Search', klass='issue')
db.security.addPermissionToRole ('Provisional User', p)


# Assign the Permissions for issue-related classes
for cl in 'file', 'msg', 'query', 'keyword':
    db.security.addPermissionToRole('Provisional User', 'View', cl)
    db.security.addPermissionToRole('Provisional User', 'Edit', cl)
    db.security.addPermissionToRole('Provisional User', 'Create', cl)
for cl in 'priority', 'status':
    db.security.addPermissionToRole('Provisional User', 'View', cl)

# and give the new users access to the web and email interface
db.security.addPermissionToRole('Provisional User', 'Web Access')
db.security.addPermissionToRole('Provisional User', 'Email Access')

# make sure they can view & edit their own user record
def own_record(db, userid, itemid):
    '''Determine whether the userid matches the item being accessed.'''
    return userid == itemid
p = db.security.addPermission(name='View', klass='user', check=own_record,
    description="User is allowed to view their own user details")
db.security.addPermissionToRole('Provisional User', p)
p = db.security.addPermission(name='Edit', klass='user', check=own_record,
    description="User is allowed to edit their own user details")
db.security.addPermissionToRole('Provisional User', p)

Then, in config.ini, we change the Role assigned to newly-registered users, replacing the existing 'User' values:

[main]
...
new_web_user_roles = Provisional User
new_email_user_roles = Provisional User

All users may only view and edit issues, files and messages they create

Replace the standard “classic” tracker View and Edit Permission assignments for the “issue”, “file” and “msg” classes with the following:

def checker(klass):
    def check(db, userid, itemid, klass=klass):
        return db.getclass(klass).get(itemid, 'creator') == userid
    return check
for cl in 'issue', 'file', 'msg':
    p = db.security.addPermission(name='View', klass=cl,
        check=checker(cl),
        description='User can view only if creator.')
    db.security.addPermissionToRole('User', p)
    p = db.security.addPermission(name='Edit', klass=cl,
        check=checker(cl),
        description='User can edit only if creator.')
    db.security.addPermissionToRole('User', p)
    db.security.addPermissionToRole('User', 'Create', cl)
# This allows the interface to get the names of the properties
# in the issue. Used for selecting sorting and grouping
# on the index page.
p = db.security.addPermission(name='Search', klass='issue')
db.security.addPermissionToRole ('User', p)

Moderating user registration

You could set up new-user moderation in a public tracker by:

  1. creating a new highly-restricted user role “Pending”,

  2. set the config new_web_user_roles and/or new_email_user_roles to that role,

  3. have an auditor that emails you when new users are created with that role using roundup.mailer

  4. edit the role to “User” for valid users.

Some simple javascript might help in the last step. If you have high volume you could search for all currently-Pending users and do a bulk edit of all their roles at once (again probably with some simple javascript help).

Changes to the Web User Interface

Adding action links to the index page

Add a column to the item.index.html template. In that column add a form to trigger the action. Note: the form must use the POST method for security.

Resolving the issue:

<form method="POST" tal:attributes="action string:issue${i/id}">
  <button tal:replace="structure
    python:context.submit(label='resolve', action='edit')" />
  <input type="hidden" name="status" value="resolved">
</form>

“Take” the issue:

<form method="POST" tal:attributes="action string:issue${i/id}">
  <button tal:replace="structure
    python:context.submit(label='take', action='edit')" />
  <input type="hidden" name="assignedto"
    tal:attributes="value request/user/id">
</form>

… and so on.

Colouring the rows in the issue index according to priority

A simple tal:attributes statement will do the bulk of the work here. In the issue.index.html template, add this to the <tr> that displays the rows of data:

<tr tal:attributes="class string:priority-${i/priority/plain}">

and then in your stylesheet (style.css) specify the colouring for the different priorities, as follows:

tr.priority-critical td {
    background-color: red;
}

tr.priority-urgent td {
    background-color: orange;
}

and so on, with far less offensive colours :)

Editing multiple items in an index view

To edit the status of all items in the item index view, edit the issue.index.html:

  1. add a form around the listing table (separate from the existing index-page form), so at the top it reads:

    <form method="POST" tal:attributes="action request/classname">
<table class="list">

    and at the bottom of that table add:

     </table>
</form

    making sure you match the </table> from the list table, not the navigation table or the subsequent form table.

  2. in the display for the issue property, change:

    <td tal:condition="request/show/status"
   tal:content="python:i.status.plain() or default">&nbsp;</td>

    to:

    <td tal:condition="request/show/status"
   tal:content="structure i/status/field">&nbsp;</td>

    this will result in an edit field for the status property.

  3. after the tal:block which lists the index items (marked by tal:repeat="i batch") add a new table row:

    <tr>
 <td tal:attributes="colspan python:len(request.columns)">
  <input name="@csrf" type="hidden"
       tal:attributes="value python:utils.anti_csrf_nonce()">
  <input type="submit" value=" Save Changes ">
  <input type="hidden" name="@action" value="edit">
  <tal:block replace="structure request/indexargs_form" />
 </td>
</tr>

    which gives us a submit button, indicates that we are performing an edit on any changed statuses, and provides a defense against cross site request forgery attacks.

    The final tal:block will make sure that the current index view parameters (filtering, columns, etc) will be used in rendering the next page (the results of the editing).

Displaying only message summaries in the issue display

Alter the issue.item template section for messages to:

<table class="messages" tal:condition="context/messages">
 <tr><th colspan="5" class="header">Messages</th></tr>
 <tr tal:repeat="msg context/messages">
  <td><a tal:attributes="href string:msg${msg/id}"
         tal:content="string:msg${msg/id}"></a></td>
  <td tal:content="msg/author">author</td>
  <td class="date" tal:content="msg/date/pretty">date</td>
  <td tal:content="msg/summary">summary</td>
  <td>
   <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">
   remove</a>
  </td>
 </tr>
</table>

Enabling display of either message summaries or the entire messages

This is pretty simple - all we need to do is copy the code from the example displaying only message summaries in the issue display into our template alongside the summary display, and then introduce a switch that shows either the one or the other. We’ll use a new form variable, @whole_messages to achieve this:

<table class="messages" tal:condition="context/messages">
 <tal:block tal:condition="not:request/form/@whole_messages/value | python:0">
  <tr><th colspan="3" class="header">Messages</th>
      <th colspan="2" class="header">
        <a href="?@whole_messages=yes">show entire messages</a>
      </th>
  </tr>
  <tr tal:repeat="msg context/messages">
   <td><a tal:attributes="href string:msg${msg/id}"
          tal:content="string:msg${msg/id}"></a></td>
   <td tal:content="msg/author">author</td>
   <td class="date" tal:content="msg/date/pretty">date</td>
   <td tal:content="msg/summary">summary</td>
   <td>
    <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>
   </td>
  </tr>
 </tal:block>

 <tal:block tal:condition="request/form/@whole_messages/value | python:0">
  <tr><th colspan="2" class="header">Messages</th>
      <th class="header">
        <a href="?@whole_messages=">show only summaries</a>
      </th>
  </tr>
  <tal:block tal:repeat="msg context/messages">
   <tr>
    <th tal:content="msg/author">author</th>
    <th class="date" tal:content="msg/date/pretty">date</th>
    <th style="text-align: right">
     (<a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>)
    </th>
   </tr>
   <tr><td colspan="3" tal:content="msg/content"></td></tr>
  </tal:block>
 </tal:block>
</table>

Setting up a “wizard” (or “druid”) for controlled adding of issues

  1. Set up the page templates you wish to use for data input. My wizard is going to be a two-step process: first figuring out what category of issue the user is submitting, and then getting details specific to that category. The first page includes a table of help, explaining what the category names mean, and then the core of the form:

    <form method="POST" onSubmit="return submit_once()"
      enctype="multipart/form-data">
   <input name="@csrf" type="hidden"
      tal:attributes="value python:utils.anti_csrf_nonce()">
  <input type="hidden" name="@template" value="add_page1">
  <input type="hidden" name="@action" value="page1_submit">

  <strong>Category:</strong>
  <tal:block tal:replace="structure context/category/menu" />
  <input type="submit" value="Continue">
</form>

    The next page has the usual issue entry information, with the addition of the following form fragments:

    <form method="POST" onSubmit="return submit_once()"
      enctype="multipart/form-data"
      tal:condition="context/is_edit_ok"
      tal:define="cat request/form/category/value">

  <input name="@csrf" type="hidden"
      tal:attributes="value python:utils.anti_csrf_nonce()">
  <input type="hidden" name="@template" value="add_page2">
  <input type="hidden" name="@required" value="title">
  <input type="hidden" name="category" tal:attributes="value cat">
   .
   .
   .
</form>

    Note that later in the form, I use the value of “cat” to decide which form elements should be displayed. For example:

    <tal:block tal:condition="python:cat in '6 10 13 14 15 16 17'.split()">
 <tr>
  <th>Operating System</th>
  <td tal:content="structure context/os/field"></td>
 </tr>
 <tr>
  <th>Web Browser</th>
  <td tal:content="structure context/browser/field"></td>
 </tr>
</tal:block>

    … the above section will only be displayed if the category is one of 6, 10, 13, 14, 15, 16 or 17.

  1. Determine what actions need to be taken between the pages - these are usually to validate user choices and determine what page is next. Now encode those actions in a new Action class (see defining new web actions):

    from roundup.cgi.actions import Action

class Page1SubmitAction(Action):
    def handle(self):
        ''' Verify that the user has selected a category, and then move
            on to page 2.
        '''
        category = self.form['category'].value
        if category == '-1':
            self.client.add_error_message('You must select a category of report')
            return
        # everything's ok, move on to the next page
        self.client.template = 'add_page2'

def init(instance):
    instance.registerAction('page1_submit', Page1SubmitAction)

  2. Use the usual “new” action as the @action on the final page, and you’re done (the standard context/submit method can do this for you).

Silent Submit

When working on an issue, most of the time the people on the nosy list need to be notified of changes. There are cases where a user wants to add a comment to an issue and not bother other users on the nosy list. This feature is called Silent Submit because it allows the user to silently modify an issue and not tell anyone.

There are several parts to this change. The main activity part involves editing the stock detectors/nosyreaction.py file in your tracker. Insert the following lines near the top of the nosyreaction function:

# Did user click button to do a silent change?
try:
    if db.web['submit'] == "silent_change":
        return
except (AttributeError, KeyError) as err:
    # The web attribute or submit key don't exist.
    # That's fine. We were probably triggered by an email
    # or cli based change.
    pass

This checks the submit button to see if it is the silent type. If there are exceptions trying to make that determination they are ignored and processing continues. You may wonder how db.web gets set. This is done by creating an extension. Add the file extensions/edit.py with this content:

from roundup.cgi.actions import EditItemAction

class Edit2Action(EditItemAction):
  def handle(self):
      self.db.web = {}  # create the dict
      # populate the dict by getting the value of the submit_button
      # element from the form.
      self.db.web['submit'] = self.form['submit_button'].value

      # call the core EditItemAction to process the edit.
      EditItemAction.handle(self)

def init(instance):
  '''Override the default edit action with this new version'''
  instance.registerAction('edit', Edit2Action)

This code is a wrapper for the Roundup EditItemAction. It checks the form’s submit button to save the value element. The rest of the changes needed for the Silent Submit feature involves editing html/issue.item.html to add the silent submit button. In the stock issue.item.html the submit button is on a line that contains “submit button”. Replace that line with something like the following:

<input type="submit" name="submit_button"
       tal:condition="context/is_edit_ok"
       value="Submit Changes">&nbsp;
<button type="submit" name="submit_button"
        tal:condition="context/is_edit_ok"
        title="Click this to submit but not send nosy email."
        value="silent_change" i18n:translate="">
  Silent Change</button>

Note the difference in the value attribute for the two submit buttons. The value “silent_change” in the button specification must match the string in the nosy reaction function.

Changing How the Core Code Works

Changing Cache-Control Headers

The Client class in cgi/client.py has a lookup table that is used to set the Cache-Control headers for static files. The entries in this table are set from interfaces.py using:

from roundup.cgi.client import Client

Client.Cache_Control['text/css'] = "public, max-age=3600"
Client.Cache_Control['application/javascript'] = "public, max-age=30"
Client.Cache_Control['rss.xml'] = "public, max-age=900"
Client.Cache_Control['local.js'] = "public, max-age=7200"

In this case static files delivered using @@file will have cache headers set. These files are searched for along the static_files path in the tracker’s config.ini. In the example above:

  • a css file (e.g. @@file/style.css) will be cached for an hour

  • javascript files (e.g. @@file/libraries/jquery.js) will be cached for 30 seconds

  • a file named rss.xml will be cached for 15 minutes

  • a file named local.js will be cached for 2 hours

Note that a file name match overrides the mime type settings.

Implement Password Complexity Checking

This example uses the zxcvbn module that you can place in the zxcvbn subdirectory of your tracker’s lib directory.

If you add this to the interfaces.py file in the root directory of your tracker (same place as schema.py):

import roundup.password as password
from roundup.exceptions import Reject
from zxcvbn import zxcvbn

# monkey patch the setPassword method with this method
# that checks password strength.
origPasswordFunc = password.Password.setPassword
def mpPasswordFunc(self, plaintext, scheme, config=None):
    """ Replace the password set function with one that
        verifies that the password is complex enough. It
        has to be done at this point and not in an auditor
        as the auditor only sees the encrypted password.
    """
    results = zxcvbn(plaintext)
    if results['score'] < 3:
        l = []
        map(l.extend, [[results['feedback']['warning']], results['feedback']['suggestions']])
        errormsg = " ".join(l)
        raise Reject ("Password is too easy to guess. " + errormsg)
    return origPasswordFunc(self, plaintext, scheme, config=config)

password.Password.setPassword = mpPasswordFunc

it replaces the setPassword method in the Password class. The new version validates that the password is sufficiently complex. Then it passes off the setting of password to the original method.

Enhance Time Intervals Forms

To make the user interface easier to use, you may want to support other forms for intervals. For example you can support an interval like 1.5 by interpreting it the same as 1:30 (1 hour 30 minutes). Also you can allow a bare integer (e.g. 45) as a number of minutes.

To do this we intercept the from_raw method of the Interval class in hyperdb.py with:

import roundup.hyperdb as hyperdb
origFrom_Raw = hyperdb.Interval.from_raw

def normalizeperiod(self, value, **kw):
    ''' Convert alternate time forms into standard interval format

        [+-] [#y] [#m] [#w] [#d] [[[H]H:MM]:SS]

        if value is float, it's hour and fractional hours
        if value is integer, it's number of minutes
    '''
    if ":" not in value:
        # Not a specified interval
        # if int consider number of minutes
        try:
            isMinutes = int(value)
            minutes = isMinutes%60
            hours = (isMinutes - minutes) / 60
            value = "%d:%d"%(hours,minutes)
        except ValueError:
            pass
        # if float, consider it number of hours and fractional hours.
        import math
        try:
            afterdecimal, beforedecimal = math.modf(float(value))
            value = "%d:%d"%(beforedecimal,60*afterdecimal)
        except ValueError:
            pass

    return origFrom_Raw(self, value, **kw)

hyperdb.Interval.from_raw = normalizeperiod

any call to convert an interval from raw form now has two simpler (and more friendly) ways to specify common time intervals.

Modifying the Mail Gateway

One site receives email on a main gateway. The virtual alias delivery table on the postfix server is configured with:

test-issues@example.com  roundup-test@roundup-vm.example.com
test-support@example.com roundup-test+support-a@roundup-vm.example.com
test@support.example.com roundup-test+support-b@roundup-vm.example.com

These modifications to the mail gateway for Roundup allows anonymous submissions. It hides all of the requesters under the “support” user. It also makes some other modifications to the mail parser allowing keywords to be set and prefixes to be defined based on the delivery alias.

This is the entry in interfaces.py:

import roundup.mailgw
import email.utils

class SupportTracker(object):
    def __init__(self, prefix=None, keyword=None):
        self.prefix = prefix
        self.keyword = keyword

# Define new prefixes and keywords based on local address.
support_trackers = {
    ### production instances ###

    ### test instances ###
    'roundup-test+support-a':
        SupportTracker(prefix='Support 1', keyword='support1'),
    'roundup-test+support-b':
        SupportTracker(prefix='Support 2', keyword='support2'),
    'roundup-test2+support-a':
        SupportTracker(prefix='Support 1', keyword='support1'),
    'roundup-test2+support-b':
        SupportTracker(prefix='Support 2', keyword='support2'),
}

class parsedMessage(roundup.mailgw.parsedMessage):
    def __init__(self, mailgw, message, support_tracker):
        roundup.mailgw.parsedMessage.__init__(self, mailgw, message)
        if support_tracker.prefix:
            self.prefix = '%s: ' % support_tracker.prefix
        else:
            self.prefix = ''
        self.keywords = []
        if support_tracker.keyword:
            try:
                self.keywords = [
                    self.db.keyword.lookup(support_tracker.keyword)]
            except KeyError:
                pass
        self.config.ADD_AUTHOR_TO_NOSY = 'no'
        self.config.ADD_RECIPIENTS_TO_NOSY = 'no'
        self.config.MAILGW_KEEP_QUOTED_TEXT = 'yes'
        self.config.MAILGW_LEAVE_BODY_UNCHANGED = 'yes'
        self.classname = 'issue'
        self.pfxmode = 'loose'
        self.sfxmode = 'none'
        # set the support user id
        self.fixed_author = self.db.user.lookup('support')
        self.fixed_props = {
            'nosy': [self.fixed_author],
            'keyword': self.keywords,
        }

    def handle_help(self):
        pass

    def check_subject(self):
        if not self.subject:
            self.subject = 'no subject'

    def rego_confirm(self):
        pass

    def get_author_id(self):
        # force the support user to be the author
        self.author = self.fixed_author

    def get_props(self):
        self.props = {}
        if not self.nodeid:
            self.props.update(self.fixed_props)
            self.props['title'] = ("%s%s" % (
                self.prefix, self.subject.replace('[', '(').replace(']', ')')))

    def get_content_and_attachments(self):
        roundup.mailgw.parsedMessage.get_content_and_attachments(self)
        if not self.content:
            self.content = 'no text'
        intro = []
        for header in ['From', 'To', 'Cc']:
            for addr in self.message.getaddrlist(header):
                intro.append('%s: %s' % (header, email.utils.formataddr(addr)))
        intro.append('Subject: %s' % self.subject)
        intro.append('\n')
        self.content = '\n'.join(intro) + self.content

class MailGW(roundup.mailgw.MailGW):
    def parsed_message_class(self, mailgw, message):
        support_tracker = None
        # The delivered-to header is unique to postfix
        # it is the target address:
        #   roundup-test+support-a@roundup-vm.example.com
        # rather than
        #   test-support@example.com
        recipients = message.getaddrlist('delivered-to')
        if recipients:
            localpart = recipients[0][1].rpartition('@')[0]
            support_tracker = support_trackers.get(localpart)
        if support_tracker:
            # parse the mesage using the parsedMessage class
            # defined above.
            return parsedMessage(mailgw, message, support_tracker)
        else:
            # parse the message normally
            return roundup.mailgw.parsedMessage(mailgw, message)

This is the most complex example section. The mail gateway is also one of the more complex subsystems in Roundup, and modifying it is not trivial.

Other Examples

See the rest interface documentation for instructions on how to add new rest endpoints or change the rate limiting method using interfaces.py.

The reference document also has examples:

Examples on the Wiki

Even more examples of customisation have been contributed by users. They can be found on the wiki.