================== Developing Roundup ================== .. note:: The intended audience of this document is the developers of the core Roundup code. If you just wish to alter some behavior of your Roundup installation, see `Customising Roundup`_. Contents .. contents:: :local: Getting Started --------------- If you are looking for a good first issue, search for `StarterTicket on https://issues.roundup-tracker.org`_. These include issues where Python development, documentation or web design skills are useful. You can continue the conversation using the issue or join the Roundup-devel list to get assistance and verify your planned changes. All development is coordinated through two resources: - roundup-devel mailing list at https://sourceforge.net/projects/roundup/lists/roundup-devel - The issue tracker running at https://issues.roundup-tracker.org/ In addition, the Roundup IRC channel on irc.oftc.net can be accessed via the web interface shown on the Contact page. The channel is logged and the web sites for the logs are shown in the channel topic. You can ask questions and use it to coordinate work discussed using the resources above. Anyone wishing to help in the development of the Roundup Python core may find `Roundup's Design Document`_ and the `implementation notes`_ helpful. People working on Documentation or designing changes to the Web interface don't need to get into the implementation internals. Project Rules ------------- Be polite to others. There is no place for ad hominem attacks. Mostly the project follows Guido's Python Style (though naming tends to be a little relaxed sometimes). In short: - 80 column width code - 4-space indentations Other project rules: - new functionality must be documented, even briefly (so at least we know where there's missing documentation) and changes to tracker configuration must be logged in the upgrading document. - discuss any changes with the other developers on roundup-dev. If nothing else, this makes sure there's no rude shocks. - write unit tests for changes you make (where possible), and ensure that all unit tests run before committing changes. - run flake8_ or pylint_ over changed code. - if you have direct commit access to the repository, subscribe to roundup-checkins to receive checkin notifications from the other developers with write access to the source-code repository. The goal is to have no flake8 issues. Current code has complex functions, some long lines and use of mutable objects in function signatures. Some third party code (e.g. ZPT) vendored into the codebase has more issues. The administrators of the project reserve the right to boot developers who consistently check in code which is either broken or takes the codebase in directions that have not been agreed to. Setting up a Development Environment ------------------------------------ Roundup doesn't require any external libraries. Installing Python 3 and its core libraries is enough to get Roundup running. The easiest way to work with Roundup is to clone the repository. Roundup is developed using the `Mercurial distributed version control system (DVCS)`_ [1]_. It is `hosted at Sourceforge`_. See https://www.roundup-tracker.org/code.html for details. If you are used to git, Mercurial's `command equivalence table`_ can help. Most of the concepts from git (except for staging) should be familiar. To clone the repository use:: hg clone http://hg.code.sf.net/p/roundup/code roundup (Yes, that is an http url.) This will create a read only clone (you can't ``hg push``) of the repo. Changes you make can be attached as patches (created using ``hg diff``) to tickets in our `issue tracker`_. See https://www.roundup-tracker.org/code.html for URL's and directions on getting read write access which allows pushing to the repository. Once you have your clone, you can run ``python3 ./demo -b sqlite`` and get a working Roundup server. This will start the server using the ``sqlite`` backend. The code is in the ``roundup`` subdirectory. To test internationalization in your environment, you have to process the locale sub-directory into a form that roundup's i18n code will recognize. To do this use: make -C locale local_install which will compile the ``.po`` source files into binary ``.mo`` files and install them under ``locale/locale//LC_MESSAGES/roundup.mo``. For German this will be ``locale/locale/de/LC_MESSAGES/roundup.mo``. By default this requires ``msgfmt`` from the the GNU gettext tools installed on your system. If you do not have ``msgfmt`` installed, you can use: make -C locale local_install MSGFMT="python3 ../roundup/msgfmt.py" Also to compile a single .po file, you can invoke ``python roundup/msgfmt.py po_file mo_file`` to compile your .po file to a .mo file. It's slow but seems to work in basic tests. The msgfmt.py file is used by Roundup to compile out of date .po files. Submitting Changes ------------------ Most small changes can be submitted as patches through the `issue tracker`_ or sent to `Roundup-devel mailing list`_. Your account on sourceforge can be set up to allow direct pushes to the repo. Once that is done, using:: hg push https://user@hg.code.sf.net/p/roundup/code or hg push ssh://user@hg.code.sf.net/p/roundup/code will commit your changes. Other Resources - CI, Code Coverage ----------------------------------- Roundup has a `copy of the mercurial repository on GitHub`_. It is updated manually after every few commits to the Mercurial repository. Updates trigger the CI pipeline which happens on two services: 1. `GitHub Actions`_. It runs Docker container scans using Anchore as well as security scans for dependencies using CodeQL. Also it runs the test suite on multiple versions of Python. 2. `TravisCI`_ is also used to run CI. It runs the test suite on multiple Python versions. It also provides alpha and development Python releases faster than GitHub. GitHub actions upload coverage statistics to both `CodeCov`_ and `Coveralls`_. TravisCI only uploads to CodeCov. We run our own issue tracker so we can dogfood the code. As a result, we do not use GitHub issues. Pull requests are grudgingly accepted. They have to be exported and applied to the Mercurial repository. This is time consuming so patches attached to an issue in our tracker are preferred. Debugging Aids -------------- Try turning on logging of DEBUG level messages. This may be done a number of ways, depending on what it is you're testing: 1. If you're testing the database unit tests, then set the environment variable ``LOGGING_LEVEL=DEBUG``. This may be done like so: LOGGING_LEVEL=DEBUG python -m pytest test/ This variable replaces the older HYPERDBDEBUG environment var. 2. If you're testing a particular tracker, then set the logging level in your tracker's ``config.ini``. If you set the environment variable SENDMAILDEBUG to a filename, roundup will write each email message that it sends to that file instead to the internet. This environment variable is independent of the python -O flag. Documentation Notes ------------------- All docs are written using sphinx. Use pip or your package manager to install sphinx. In addition you will need to install a couple of sphinx extensions to process the files. A requirement's file can be found in docs/requirement.pip or website/www/requirements.pip. You can install these requirements using ``python -m pip install -r ``. The extensions sphinx-sitemap generates a sitemap when building for the website. The sphinx-tabs extension generates tabs for displaying code examples in both TAL and Jinja2. To build the documentation distributed with a Roundup release, run ``python setup.py build_doc`` at the root of the source tree. To build docs for the website, see the ``updating www.roundup-tracker.org`` section of ``website/README.txt`` or TL;DR ``cd website/www; make html``. Testing Notes ------------- Create tests for your changes. Also run the tests in reverse to try to identify test dependencies. You can do this by creating a ``conftest.py`` in the top of the source tree and include the following contents:: def pytest_collection_modifyitems(items): items.reverse() to run all the tests in reverse. More tips at: https://testmon.org/blog/hidden-test-dependencies/. The full test suite can take a while to run. The `pytest-testmon `_ package can be installed with pip. It analyzes changes to code and run tests that test that code. This can significantly reduce the time it takes to run a basic test suite. Use it by running:: python3 -m pytest -v --testmon test once over the whole test suite. Then subsequent calls will analyze the changed files/functions and run tests that cover those changes. To run some tests (test_liveserver.py, test_indexer.py, ...) you need to have some additional modules installed. These include: * requests * mock If you are working with a docker container that is set up to execute Python application and not for development, you will need to install pytest. Internationalization Notes -------------------------- How stuff works: 1. Strings that may require translation (messages in human language) are marked in the source code. This step is discussed in `Marking Strings for Translation`_ section. 2. These strings are all extracted into Message Template File ``locale/roundup.pot`` (_`POT` file). See `Extracting Translatable Messages`_ below. 3. Language teams use POT file to make Message Files for national languages (_`PO` files). All PO files for Roundup are kept in the ``locale`` directory. Names of these files are target locale names, usually just 2-letter language codes. `Translating Messages`_ section of this chapter gives useful hints for message translators. 4. Translated Message Files are compiled into binary form (_`MO` files) and stored in ``locale`` directory (but not kept in the source code repository, as they may be easily made from PO files). See `Compiling Message Catalogs`_ section. 5. Roundup installer creates runtime locale structure on the file system, putting MO files in their appropriate places. 6. Runtime internationalization (_`I18N`) services use these MO files to translate program messages into language selected by current Roundup user. Roundup command line interface uses locale name set in OS environment variable ``LANGUAGE``, ``LC_ALL``, ``LC_MESSAGES``, or ``LANG`` (in that order). Roundup Web User Interface uses language selected by currently authenticated user. Additional details may be found in `GNU gettext`_ and Python `gettext module`_ documentation. Additional information on ``.po`` translation files can also be found at pology_. `Roundup source distribution`_ includes POT and PO files for message translators, and also pre-built MO files to facilitate installations from source. Roundup binary distribution includes MO files only. .. _GNU gettext: GNU gettext package ~~~~~~~~~~~~~~~~~~~ This chapter is full of references to GNU `gettext package`_. GNU gettext is a "must have" for nearly all steps of internationalizing any program, and it's manual is definitely a recommended reading for people involved in `I18N`_. There are GNU gettext ports to all major OS platforms. Windows binaries are available from `GNU mirror sites`_. Roundup does not use GNU gettext at runtime, but it's tools are used for `extracting translatable messages`_, `compiling message catalogs`_ and, optionally, for `translating messages`_. Note that ``gettext`` package in some OS distributions means just runtime tools and libraries. In such cases gettext development tools are usually distributed in separate package named ``gettext-devel``. Marking Strings for Translation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Strings that need translation must be marked in the source code. Following subsections explain how this is done in different cases. If translatable string is used as a format string, it is recommended to always use *named* format specifiers:: _('Index of %(classname)s') % locals() This helps translators to better understand the context of the message and, with Python formatting, remove format specifier altogether (which is sometimes useful, especially in singular cases of `Plural Forms`_). When there is more than one format specifier in the translatable format string, named format specifiers **must** be used almost always, because translation may require different order of items. It is better to *not* mark for translation strings that are not locale-dependent, as this makes it more difficult to keep track of translation completeness. For example, string ```` (in ``index()`` method of the request handler in ``roundup_server`` script) has no human readable parts at all, and needs no translations. Such strings are left untranslated in PO files, and are reported as such by PO status checkers (e.g. ``msgfmt --statistics``). Hints for the translator can be added by adding a comment block starting with ``# .Hint``. The hints will be extracted into the .pot file and merged to the .po files used by the translators. This can add context to how the string is used. For example from admin.py:: # .Hint read as: assignedto was admin(1) # where assignedto is the property # admin is the key name for value 1 _("%(prop)s was %(name)s(%(value)s)") is shown in the .po file as:: #. .Hint read as: assignedto was admin(1) #. where assignedto is the property #. admin is the key name for value 1 #: ../roundup/admin.py:1091 #, python-format msgid "%(prop)s was %(name)s(%(value)s)" msgstr "" allowing the translator to see the replaced string so they can understand how name and value are used. Command Line Interfaces ~~~~~~~~~~~~~~~~~~~~~~~ Scripts and routines run from the command line use "static" language defined by environment variables recognized by ``gettext`` module from Python library (``LANGUAGE``, ``LC_ALL``, ``LC_MESSAGES``, and ``LANG``). Primarily, these are ``roundup-admin`` script and ``admin.py`` module, but also help texts and startup error messages in other scripts and their supporting modules. For these interfaces, Python ``gettext`` engine must be initialized to use Roundup message catalogs. This is normally done by including the following line in the module imports:: from i18n import _, ngettext Simple translations are automatically marked by calls to builtin message translation function ``_()``:: print(_("This message is translated")) Translations for messages whose grammatical depends on a number must be done by ``ngettext()`` function:: print(ngettext("Nuked %i file", "Nuked %i files", number_of_files_nuked)) Deferred Translations ~~~~~~~~~~~~~~~~~~~~~ Sometimes translatable strings appear in the source code in untranslated form [#note_admin.py]_ and must be translated elsewhere. Example:: for meal in ("spam", "egg", "bacon"): print(_(meal)) In such cases, strings must be marked for translation without actual call to the translating function. To mark these strings, we use Python feature of automatic concatenation of adjacent strings and different types of string quotes:: strings_to_translate = ( ''"This string will be translated", ""'me too', ''r"\raw string", ''""" multiline string""" ) .. [#note_admin.py] In current Roundup sources, this feature is extensively used in the ``admin`` module using method docstrings as help messages. Web User Interface ~~~~~~~~~~~~~~~~~~ For Web User Interface, translation services are provided by Client object. Action classes have methods ``_()`` and ``gettext()``, delegating translation to the Client instance. In HTML templates, translator object is available as context variable ``i18n``. HTML templates have special markup for translatable strings. The syntax for this markup is discussed at `ZPTInternationalization`_. (Originally documented at http://dev.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/ZPTInternationalizationSupport which is now gone.) Roundup translation service currently ignores values for ``i18n:domain``, ``i18n:source`` and ``i18n:target``. Template markup examples: * simplest case::
Say no more!
this will result in msgid ``"Say no more!"``, with all leading and trailing whitespace stripped, and inner blanks replaced with single space character. * using variable slots::
And now...
No.
THE LARCH
Msgid will be: ``"And now...
No.${slideNo}
THE LARCH"``. Template rendering will use context variable ``number`` (you may use any expression) to put instead of ``${slideNo}`` in translation. * attribute translation::