Development

Abstract

This document describes the development process of rs.

Communication

The main venues for developer communication are twofold:

• IRC: #rs on irc.freenode.net
• Mailing list: projects.kittensoft.org/mailman/listinfo/rs-dev_projects.kittensoft.org

  This is the developer mailing list (users will have a separate one) and the method by which patches are submitted.

The distribution contains a file doc/AUTHORS. Please add yourself there in the same format as the already-existing entries. Of course, it would be best to do this only when you have contributed some code :)

Organization

• rs uses a distributed version control system called darcs. A thorough introduction to it follows.
• Rake is used as the build tool and in automating certain developer processes. More on this later also.
• Test::Spec is the preferred testing framework but Test::Unit is fine as well.
• The directory layout follows the Ruby standard. The main directory branches to doc/ for all documentation, bin/ for executables (such as the rs program itself) lib/ for library files—and underneath it rs/ since we do not want to clutter the global requirespace—and lastly but not leastly, test/ for all tests and specs.

The easiest way to get yourself up to speed with the code is to take a look at the files in test/ since they describe what the various classes and methods are supposed to do. If you have any questions, head for IRC or the mailing list and we can update this guide if necessary!

The basic idea is very simple: ui.rb maintains a user interface that prompts and waits for input, passes that input to parser.rb and then prints the output passed back.

The parser simply tries to determine if it has gotten a complete piece of Ruby code and, if so, evals it.

Everything else is simply Ruby code that the user can access through the UI.

Rules

The development model will be as relaxed as possible. However, to make sure everything works smoothly, there are some rules that must be adhered to:

• Changelog. Every patch you submit must be accompanied by a decent log entry in the darcs patch. The easiest way to make sure you have one ready is to actually just have a file open constantly during development (I call mine CHANGES) and jot down notes on it as you code. Then, at the time of recording a patch, you can simply use that file for the log.
• Tests. Every patch you submit must be reasonably tested. Generally speaking, each behaviour or method your patch incorporates must be tested to operate as desired. Any patch submitted must not break existing tests nor may it have broken tests itself. If a bug is reported, the first thing to do is to write a spec or test that reproduces it and the second is to fix it.
• Coding standards. Nothing huge here, standard Ruby style (snake_case, not camelCase). Every file, every class/module and every method should have some sort of a comment an the code itself should be commented to make its intent clear.

Development

Introduction

darcs is a version control system, similar in purpose to for example CVS and Subversion, though its philosophy is quite different from those two, mainly in that it is a ‘distributed’ VCS. Essentially this means that each developer works on a completely independent set of source code.

The reason rs uses darcs is simply that it is one of the most flexible version control systems. It will be used until and if something better should come along.

This is an attempt to describe the essentials of using darcs for the purposes of developing rs. For an authoritative resource on darcs, please visit www.darcs.net. The manual will tell you everything you may desire to know. Also, ‘darcs -h’ and ‘darcs <command> -h’ give information right from the command-line.

For testing, we use the Test::Spec framework from cneukirchen.org which layers Behaviour-driven Design (BDD) over the familiar Test::Unit in the standard library. Using plain Test::Unit is fine, too. For information regarding BDD, see behaviour-driven.org and rspec.rubyforge.org.

The test/ directory contains all current tests and is a good place to start if you want to find out how rs is being tested.

If you have any questions about darcs or you are unfamiliar with or unsure about test/behaviour-driven development, please do not hesitate to ask.

The ‘best-practices’ development cycle using these tools is presented in the following chapters.

Installation

rs is intended to work on any UNIX platform, and of course a working Ruby 1.8.4+ installation is required.

(I develop on an amd64 FreeBSD 6-RELEASE with Ruby 1.8.5 —rue.)

darcs is written in Haskell, which means that most users will choose to install a precompiled binary (though if you have the right tools, by all means feel free to compile it yourself). Instructions and binaries for most operating systems can be found here:

darcs.net/DarcsWiki/CategoryBinaries

Once you have installed it, type ‘darcs’ on your command-line to make sure everything works.

Both Test::Spec and FlexMock are included in the test/vendor/ directory in the project so it is not necessary to download anything extra.

Preparations

Select the directory in which you want to set up and create a suitable subdirectory, for example /home/name/code/rs. This directory will be called $rs from now on for brevity.

Getting rs for the first time

cd into $rs. The current development repository for rs is located at repos.kittensoft.org/rs and this should be considered to be the authoritative development master repository. The command to get yourself a local repository to start working on is:

    darcs get http://repos.kittensoft.org/rs master

This will create the directory $rs/master with the current source. Take a moment to cd around in the source to look over the directories and files. The reason for calling your repository ‘master’ and having it be a subdirectory of $rs is to mainly simplify making changes. You will see why once we get to Making Changes.

Currently it is possible that the version from the above URL is not the most current version in the sense that there may still be some patches that have not made it there yet. With darcs this is never a problem and at worst the URL will behind by a day‘s worth of patches. To learn how to update your copy, read ahead.

Learning from history

Now that you have a local repository, you can investigate it a bit. The simplest way to do this is the changes command (make sure you are in $rs/master):

    darcs changes

This will give you a newest-oldest list of patches and patch log notes for the life of the repository in a fairly easy-to-read form. You can give the —reverse or —last=N switches to change the output a bit. Another useful (but very verbose) command is

    darcs annotate

Which annotates a specific file or the entire source with data such as the who and when for a certain line in a source file.

    darcs help
    darcs help changes
    darcs help annotate

Will give you more information. —xml-output and —summary are among the more useful switches, too.

Configuration

There are few things that you should do before really starting to work on the rs source. Essentially we just need to be able to identify you and this is achieved with two environment variables you should set in your shell‘s rc-file.

    RS_NAME
    RS_EMAIL

The nickname is up to you, just check the doc/AUTHORS file to make sure no-one has taken it already. It would be handy if you used your IRC nick here to make it simpler to recognize folks on ==rs.

Ornery though it may sound, you definitely should include your real name in RS_EMAIL. You are of course free to leave it out too. Depending on your choice, RS_EMAIL takes one of the following standard formats:

    RS_EMAIL='email@address.net'
    RS_EMAIL='Real Name <email@address.net>'

In addition to this, darcs allows some configuration either using environment variables, user-specific changes or repository-specific changes. You can read up on these on www.darcs.net.

One environment variable you should set is DARCS_EDITOR. This should be the name of the text editor that you want to use when entering text darcs occasionally needs (log comments, for example), otherwise you may get an unfamiliar editor (and, boy, is there anything more profoundly terrifying than that?). You should use a textmode editor like vi, vim, emacs, nano, pico, joe, ed or something similar.

There may be certain preference that you want to persist either over all your repositories or only certain ones. For these, use either ~/.darcs/prefs or $REPO/_darcs/prefs. My ~/.darcs/prefs, for example, contains a file named defaults, the contents of which are options (like —verbose) that I always tend to use: verbose, edit-long-comment, compress. Each should be on its own line and without the preceding —. To find out more about the configuration, see the manual on the web.

Updating your copy

All rs contributors including rue currently share their changes by sending e-mail patches to the main rs mailing list (see Communication). Plans are to eventually switch to a completely web-based system in the future.

Important: always apply any ‘outside world’ patches to your master repository first. More on this later.

Once you have done your initial pull from the main repository, it is simple to have your local repository updated from the same source. Just cd back to $rs/master and issue the following command (darcs automatically pulls from the repository this one was gotten from):

    darcs pull

Generally, this should be enough when updating. Once your local master is up-to-date, you can propagate the changes to your working directories.

It is absolutely not necessary but if you want to be on the bleeding-most edge of patches, you can get those from the mailing list as people send them in. All patches should be identified by an initial ‘darcs patch:’ on their subject line. The patches will be propagated to the master repository so you will be eventually able to just update to them with a pull. It is not usually necessary to have the most recent changes when starting a new patch because they can be merged in at any point.

If you do want to apply the e-mailed patches, the process is simple. Just save the patch files somewhere, cd to $rs/master and:

    darcs apply --verbose /path/to/patchfile

This will merge the code in the patch with the code that you have in your source directory, hopefully creating a new version with both developers’ changes incorporated. Occasionally incompatible changes occur, causing a conflict. You will be notified where these are if you have the —verbose option on. You are given a file name, and the file itself will contain both changes indicated with special syntax. When this occurs, it is your responsibility to revise the code to get it working with both changes. If you are unable to do so, try to contact the other developer to find out how you two can best resolve the issue. This, of course, happens fairly rarely in practice: most conflicts will be easy to resolve.

Making changes

Before we get to actually making changes, there is one important technique to make changing the code much easier.

There is a tendency for a programmer to do multiple, often unrelated, things (code-wise) at one time and the "hey, let me add this now that I thought of it" apropos fixes and enhancements are prevalent. Mind you, there is nothing wrong with fixing stuff immediately (in fact, it is quite encouraged), but there are three reasons:

1. Bugfixes or new features do not need to wait other work to finish.
2. Keeping track of—and possibly choosing—patches is much easier.
3. The repository works better with smaller changesets.

For example, if you are working on a new VFS feature and think of an unrelated fix in the parser, you would not want to include both of those in the same patch.

So, to avoid these problems, try to always keep your changesets as small and strictly related as possible —while still ensuring that the code as a whole works after applying that change. For example, you could consider one task or changeset to be 1) adding a new command, 2) fixing one line in all existing commands or 3) reformatting all documentation files. These are fine by themselves whether they use one file or everything in the repository because they are logically grouped. You would not want to mix any two of those together, though.

Wanting to do multiple things at once is unavoidable, though, and this is where darcs will help you. Its distributed nature means that there is no central repository (other than what the developers amongst themselves decide); each is equal in its ‘capabilities’. We can further extend this idiom by, each individual developer, maintaining one ‘master’ repository and multiple ‘working’ repositories (this is why we have $rs/master).

The master repository should be your window to the world and the other developers: it is where you update others’ code from the web and it is where you send your updates to others. The working repositories, on the other hand, are where you write your code. To illustrate the point, assume you want to brush up some of these documentation pieces. First you need to create the working copy. cd into your $rs and then:

    darcs get master docfix

This creates a new repository docfix (at this time, an exact replica of master) just like you created master from the ‘actual’ master repository on the web. Then, just start making your changes in docfix.

If somewhere along the way you get, say, a bug report from a user and immediately know how to solve it, instead of just starting to work on the fix in docfix, you should create another repository just for that purpose. Then you can work on the two independently.

    cd $rs
    darcs get master bugfix

Once you have completed and recorded the change you intended (more of which below), you can transfer it back to your master repository. Before doing that, however, you should make sure that the master and your working repository have the same code (other than your new changes) unless you know you have not otherwise updated master from either the web, mailing list or one of your other working repositories (again, darcs knows where you originally pulled from). Let us assume you got the bugfix done right away.

    cd $rs/bugfix
    darcs pull

This will merge all changes in master that are not in your working repository. After you have merged and resolved any conflicts, you are ready to send in your changes. For this, still in $rs/bugfix, issue:

    darcs push

That is all. Your bugfix changes are now incorporated into master -in fact, it is exactly as if you had made the changes in ‘master’. Then just delete the bugfix/ repository (rm -Rf is fine here).

Then, since $rs/master now contains code that $rs/docfix does not have (the fixed bug), you can cd to $rs/docfix and issue a darcs pull to get that change incorporated in that repository too.

If there is a situation where two branches may need to be developed simultaneously (say, 0.2 for new features and 0.1 for bugfixes and backported improvements), instead of a $rs/master, one might have $rs/0.1/master and $rs/0.2/master. Such a situation is unlikely to happen in this project, though.

Before delving into recording changes, let us take a brief interlude of testing information.

Testing

Testing is an important part of the rs development cycle—so much so that it is automated through rake and is considered to be a prerequisite to being able to create a patch.

The main benefit of testing is of course a greater confidence in the correctness of the code but there are important auxiliary considerations such as being immediately able to see whether a change or addition of code breaks anything and the ability to easily codify the bugfixing process—reproduce the bug as a failing test and then correct the implementation so that the test passes.

The preferred framework is Test::Spec, but Test::Unit tests are also OK. The former live as $rs/test/*_spec.rb and the latter as $rs/test/*_test.rb. Both are automatically run by the Rake task.

Each of your methods—internal and public—should be tested at least for correct behaviour.

Behaviour-driven development is based on the idea that instead of writing a method and then testing its functionality you will first write the specification of how the method should behave in the form of a test and then implement a method that fulfills that specification, piece by piece, and examine the objects and methods in different contexts.

It is strongly recommended that you take the practice of writing tests first when developing rs. With this, you will have greater confidence in your own code.

There is a lot of literature for TDD and BDD both. If you have questions, please do ask. The test/ directory may also serve as a good introduction to how testing happens.

Back to Making Changes

Alright, ready to make changes? The coding itself is, hopefully, a familiar process to you; once you have made the necessary changes, you are ready to record them. It is recommended that you record your changes in the smallest chunks possible while still making sure that the code works. Since darcs is, again, a distributed system, at this stage you are only making changes to your own repository using the ‘record’ command (akin to ‘commit’ in CVS). You will see how to make your changes available to others later on.

(You can find out what has changed since the last ‘record’, incidentally, by typing ‘darcs whatsnew’. The output is fairly verbose so you may want to give the —summary option to it.)

darcs is fairly smart about file changes in the repository. There are three common scenarios:

• Adding files. darcs does need to be notified about new files that have been added. The command for this is, logically enough, ‘darcs add x’ where ‘x’ may be an individual file or a directory (in which case the files are added recursively). If you use scrap files (such as CHANGES), do not add those, that way they do not propagate to other users.
• Deleting files. darcs automatically detects when you have deleted a file in the repository directory. If, however, you want to keep the physical file but remove it from the repository, you can use the ‘darcs remove x’ command.
• Moving files. You should use the command ‘darcs mv x’ to move files and directories in the repository. Be aware that this *actually moves the file* so it is not necessary (or recommended) for you to first move the file and then try to tell about it to darcs. Similar to the standard UNIX mv, darcs mv can also be used for renaming files.

Changes within files are, of course, automatically recorded. Once you have made all your changes and updated the repository to correctly reflect the files it should contain, you are ready to execute this command:

    rake record

The Rakefile at the top-level of each repository is configured to do a few things with this so you need not worry about it. First it checks your developer credentials, then it runs all tests (and aborts if any fail), after which the actual patch creation happens. The patch name is generated automatically and you should find yourself in $DARCS_EDITOR with a patchlog file open. This file has a specific format: the first line is the patch name—leave it as-is. The second line and any there- after until the separator line are considered to be the patch log comment. Here you should enter a hyphenated list of the changes you have made (no need to go into much detail, general list is fine). Once done, your file looks something like this:

    2006.09.18-rue-02
    - Created doc/HOWTO.development
    - Made some other change
    ***END OF DESCRIPTION***

    <Some autogenerated text here>

If you happen to record a change that you notice should not have been recorded, you can use the command ‘darcs unrecord’, just be careful to not unrecord too many changes :)

Sharing your changes

Once you have recorded some properly tested, working changes, you should make those available for other developers so that everyone is on the same page. Before doing this, there are two things to do:

First, make sure that you have updated your $rs/master repository with the changes from any of the working repositories containing the code you want to release this time. Then cd to $rs/master.

The second thing to do is to ensure you have the most recent web copy of rs (you can apply any e-mail patches too but this is not necessary) by doing a darcs pull. Merge in all changes to your local master and make sure nothing breaks by running all tests. This should be automated with rake shortly.

At this time, the proper method of sharing the changes is by e-mail (this will change in the near future). If you have sendmail or similar installed, darcs can actually send the mail itself but you can also just save the patch bundle into a file and use it as an attachment if you prefer to do your mailing lists through GMail or something. The commands, respectively, are:

   darcs send --to rs-dev@projects.kittensoft.org
   darcs send --output <filename>

Darcs will be asking you which patches you want to send. Usually, of course, you will only be sending the most recent one since you have already sent all previous ones, right? :)

Conclusion

That was it! Following these instructions and taking a look at the linked web pages, you should be able to start development. If you do have any questions, the IRC channel and mailing list are there just for that.