The 8th Darcs Hacking Sprint happened on 15-17th February in Paris, at IRILL like in 2011. This sprint occured one week after the latest stable release (2.8.4), and after a process of integrating many new features to the HEAD repository of darcs: rebase, the patch index optimization, the last regrets prompt, and a lot of refactoring in the code base. We are currently looking at the next important milestone for darcs: the release of 2.10, that should happen sometimes this year. This sprint was about polishing as much as possible these new features and take a few short and medium-term decisions.
This time we had five people attending: Florent Becker, Ganesh Sittampalam, Guillaume Hoffmann, Owen Stephens and Radoslav Dorcik. We also had a short visit of Pierre-Yves David, a Mercurial hacker.
Sprinters' Backlog
Ganesh mainly worked on rebase in preparation for the upcoming 2.10 release: he resolved the issue 2282 and 2227, and started work on a `darcs rebase changes` command. He also made various minor code cleanups, including getting the unit tests back to green which will hopefully encourage people to run them in future.
Radoslav worked on a couple of ProbablyEasy bugs (darcs obliterate -O overwrites existing files and implement `darcs rebase unsuspend --summary`). He created a wiki page to help us think about a future darcs flags overhaul. Last day he wrote a prototype patch on the issue make darcs amend -A use the default author id,
which he will continue to work on after sprint along with elaboration of big overhaul of the darcs command line flags.
With Guillaume he also worked on the manual and completed the help for environment variables and the output of `darcs help markdown`. We decided that http://darcs.net/manual should always have the manual corresponding to the current stable branch of darcs (as of now 2.8), because sometimes commands and flags change. We thus continue the process of moving away the documentation from literate haskell and latex and to have everything in markdown (website documentation and darcs-generated help). Guillaume also fixed bugs in the testsuite, in particular network tests.
Florent worked around the UI/Selectchanges code, with three aims:- make the code clearer, and more easily usable by a gui / web ui (in progress)
- allow the user to preview dependencies (done) and conflicts (todo) before selectenig a patch
- adding darcs diff --interactive and darcs trackdown --interactive (interactive choice of a non-contiguous set of patches to search among.) unleashing the power of implicit branching! (in progress)
The current state of the branch is available on hub.darcs.net, and is subject to agressive rebases (as witnessed by the number of "brouillon" patches).
Owen spent the weekend of the sprint working on darcs-bridge. Spending time getting the theory right for exporting merges, including some tricky corner cases. He implemented a proof-of-concept of the new exporter and has started to integrate it back into the bridge. With lots of help from Ganesh (thanks!), he discussed and worked through most of the difficult implementation points.
Last regrets for 2.10
We decided that 2.10 will contain the last regrets prompt (an extra final question "Do you want to push/pull these patches? [yn...]" ) in its current form.
Darcs on sshfs
We finally closed http://bugs.darcs.net/issue904 , accepting a patch that makes darcs work on sshfs-mounted directories. This, combined with the bare repositories introduced in 2.8, will make darcs easier to work with dumb servers, i.e., ssh-accessible servers which do not have darcs installed.
Google Summer of Code and 2.10 beta
We plan to apply to this year's Google Summer of Code as an independent organization. This means we will ask for two slots. We discussed possible projects. Organizations sumbissions for GSoC are at mid-march, so we estimated a beta release of Darcs 2.10 would make sense by then.
Mercurial's Changesets Evolutions
Pierre-Yves David, a Mercurial developer, came to see us and gave us a short version of his FOSDEM talk "Changesets evolutions with Mercurial". Changesets evolution is a feature that recently made its way into Mercurial, that enables automatic merging of rewritten histories. We discussed similarities and differences with the way darcs commutes patches and how `darcs rebase` works.
IRC Sprinters
We also counted with a good participation on #darcs. Eric Kow did intensive bug triaging on the tracker. Mark Stosberg worked on stabilizing two features: rebase and patch index. Iago Abal helped Ganesh improve the patch code unit tests. Simon Michael updated darcsden (the software behind hub.darcs.net) to the latest API of libdarcs, which changed before and during the sprint. Petr Rockai kept us up to date about the state of the buildbot infrastructure.
Meatspace sprinters
Thanks!
We would like to thank the generous people and organizations that made this sprint possible:
- the IRILL for kindly hosting the sprint again.
- all our donators, that help afford the prices of travelling and accomodations for the sprinters.
Here's a short shell script that I use when reviewing patches from the Darcs
bugtracker, roundup.
I often review patches by following simple steps:
- Downloading the submitted dpatch file.
- Grabbing a copy of the "screened" repo - containing all "quick sanity
checked" submitted patches - and removing patches not in the dpatch's
context.
- Applying the dpatch.
I can then check the implementation of the patch, and screen away!
Of course, I noticed myself redoing these three steps several times, before
finally submitting and hacking some bash:
#! /bin/bash
# Exit on any failures.
set -e
PATCH_NAME=downloaded_patch.dpatch
REPO_NAME=darcs.net_with_downloaded_patch
# I often do this in /tmp, so the target may already exist...
rm -rf $REPO_NAME $PATCH_NAME
# Grab the patch.
wget $1 --output-document $PATCH_NAME
# No need to copy the patch contents of the screened repo.
darcs get --lazy --context $PATCH_NAME http://www.darcs.net $REPO_NAME
cd $REPO_NAME
# Apply the submitted dpatch.
darcs apply ../$PATCH_NAME
# echo the directory, so we can pipe the output into cd, for example.
echo $REPO_NAME
This week, I have added support to user interrupt at patch index creation,
cleaned up timing and correctness scripts, run timing and correctness scripts
on xmonad repository, extended and cleaned up tests and documentation.
User Interrupt The user can interrupt darcs get with Ctrl-C. The rest of the patches are downloaded on when needed. However, the entire set of patches are required to build patch index. Now, when the user interrupts get with Ctrl-C, patch index will be disabled, and get will stop immediately.
Patch index will be created automatically when patch index darcs is run on an existing repo. Now, the user can interrupt this with Ctrl-C. patch index will be disabled, and the command will continue.
Timing and Correctness Scripts I have written bash timing and correctness scripts. The files are in then contrib folder of my darcs patch index repository.
patch-index-correctness.sh REPO_URL, gets the darcs repo at REPO_URL, and compares the patch index/existing changes and annotate output for every file in the directory.
patch-index-timing.sh REPO_URL DEV_FILE MOUNT_POINT, mounts the partition of DEV_FILE at MOUNT_POINT, gets the repo at REPO_URL there, and measures the time taken by patch index/existing changes and annotate, with and without clearing the disk cache. This file needs to be run as root.
I have run these scripts on xmonad repository. The correctness test passed, and the result for the timing tests are here:
changes, and
annotate. Annotate is 63(cold cache)-74(hot cache)% faster, and changes is 45(cold cache)-78(hot cache)% faster on average. This is comparatively slower than on darcs repository(changes 78-92% faster, and annotate 86-94% faster). This is mostly because the number of patches in darcs is larger. A larger repository will get larger gains. I am going to run the timing test on ghc and agda repositories this week.
Next week I will finish up documentation and elaborate and add more tests.
My darcs repository is
here.
This week, I have improved tests and documentation on patch index structures, completed timing tests on changes and annotate, and made some ui fixes.
Patch index structures Patch index maintains four data structures. I have written a subcommand to test the logical consistency of the structures. Invoke it using "darcs show patch-index-test". This subcommand can run on any darcs repository with patch index. I have tested darcs development repository and a few test repositories using this command, and found that it passed all tests.
I have expanded the shell test on patch index structures. The shell test creates a new darcs repository, and uses the additional knowledge of the exact steps involved to test the structures more thoroughly.
As per Eric's suggestion, I have added examples of the contents of these structures to the documentation.
Timing Tests I have completed the timing tests of changes and annotate. The time taken by changes or annotate varies according to the contents of the disk cache of the operating system. I have measured both extremes of the time taken, with and without using patch index. On average, patch index saves 78%-92% of time taken on changes, and 86%-94% on annotate.
The exact results for
changes and
annotate are here.
UI Fixes I have added support to Ctrl-C at get. Now, if you stop getting the patches in the middle of get, patch index will be disabled.
I have renamed --no-patch-index to the more descriptive --disable-patch-index.
Next week, I will:
- Port latest changes to Eric's newest rebase: Eric has just rebased the patch index repository. However, there are a few new changes I have made after that. I will port the new changes, and start using the rebase.
- Support Ctrl-C at patch index creation on existing repositories: Patch index will be automatically created when patch index darcs is run on a existing repository. I intend to allow the user to interrupt in the middle, if he wishes to disable patch index.
- Fill missing Haddok documentation for some patch index functions.
- Document the various tests on patch index.
The latest patch index repository is at: http://darcsden.com/bsrkaditya/darcs-patch-index
This week I have wrote a patch to fix annotate in screened, did timing tests with patch index on clean disk caches, wrote a shell test on patch index internal structures, and made some changes to the UI.
Annotate on directories When annotate is applied on a directory, annotate will output details of the last modifying patch for each file in the directory.
However, annotate fails in some files, and gives "unknown" as the patch. I have written a bug fix, and ported it to darcs development repository. The patch got reviewed and accepted.
UI changes I have simplifed the UI by removing single instance disable. Now, the ui stands as thus:
- Enable or disable patch index using optimize --patch-index or optimize --no-patch-index
- Patch index will be used and updated if the patch index is enabled
- Patch index will be enabled automatically at repository creation. If you wish to disable patch index pass --no-patch-index.
- When patch index darcs is run on an existing repository, patch index will be created automatically. You can preemptively disable patch index using optimize --no-patch-index
I have made a post in darcs users, suggesting this UI.
Timing tests
The time taken by darcs on a command varies with the content in the disk cache. In practice, this means that the first few commands are slower in a typical session. I have written tests which measure the time taken for patch index changes/annotate to run with a cleared disk cache. These are the results for
changes and
annotate. While the data for non pi changes and annotate is still incomplete, the results suggests that patch index universally improves the speed of annotate and changes, irrespective of disk cache. However, there is a concern about the time it takes to create patch index with disk cache cleaned(around 1min instead of 6sec).
Patch index data structures tests
Patch index stores information about the repository on the disk. I have begun writing shell tests on patch index internal structures.
Next week, I plan to- Write more tests on patch index data structures. There are two kinds of tests that can be written:
- Testing if the patch index structures uphold some properties. These tests can be run on any arbitrary repository.
- Create a small(but complicated) repository, and check if the information stored is valid
- More documentation on patch index structures
- Documentation on the timing tests
This week, I have worked on allowing a variable default for patch index, and make the failing tests of darcs development repository pass.
Variable default Previously, patch index is created/updated/used whenever the opportunity arises. You could disable this behavior and not create/update/use patch index, but only for that instance of the command (by passing --no-patch-index). Now, you can set the default to not use patch index.
There are three possible states a repository can be in:
- patch index usage is default: A repository will, by default be created in this state. You can also manually set the repository to this state by running optimize --patch-index. Any subsequent command in this repository will use/update patch index whenever appropriate. You can not use/update patch index for a single instance by passing --no-patch-index.
- patch index is disabled: You can create a repository in this state by passing --no-patch-index at creation. Also, if you do a lazy get, it will be created in this state. You can manually set to this state by running optimize --no-patch-index. Any subsequent command will not use/update patch index.
- undefined state: If patch index darcs is run on a repository that is created by a non patch index darcs, patch index will be created, and the default state is set to 1. If you wish to set to state 2, run optimize --no-patch-index. If you wish to run patch index darcs for that instance and not set a default pass --no-patch-index.
Failing Tests
There were 5 tests that failed in patch index darcs:
- repair-corrupt.sh
- repair-corrupt-add.sh
- lazy-optimize-reorder.sh
- issue1645-ignore-symlinks.sh
- issue1645-ignore-symlinks-case-fold.sh
The tests failed because of two reasons:
- These scripts pass --patch which could match to --patches or --patch-index. Script writers, take note! You may have to update your scripts, if you have a prefix of --patch-index in them.
- The scripts have an internal setup logic that fails due to patch index changes: For example repair-corrupt, repair-corrupt-add fail because a corrupt record fails due to the extra code in patch index. Similarly lazy-optimize-reorder fails because some critical patches get downloaded in a lazy get due to the original repo having patch index. The solution is to pass --no-patch-index at the creation of the repo.
Next week I will:
- Redo timing tests: In my last meeting with Benedikt, he pointed out that my timings assume "hot" cache(ie, the patch files are in the disk cache of the operating system). By turning the cache "cold"(ie, resetting the disk cache of the operating system), the time taken for running the commands expand dramatically. For example, on annotate on README of darcs development repo:
- hot pi: 0.3 sec
- hot nopi: 3 sec
- cold pi: 4 sec
- cold nopi: 20 sec
More distressingly, on cold cache, it is takes 1 minute to build patch index. Hence it is ideal to build patch index immediately with get where it takes around 6 sec. - Tests on patch index integrity: There are yet not tests on the patch index data structures.
- Integrate timing tests with darcs-benchmark
- UI discussion in darcs-users mailing list.
I am not able to push to darcsden public repository. Find my patch index repository mirror
here.
This week, I have worked on testing annotate on directories, and refactoring code according to the suggestions of Benedikt and Ganesh.
Annotate on directories If a directory name is given as input, annotate will output the last modifying patch for each file in the directory. However, the annotate in screened often fails and gives "unknown" as the corresponding patch for a file. A simple test case:
I have corrected this bug, and built patch index annotate on top of this. I have tested patch index annotate with the corrected version, and I found that they give the same output for all folders in darcs development repository.
However, when I tested corrected annotate with screened annotate, I found that they gave different output two files in /tests of darcs development repository.(They were no differences in the other folders)
The files are ./tests/issue1446.sh, and ./tests/issue1248.sh. However, even though they give different patches, they both give the correct answer as per the documentation.
This is because the filenames correspond to different files in the repository history. The corrected version gives the last patch modifying the most recent file, but the screened version gives the last patch modifying a previous file. Eric suggested that I patch the corrected version of annotate, and see if it gets accepted into the development branch. I have done so.
Code Refactors Benedikt suggested some refactors, most of which are to help the code get closer to existing coding conventions. Ganesh suggested (a) refactor to help with the safety of the coerce used in patch index code.
Next week, I will work on:
- More flexibility in the default option: Patch index should be ideally invisible to the user. Currently, it is invisible only if the user uses patch index. If he wishes to not use patch index, he has to pass --no-patch-index to nearly every command. After a discussion with Eric, and Benedikt, the solution I have in mind is this:
- Change the patch index automatic update, so that it works only if the patch index is created. (and does not create it automatically) This means that by default, patch index will be used/updated only if it exists.
- Give an option to delete patch index, so that the user can go back to a state where the default is to not use/update patch index.
- Disabling patch index on lazy get: patch index gets created by default at get. However, it requires all patches to be present in the repository. --lazy is used, so that patches will not be downloaded until needed. Thus by using patch index, the purpose of lazy is defeated, as it will get all the patches.
- Integrate the timing tests into darcs-benchmark.
This week, I have made user interface changes, some code refactors, solved bugs, and expanded tests.
User Interface Changes A darcs command may create(init, get), update(record, amend-record, push, pull, apply, unrecord, tag) or use patch index(changes, annotate). darcs will automatically create, update, or use patch index whenever applicable. To override this, use flag --no-patch-index.
Code Refactors I had duplicated some darcs annotate functions, and made minor modifications so that they do not use witnesses. Ganesh and I had a discussion on how to modify my code, so that I can use the original functions instead. I have implemented the suggestions, and removed the duplicate code.
I have merged patch index annotate code into the original annotate. This removes code duplication between the two annotates. (changes was merged previously)
I have made other minor refactors, like removing compiler warnings, and unnecessary exports.
Bug-fixes I have solved two bugs this week. The first is changes on directorates. I can now confirm that changes on directories works for all directories in darcs screened repository.
The second bug-fix is for the function that checks if patch index is up to date. It used to crash if patch index was not yet created. This used to indirectly crash the function that loads patch index.
Tests I have written shell scripts that measure the time taken by pi annotate, changes and compare it with the existing annotate and changes. Find the result spreadsheets here:
Changes Files,
Changes Directories,
Annotate Files.
For now, you can expect a speed up of 6.7x for changes on files, 3.5x for changes on directories, 8.1x for annotate on files on average.
I have written a test for patch index load when it does not exist, and updated the test on creating and updating patch index.
Next week, I will get suggestions on the necessary refactors for integrating the code with screened(mid-term goal), and making sure annotate on directories works properly.
This week I worked on support for directories in patch index changes, testing the correlation between patch index and existing commands, and some minor refactoring.
Implementation of directory support in changes Patch index changes uses patch index to filter out the unrelated patches of the repository. A patch that does not modify any selected file gets filtered out. A file will be selected if it had a path that is a subpath of the given directory at any point in the repository history.
Tests on correlation:I have tested for correlation on changes with files using the below script. I ran the script on darcs screened repository, and found that it succeeded on all files. A similar script for annotate has also succeeded.
Changes on directories fails to give the same output, with a simple example given below: In the above example, the patch index version fails to give the middle two patches. This is so because the file a was never under dir2 at any point in repository history. An implementation which checks if the file was in the given directory's history will solve this issue.
However, I found that the patch index version was as slow(and sometimes slower) as the current version for src of screened. I suggest that we either:
- Use patch index only on small directories, and fix the above bug
- Change implementation of patch index, so that it directly stores the relevant patches of a directory
Testing annotate on directories proved to be difficult. This is because patch-index annotate can give all the output of current annotate and still fail. Annotate on directories could fail for a particular file or sub directory, giving an unknown as a corresponding patch. Somehow, patch index annotate may succeed for a file even when current annotate fails. The current output format makes it difficult to filter out the "wrong" cases.I have also tested the correlation when various options are passed. I yet have to find a case where a patch-index command gives a different output when compared with the current command (when applied on a file).
For the next week, I plan to get the code up to mid term inspection. My mentors gave me the following criteria:
- that it does what is intended
- that it supports all of the options
- that the code itself is of push-to-mainline quality
- and there are adequate tests
I am to give priority to the first goal of the project (automatic update of patch index). For now, this means
- implementing --patch-index and --no-patch-index for all relevant commands
- fixing a bug where patch index was assumed to exist when loading patch index
- some refactors
- more tests
I am now using the rebased repo created by Eric. Find my fork
here.
My project is to optimize darcs changes and darcs annotate by using patch-index. Patch-index can quickly identify the patches that modified a given set of files.
Previously, I have created prototype versions of these commands using patch index.
These two weeks, I worked on adding support for the rest of the features.
The patch index version of changes can be called by darcs changes --patch-index [OPTIONS] [FILES]
For annotate, call darcs show patch-index-annotate [OPTIONS] [FILE or DIR].
The caveats:
- patch index versions sometimes do not give the same output as the existing versions:
- patch index changes on directories
- patch index annotate when applied on a directory, containing a file-path, which pointed to a different file in history.
- The speed of patch index commands directly depends on the number of files in the command. Annotate on directories with a large number of files is slower than annotate on a single file( but still faster than non patch index annotate).
You can find my public repo of patch index darcs
here.
Meanwhile Eric is rebasing my repo with darcs HEAD. You can find it
here.
Next week, I will thoroughly test the correlation between the patch-index commands and the existing ones, and fix the issue on applying a directory on changes.
EDIT: There is a newer repo of Eric's rebase: http://darcsden.com/kowey/darcs-patch-index-rebase-2012-06-17
This week, I focused mostly on improving the performance of patch-index. I am using the darcs screened repository as a benchmark to test the performance improvements.
The results are:
1) patch-index-changes on GNUmakefile took 0.43sec as compared to last week's 1sec
2) patch-index-annotate on GNUmakefile took
0.86 sec as compared to last week's 2sec
These improvements are due to two re implementations. First, I have re implemented the code that checks for correlation between patch index cache and the patches of the repository. This code is run every time patch index is loaded, and used to take 0.3 sec to complete. It is now reduced to around 0.03 sec.
Second(and more significant), I have re implemented the code that identifies the patches that correspond to a particular file. The new version is twice as fast as the old version.
I have also added annotate on directories. Try it by darcs show patch-index-annotate DIR/.
I am keeping the patch index repository up to date with screened, for a painless integration in the later stages.
The repository
here.
My project is to implement darcs changes and darcs annotate using patch index.
I have created a darcsden repository, find it
here.
Until now, The user visible changes are:
1) Automated update/creation of patch index to repository change/creation
Patch Index maintains some data structures independent of the darcs repository.
With automatic update, the index invisibly updates/creates itself.
2) A prototype annotate which uses patch index.
You can call the patch index annotate using darcs show patch-index-annotate FILE.
3) A prototype changes was implemented before.
The result for the annotate prototype is promising, giving a speed up of around 5 times(1sec instead of 5.5sec for GNUmakefile).
As of now, patch-index-annotate does not support most of the flags. Some of the unimplemented features, include annotate on a directory, non range matching. Implementing these will be my goal for the next week.
This year, Darcs had a sprint in two phases. It started with a one-day pre-sprint in Cordoba, Argentina (9 March), and then moved over to Southampton, England for a 3 day hackfest at the end of the month (30 March to 1 April).
No Darcs hackers being shipped between the two sprints though, but we did have one visitor from afar. Potential GSoC student Bhimanavajjula Sri Rama Krishna (BSRK) Aditya flew the 9 hours between India and England to join us for the sprint. It was great to meet him in person!
Presprint
We think that Darcs could make a great project for people get started with some practical Haskell hacking. It's a bit of a fixer-upper, but that also means there's a lot of difference to make!
Darcs veteran (and weekly news editor) Guillaume Hoffman was joined by two students, Miguel Pagano and Mathías Etcheverry, who within a day and with no prior knowledge of the Darcs code base were able to make the following contributions:
- bringing the darcs backup filename conventions in line with CVS conventions, eg. ./foo.txt.~1~ rather than the unwieldy ./foo-darcs-backup0 [Miguel]
- making the darcs-test harness respect the -fcurl cabal flag [Miguel]
- investigating a wishlist item to print filenames under large hunks in darcs record, alas, not as “ProbablyEasy” as we'd expected [Mathías]
In between getting Miguel and Mathías, Guillaume also got a chance to make some improvements himself, namely:
- adding the --unified flag to record, revert, amend-record
Thanks to Miguel and Mathías for joining us at the sprint. Hopefully we'll be able to repeat the cycle of Darcs hacking with them. And since little one-day mini sprints like the one Guillaume started are so easy to organise, there's a chance we'll be seeing more of these in the future.
Summer of Code
If he participates in this year's summer of code, Aditya will be helping us to integrate the long-promised
patch index optimisation into Darcs. The patch index was originally developed by Benedikt Schmidt. It caches a mapping from filenames to the patches that affect those files, which saves a lot of work for commands like
darcs changes or darcs annotate, commands that would otherwise have to trawl through the entire darcs history
Over the sprint, Aditya rebased the patch index code from Benedikt onto the current Darcs mainline. He studied the code a bit to understand what exactly was behind the index, and started working on implementing the integration with commands like darcs changes. He also got to explore a bit of Darcs internals, notably how Darcs makes use of matchers like 'date "before tea time"' to filter through patches.
One very concrete result of the sprint, we now have prototype of a patch-index-enabled
darcs changes.If you can't wait to try it out, you could try applying the latest version of
his patch.
Filepaths: bytes or code points?
The main thing Ganesh worked on was fixing a problem with character set handling that has been outstanding for several months. The underlying problem was caused by recent versions of GHC changing the way it handles filenames on Linux; previously it treated them as a stream of raw bytes, but now it translates them into strings using an encoding. The eventual workaround was very short - explicitly set a global at the beginning of darcs, telling the GHC library to use no encoding at all - but it took a lot of investigation to get to that point, and the end result isn't very satisfactory for darcs as a library.
Darcs 2.8 Release Candidate 1
Florent and Ganesh also worked on getting a 2.8 release candidate ready. We'd love any feedback you could give us on it, so if you're up for a little beta testing:
cabal update
cabal install darcs-beta
The character set handling problem with GHC 7.2/7.4 was the main blocker for a release, so hopefully we can get the real release out
pretty soon now.
Can you duplicate a rotcilfnoc (inverse conflictor)?
We are painfully aware that our current version of patch theory is broken with respect to conflicts. Owen Stephens (from Summer of Code 2011!), who generously hosted the sprint (thanks, Owen!) spent a good chunk of Friday staring at one example of the brokenness, a
failing QuickCheck test which he minimised to a simple 3-way conflict: create a directory and a file, (A) remove the directory, rename the file, (B) remove the directory, (C) move the file inside the directory under a different name.
Ouch.
After much discussion with
Camp hacker Ian Lynagh, Owen discovered that this was just a fundamental bug in the conflictor-based approach. We've already been back to the drawing board for a while, but now we have yet another test case for what the next patch theory should deal with.
Next Patch Theory?
We spent a bit of the weekend working on and discussing the new patch theory. Ian worked some more on Camp (more proofs!). Ganesh explained a bit more what he had in mind with the graphictors ideas he was exploring (each conflicting patch would be in a minimal context with respect to the conflict). And Owen talked us through some thinking he found digging through the archives of the old darcs-conflicts list. We know we need to a successor to the current version of patch theory. But where will we end are we going to end up?
Clean clean clean that code
The new patch theory won't be for a while. In the meantime, there is a ton of work we can do to prepare the ground for it. One thing we can do to help is to improve the Darcs code base to the point where shifting to a new patch theory, or a new repository format, or a new set of primitive patches is relatively smooth and easy. Darcs needs a cleanup effort.
Owen, Ganesh and Eric made several pushes towards making the Darcs code more approachable:
- Owen cleaned up a darcs module Darcs.Repository.HashedRepo
- Owen and Ganesh made the darcs code base warnings free (at last! again)
- Eric made use of Cabal 1.8's shared library feature so that Darcs only has to be built once rather than 3 times
- Eric (and a little bit of sed) replaced the confusing type witness C preprocessor macros with some more straightforward Haskell
We have a very long way to go. But we are
thinking harder about the concrete steps we can take to making the Darcs code more respectable.
More helpful interactive mode
Florent worked on adding some more intelligence to the Darcs patch selection code. Hopefully this work will lead to more feedback and some nice new features like an interactive darcs diff. Cherry picking is one of the more unique aspects of Darcs, and one of the reasons we're so interested in making patch theory right one day. The patch theory is what allows us to cherry picking in almost all of our commands.
But while interactive mode can be pretty helpful, it can also provoke for the kind of situation where good just makes you hungrier for better. For example, if you try to pull some patches but interactively decide that you want to skip some patches, Darcs will also skip over the patches that depend on it. But figuring out why exactly patches get skipped can still be a bit mysterious. What if instead of telling you it skipped some patches, Darcs could give name the dependencies you'd need to pull in too? Hopefully, Florent's investigations will pay off!
Rebase
Owen and Eric spent some time getting to know the new darcs rebase feature that Ganesh has been working on. It's nice! Darcs rebase is for those situations where Darcs patch theory falls over (and fall over it does). It allows us to rescue long-term branches previously lost to intractable conflicts, or to do “deep amend-record” operations that break through dependency barriers.
And this being Darcs, it's done with the interactive cherry-picking interface which should be familiar to users. There's starting to be talk of getting this code in HEAD darcs so that people can try it out and we can start working towards refining the user interface.
Darcs Bridge
Darcs bridge isn't ready for prime time, we're afraid. It's good for one-shot conversions, but if you're hoping to maintain a long-term bridge and you have to deal with Git branches, we'd advise waiting. But we're getting closer. Owen and Ganesh spent some time hashing out the design for the darcs bridge and thinking more about how the respective Darcs and Git models of the universe mesh together.
Where next?
Finally, among our many discussions was a more general question of strategy. Darcs is a very long term project and it could take many years for us to get the version control system that we want. Over the past few years, we'd placed a great emphasis on performance, addressing some day to day issues to bring Darcs to a more acceptable place; and now the efforts are starting to pay off. We now have faster local repository operations, repository fetching (mainly by deprecating the old fashioned format and getting people to switch to hashed repositories), and a much more usable darcs annotate command (in the upcoming 2.8 release). So Darcs is faster now— it's certainly no Git and the conflict merging issue is still there, but it's in a much better place than it was 4 years ago. Now what?
Now we start digging in for the long haul. We have essentially 3
development priorities for the future of Darcs:
- Cleanup: There is a massive amount of work to be done here, ranging from entry level tweaks like shifting to a uniform coding style and getting more disciplined about haddocks; to deeper software engineering issues, like developing a cleaner separation between repository-management and core patch theory code. The code needs a lot of loving, and if you're ready to roll up your sleeves, we could use the help.
- Hosting: Darcs isn't enough. We need to think about online hosting and GUIs. One of our goals is to have a Darcs library that makes it easier to write things like Patch-Tag, or Darcsden; or whatever interesting ideas the community may come up with. If we have to, we may even prototype some code ourselves to push the library forward.
- Theory: The one thing that we absolutely have to get right for the next patch theory is our story on conflicts. As you can see, we are thinking about quite a few different ideas. It's too early to tell which of these we'll end up running with. More news when we have some more solid ideas.
Thanks!
Darcs is a long term project and with all the ups and downs we've been through over the years, we are grateful for the support the community has shown over the years. Thanks to Guillaume and Owen for their sprint organisation efforts, to our donors for making it possible for students like Aditya to get to sprints, and the Software Freedom Conservancy for helping us with the administrative side of running an open source project.
If you'd like to support the Darcs team in our efforts to make an easy to use, flexible, formally backed version control system into a reality one day, we would be thrilled if you could submit patches, bug reports, comments on the IRC channel or
darcs reddit. If you just want to send a little cash our way to push sprints along, we most certainly appreciate your
donations.
Until next time!
 |
| No sprint is complete without an Awkward Group Photo |
I've recently been checking-in with my GSoC darcs-bridge work. I've re-worked
the horrible state-file format I originally used, and have been making various
other tweaks to the code.
Unfortunately, I've come across a few bugs (thanks to Brent Yorgey and Jesper
Reenburg for reporting some initial problems they've hit), and have been trying
to iron them out. With that in mind, I've added a bug tracker "Topic" for
darcs-bridge on the darcs.net bug tracker: Bug
tracker.
Hopefully, I'll be able to sort out the remaining bugs, and then move onto a
tidy-up and refactor of the existing code.
I’ve just released the initial version of DPM
on Hackage!
The Darcs Patch Manager (DPM for short) is a tool that simplifies working
with the revision control system darcs. It is most
effective when used in an environment where developers do not push their
patches directly to the main repository but where patches undergo a
reviewing process before they are actually applied.
Here is a short story that illustrates how would use the DPM
in such sitations.
Suppose that Dave Developer implements a very cool feature. After
polishing his patch, Dave uses darcs send to send the patch:
$ darcs send host:MAIN_REPO
Tue Mar 16 16:55:09 CET 2010 Dave Developer <dave@example.com>
* very cool feature
Shall I send this patch? (1/1) [ynWsfvplxdaqjk], or ? for help: y
Successfully sent patch bundle to: patches@example.com
After the patch has been sent to the address patches@example.com, DPM
comes into play. For this example, we assume that mail devivery for
patches@example.com is handled by some mailfilter program such as maildrop
(http://www.courier-mta.org/maildrop/) or procmail
(http://www.procmail.org/). The task of the mailfilter program is the add
all patches sent to patches@example.com to the DPM database. This is
achieved with the DPM command add:
$ dpm add –help
add: Put the given patch bundles under DPM’s control (use ‘-’ to read from stdin).
Usage: add FILE…
Command options:
Global options:
-r DIR –repo-dir=DIR directory of the darcs repository
-s DIR –storage-dir=DIR directory for storing DPM data
-v –verbose be verbose
–debug output debug messages
–batch run in batch mode
–no-colors do not use colors when printing text
–user=USER current user
–from=EMAIL_ADDRESS from address for emails
–review-address=EMAIL_ADDRESS email address for sending reviews
-h, -? –help display this help message
Now suppose that Dave’s patch is in the DPM database. A reviewer, call him
Richard Reviewer, uses the DPM command list to see what patches are
available in this database:
$ dpm list –help
list: List the patches matching the given query.
Query ::= Query ‘ + ‘ Query — logical OR
| Query ‘ ‘ Query — logical AND
| ‘^’ Query — logical NOT
| ‘{‘ Query ‘}’ — grouping
| ‘:’ Special
| String
Special is one of "undecided", "rejected", "obsolete", "applied",
"reviewed", "open", or "closed", and String is an arbitrary sequence
of non-whitespace characters not starting with ‘^’, ‘{‘, ‘}’, ‘+’, or ‘:’.
If no query is given, DPM lists all open patch groups.
Usage: list QUERY …
Command options:
Global options:
-r DIR –repo-dir=DIR directory of the darcs repository
-s DIR –storage-dir=DIR directory for storing DPM data
-v –verbose be verbose
–debug output debug messages
–batch run in batch mode
–no-colors do not use colors when printing text
–user=USER current user
–from=EMAIL_ADDRESS from address for emails
–review-address=EMAIL_ADDRESS email address for sending reviews
-h, -? –help display this help message
In our example, the output of the list command might look as follows:
$ dpm -r MAIN_REPO -s DPM_DB list
very cool feature [State: OPEN]
7861 Tue Mar 16 17:20:45 2010 Dave Devloper <dave@example.com>
State: UNDECIDED, Reviewed: no
added
some other patch [State: OPEN]
7631 Tue Mar 16 13:15:20 2010 Eric E. <eric@example.com>
State: REJECTED, Reviewed: yes
added
…
(The -r option specifies a directory containing the DPM
database. Initially, you simply create an empty directory. The -s option
specifies the path to the darcs repository in question.)
DPM groups all patches with the same name inside a patch group. Patch
groups allow keeping track of multiple revisions of the same patch. In the
example, the patch group of name very cool feature has only a single
member, which is the patch Dave just created. The patch is identified by a
unique suffix of its hash (7861 in the example). The output of the list
command further tells us that no reviewer decided yet what to do with the
patch (its in state UNDECIDED).
At this point, Richard Reviewer reviews Dave’s patch. During the review,
he detects a minor bug so he rejects the patch:
$ dpm -r MAIN_REPO -s DPM_DB review 7861
Reviewing patch 7861
Starting editor on DPM_DB/reviews/2010-03-16_7861_swehr_24166.dpatch
<inspect patch in editor>
Mark patch 7861 as reviewed? [Y/n] y
Patch 7861 is in state UNDECIDED, reject this patch? [y/N] y
Enter a comment: one minor bug
Marked patch 7861 as reviewed
Moved patch 7861 to REJECTED state
Send review to Dave Developer <dave@example.com>? [Y/n] y
Mail sent successfully.
Now Dave Developer receives an email stating that has patch has been
rejected. The email also contains the full review so that Dave sees why
the patch has been rejected. Thus, Dave starts fixing the bug, does an
amend-record of the patch, and finally sends the patch again.
(Alternatively, he could also create a new patch with exactly the same name as the original patch.)
$ darcs send MAIN_REPO
Tue Mar 16 16:55:09 CET 2010 Dave Developer <dave@example.com>
* very cool feature
Shall I send this patch? (1/1) [ynWsfvplxdaqjk], or ? for help: y
Successfully sent patch bundle to: patches@example.com
Once the email is received, the improved patch is added to the DPM
database. The output of the list command now looks like this:
$ dpm -r MAIN_REPO -s DPM_DB list
very cool feature [State: OPEN]
2481 Tue Mar 16 17:50:23 2010 Dave Devloper <dave@example.com>
State: UNDECIDED, Reviewed: no
added
7861 Tue Mar 16 17:20:45 2010 Dave Devloper <dave@example.com>
State: REJECTED, Reviewed: yes
marked as rejected: one minor bug
some other patch [State: OPEN]
7631 Tue Mar 16 13:15:20 2010 Eric E. <eric@example.com>
State: REJECTED, Reviewed: yes
added
…
The patch 2481 is the improved revision of the original patch 7861. It is
in the same group as the original patch because both patches have the same
name. Richard Reviewer reviews the improved patch and has no complains
anymore:
$ dpm -r MAIN_REPO -s DPM_DB review 2481
Reviewing patch 2481
Starting editor on DPM_DB/reviews/2010-03-16_2481_swehr_876102.dpatch
<inspect patch in editor>
Mark patch 2481 as reviewed? [Y/n] y
Patch 2481 is in state UNDECIDED, reject this patch? [y/N] n
Enter a comment: ok
Marked patch 2481 as reviewed
Send review to Dave Developer <dave@example.com>? [y/N] n
At this point, Richard Reviewer applies the patch with the very cool
feature:
$ dpm apply 2481
About to apply patch 2481
Entering DPM’s dumb (aka interactive) apply command.
Future will hopefully bring more intelligence.
Instructions:
=============
– Press ‘n’ until you reach
Tue Mar 16 17:50:23 2010 Dave Devloper <dave@example.com>
* very cool feature
(Hash: 20100316162041-c71f4-871aedab8f4dd3bd042b9188f1496011c7dd2481)
– Press ‘y’ once
– Press ‘d’
Tue Mar 16 17:50:23 2010 Dave Devloper <dave@example.com>
* very cool feature
Shall I apply this patch? (1/1) [ynWsfvplxdaqjk], or ? for help: y
Finished applying…
Patch 2481 applied successfully
Send notification to author Dave Developer <dave@example.com> of patch 2481? [Y/n] y
Mail sent successfully.
Applying a patch closes the corresponding patch group. Per default, the list command
doesn’t display closed patch groups, but we can force it to do so with the :closed
query:
$ dpm list :closed
very cool feature [State: CLOSED]
2481 Tue Mar 16 17:50:23 2010 Dave Devloper <dave@example.com>
State: APPLIED, Reviewed: yes
marked as applied: -
7861 Tue Mar 16 17:20:45 2010 Dave Devloper <dave@example.com>
State: REJECTED, Reviewed: yes
marked as rejected: one minor bug
…
Author: Stefan Wehr
![[Atom Feed]](images/atom.png){kind=small}
