Planet Darcs

Adolfo Builes: GSoC Week 9

2010-07-26T03:14:24+00:00

Last week I mentioned I almost had ready the implementation to deal with bad entries in the cache, finally during the week I sent a first version of the patch, I got some review and now I'm working to improve it.

One of the things I will change is the way we determinate which error we had when trying to use one of the entries in the cache, in my patch I was using the error string ( but it was because at the moment we don't have an ADT for some errors, for example the ones which are thrown from libcurl), so I will implement an new data type which will helps us to determinate in a safer way, which was the error that we got.

Other thing I will rewrite is the way to determinate if a ssh is bad or not, again we have the problem of the error type with ssh ( which we couldn't infer), so what I was doing was to do a request to the server and check if it was reachable or not, but then I realized that It wasn't correct, having a ssh server listening in port 22, doesn't necessarily mean that it does in port 80 too.

Also I got a first draft of the high level documentation doc, which aim to explain how the cache system works, I did a call in the darcs-users mailing list for feedback, which wasn't very successful, I would really appreciate if you can give a look at it and give me some feedback.

During the coming weeks I will finish this patch, and work on documentation and testing, more information of my progress can be found in the wiki.

Darcs News: darcs weekly news #68

2010-07-24T22:29:20+00:00

News and discussions

``darcs stash'' was discussed, with different possible implementations and UI proposed, and some example workflows evoked:
- http://lists.osuosl.org/pipermail/darcs-users/2010-July/024499.html
Reinier listed the release blockers for darcs 2.5:
- http://lists.osuosl.org/pipermail/darcs-users/2010-July/024603.html

Issues resolved in the last week (4)

issue1716 Reinier Lamers

allow mail header lines of all whitespace in test
http://bugs.darcs.net/issue1716

issue1883 Eric Kow

rename --patch-name option to --name.
http://bugs.darcs.net/issue1883

issue1887 Petr Rockai

add a missing newline to --list-options output.
http://bugs.darcs.net/issue1887

issue1893 Ganesh Sittampalam

move fields of conditional builds within scope of condition
http://bugs.darcs.net/issue1893

Patches applied in the last week (42)

See darcs wiki entry for details.

the Patch-Tag blog: Patch-Tag Lessons Learned, and Announcing Open Source

2010-07-21T17:10:42+00:00

I started patch-tag coming on two years ago, and a blog post has been brewing about what went right, what went wrong, what I didn’t expect (or what was harder than I thought) and where I see things going. Here goes.

What went right.

Haskell is a joy to work with, the haskell community continues to be friendly for the newbie and fun for the expert, better now than when I started. The explosion of creativity on hackage has kept making simple things simpler while pushing the limits of what you can experiment with, with a simple library include.

Darcs worked out too. It is still my favorite versioning system and I use patch-tag for offsite versioning almost all my new work, and keep trying to convert my friends and cubicle mates to it. Darcs users are nice and a lot of them see to be working on interesting things.

Happstack as a server — overall I’m sold, though there are a lot of rough edges, and I watch snap and yesod with interest. With happstack too many monads and the documentation is sparse… but it got the job done for me, so ok. I really want to do a yesod test drive too — Michael Snoyman has put a lot of himself into this framework, and from browsing the docs and watching the intro videos I have to say he is doing an awesome job.

View Code: HSP/HStringTemplate.

HSP is a bit of an odd one becuase I don’t use HSP in patch-tag, but I use it for my other happstack work, and I like it. It isn’t all win. If you are using HSP you have to recompile every time you change the view code, and the time lag can get noticeable, particularly if you have monolithic View files with a lot of code, and rely on cabal install. HSP also relies on the trhsx preprocessor which means that projects that require it won’t cabal install ootb without ~/.cabal/bin
(or similar) is in your PATH. And there are some mystifying error mesages. I think it also uses template haskell, so slightly slower compiles. But even so, I like it. I like having the compiler check that my tags match up, I like being able to copy html from the wild directly into my view code, and I like that designers find it easy to work with.

HStringtemplate is what I use in patch-tag and I like that too. The compiler won’t check my html tags match up, but it is fast, no recompile is required, and it is pretty smart. I will probably depend on hsp in the future, and might even switch patch-tag to hsp at some point, but overall I have to say hstringtemplate worked out pretty well.

Linode/amazon cloud. I use linode to host patch-tag, and the aws cloud to occasionally spin up a dev server and experiment. Odd combination perhaps but so far so good.

Help from unlikely places. At different times and in different ways Matt Elder, Dan Patterson, and Ram Durbha had a major impact in moving patch-tag forward, for no tangible material reward. I hope to get more detailed in another post, but for now I would just like to say thanks. These guys taught me, maybe the most important lesson I learned from patch-tag: that the universe is basicaly a friendly place.

What went wrong.

The hyphenated name. Mama mia. Why didn’t I just call it patchtag? The actual reason is that I showed logo mockups with and without hyphens to a couple of friends and everybody seemed to like the hyphen version better. If I could go back in time and do-over, I’d probably ditch the hyphen.

The macid data store. I will probably be switching patch-tag back to some more traditional data store, probably a vanilla db. No one big reason, just a lot of small ones.
Well there is one big one. Patch-Tag app state sometimes just loses data. So, a new user account will get created and then vanish. I think this happens when the out-of-memory killer kills patch-tag before an event has been written to the macid serialization log. Honestly though, I’m not sure. Patch-tag can soldier with a bit of human intervention because the most valuable artifacts (repositories and namespaces) are on the file system and not in macid. Still not good.
Then there are the minor issues with macid. Small user / dev community. The documentation just isn’t all that great, and not that many people know how to use it. It was started by the happstack originators (alex jacobsen and co I suppose), and found a home in the current happstack cabal — kudos to jeremy shaw and the other regulars on the happstack list. But it just somehow never seems to have “clicked” as a technology from my perspective. It seems to me, somewhat impressionistically I admit, to use an inordinate amount of memory, and slow down compile time — all those template haskell directives. And I don’t really understand it all that well even after using it for a long time. Finally, macid isn’t as easy to roll back on corruption as I had initially thought — you have to edit a binary file and… ugh. Macid *wants* to be a silver bullet kind of nosql solution that will scale to multiple servers transparently, use native data structures, and make life easy for the developer. It is almost that, but it isn’t there yet and other
nosql solutions have gotten a lot of traction in the meantime.
Finally… it is somewhat painful to admit, but patch-tag really doesn’t need transactions. I probably could have just used text files and read/show serialization for state. What the heck was I thinking? I can’t even remember anymore.

Cabal/dependency management. I was a bit conflicted on whether to put cabal under the things that went wrong category because I have no intention of giving up cabal and it is clearly a core technology for this kind of project. That said, dependency wrangling proved to be a continuing time and energy drain throughout the project. On a nearly daily basis it seems, things would install one day and wouldn’t the next because somebody had updated a dependency on hackage and this broke something upstream. It doesn’t seem to happen that often in small to medium size haskell codebases, but projects that have a lot of moving parts (eg use happstack) appear more vulnerable to this problem. I suspect yi, gitit, gtk2hs have a similar installer experience.
There is a second thing about cabal that actually has an easy fix, but I didn’t discover for a long time. (Thanks Matt Brown for pointing this out!) Cabal install, which was how I compiled for the first year or so, started taking a long time. Like, over a minute. The fix is to just compile with ghc –make, after using cabal install just the first time to pull in all dependencies, and periodically to get updates. This seems to usually go about 4 times as fast, and when trying to stay in flow mode 15 seconds versus over a minute can start making a big difference.

Gitit fork. Gitit on the whole, I am happy with. I don’t think that many patch-tag users have taken advantage of this feature, including myself, but I really like the portability of keeping documentation alongside repo. Cross polination with another major happstack consumer is another win. Where I have regrets is having forked the gitit code somewhere along the line to get some look and feel features I wanted because I was in a hurry. Now gitit mainline has progressed several versions and I have to undo the fork I did and clear up all these niggling details to sync up, or maintain the fork forever. Whoops.

Unix security model alongside app security model to get ssh working. This is a mess for portability. If I want to switch servers I need to recreate system accounts, ssh keys, it’s a real drag. Alex Suraci’s Darcsden has an ssh server built in, written in native haskell, and apparently this makes it a lot easier to maintain. Transitioning patch-tag to this library and ditching the unix security details is a high priority for me in terms of future maintainability.

(Eventual) Moneti$ation. I thought I would go after darcs first, add git/svn other systems once darcs was solid, which would be a big enough market to make a sustainable business. Turned out not to be so easy!
On several occasions I was about to do payments, but then I got distracted fixing something without which it wouldn’t have felt right to start charging. Looking back on it, I kind of wish I had already gone after paying customers by this point.
The good news for me on a personal level is that having done patch-tag has really helped me get the kind of work I want to have, and advocate for using the technology I like, namely functional programming.

*********************

End ramble down memory lane, and announcing open source.

Patch-Tag source is open source at

http://patch-tag.com/r/tphyahoo/patch-tag-public

The install is non trivial, so for anybody that wants to check it
out without jumping through a whole lot of hoops (highly recommended) there is an amazon ami. Find the ami code by getting your amazon key, setting up ec2 command line tools and executing

thartman@ubuntu:~>ec2-describe-images -a | grep -i patchtag
IMAGE ami-febe5597 072945664613/patchtag dev
ami 072945664613 available public i386 machine aki-5f15f636 ari-d5709dbc ebs

You can run this ami from the comand line with ec2-run-instances, or use the aws gui or any of the numerous third party guis. Once you’re in, cd to the patch-tag directory, darcs pull the latest code, cabal install, and you should be good to go.

The main reason I am open sourcing patch-tag is to stimulate the haskell web-devel ecosystem by putting a “real world ready” app out into the open.

If anybody playing with the opened patch-tag wants to help me out with the project, here are my highest immediate priorities.

* Help me choose an open source license for patch-tag. I am considering gnu, bsd3, and CPAL (same license as reddit). What does the community think?

* Get patch-tag easier to install and on hackage. Patch-tag is not on hackage, because my feeling is that a program on hackage should Just Work or it violates legitimate expectations, and patch-tag cabal install is far from just working. Anybody is free to throw patch-tag on hackage under their own user account but I would say better to wait because the install issues *are* fixable with a little love and care.
Key subtodos for this are
* Unfork gitit and peg patch-tag to mainline gitit
* Use same ssh lib as darcsden so I can ditch most or all of the linux sysadmin stuff
* Make patch-tag machine instance for platforms other than ec2 (eg virtualbox, vmware, make it runnable on windows, etc).
* Diagnose and fix a suspected memory leak that results in regular visits from the dread oom killer.
* Transition patch-tag to a better understood / supported persistence solution than macid. My current thinking is sqlite, or possibly just text files with read/show serialization.
* Work on macid, and help realize the macid dream of straightforward nosql style persistence that uses native data structures for an easy start and scales to zillions of users with no unpleasant surprises.

Finally, and most importantly: Use haskell web-devel. Write documentation, ask questions, get on irc, and make the world a friendly place for people that like haskell and want to write the next facebook/ita/viaweb killer.

Thanks for tuning in, folks, and happy tagging!

Adolfo Builes: GSoC Report Week 8: Dealing with bad entries in the cache

2010-07-17T17:48:10+00:00

During this week I have been working in the implementation to deal with the bad entries in the cache, to get a bit of background of how the cache works read my previous post.

When we have remote entries, we consider conflictive the ones that 1) Are not accessible ( give time-out each time we try to access) and 2) are accessible but the repository doesn't exist.

We consider a bad local entry, the one which points to a repository which doesn't exist.

At the beginning the idea was to automatically remove those entries from the cache, but as I mentioned in the previous post, sometimes there are externals factors which won't allow me to reach an entry, so the approach taken to solve this problem was just not using that entries during the rest of the command, and at the end notify to the user about it and allow her to delete the bad entries interactively, something like:

$ There seems to be a problem with the following repos: repo1, repo2, repo3
Would you like to delete them from the cache ? Y/N

I haven't implemented the part of notifying to the user about the bad entries, but I introduced the changes to stop using an entry if we have a problem with it, so for the remote repositories which give time out, they get added immediately to the list of bad caches and we don't try to fetch patches from them, if it's a remote entry which doesn't give timeout but throws an error it could be because a) the requested file wasn't there b) the repository doesn't exist, so in this case when I get an error of this type I verify what caused the error, if the reason is that the repository doesn't exist, it is added to the list of bad caches, if not, it is added to a list of "good entries", if we get again other error because the file doesn't exist, we won't need to check for any of the conditions mentioned before.

An approach similar to the remote repositories is taken for the local entries which fail.

After I introduce the changes, I did a lazy get of Tahoe-LAFS, then introduce a bogus entry for the timeout case and check the time difference calling the command darcs changes, I did it with my version and with the version in hackage.

For 2.4.4 (release) which is the version in hackage, it took a bit more than 13 minutes.

real 13m38.415s
user 0m4.148s
sys 0m1.060s

Then after I rebuild with my changes the result were really good, taking less than 1 minute to fetch the changes:

real 0m55.679s
user 0m1.092s
sys 0m0.240s

Before my changes Darcs will try to establish a connection with each of the entries in the cache for each patch ( waiting for timeout), now with the changes, if it founds a bad entry it doesn't try to use it in the rest of the command, which make faster the operations since we don't waste time trying to use bad resources.

Also I implement an environment variable "DARCS_CONNECTION_TIMEOUT" which set the waiting time for a request.
To implement such functionality if we are using libcurl, I setup the CURLOPT_TIMEOUT option, if we are not using libcurl but Network.HTTP, I wrapped the operation of simpleHTTP in the function timeout from System.Timeout. In Linux it works perfect both with libcurl or simpleHTTP, but in Windows I have a problem with the 'timeout' function from System.Timeout, it doesn't behave as expected and it seems like it gets ignored.

For the next week I will focus on finishing this, extending the haddock documentation of the cache and tests. Remember you can always check my advance in the wiki.

Darcs News: darcs weekly news #67

2010-07-11T17:13:06+00:00

News and discussions

Reinier announced the first beta of Darcs 2.5:
- http://lists.osuosl.org/pipermail/darcs-users/2010-July/024459.html
The next darcs sprint will happen in October in Orleans, France:
- http://lists.osuosl.org/pipermail/darcs-users/2010-July/024429.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-July/024430.html

Issues resolved in the last week (6)

issue1288 Ganesh Sittampalam

the main darcs code now compiles and runs with witnesses
http://bugs.darcs.net/issue1288

issue1726 David Markvica

Files with _darcs prefix are always boring
http://bugs.darcs.net/issue1726

issue1825 Petr Rockai

do not omit important prims in unrecordedChanges w/ files.
http://bugs.darcs.net/issue1825

issue1845 Petr Rockai

fix "darcs record f" for f a removed, version-tracked file.
http://bugs.darcs.net/issue1845

issue1865 Petr Rockai

make scanBundle produce a well-formed PatchSet.
http://bugs.darcs.net/issue1865

issue1871 Petr Rockai

announceFiles should not give empty list for ".".
http://bugs.darcs.net/issue1871

Patches applied in the last week (83)

See darcs wiki entry for details.

Darcs News: darcs weekly news #66

2010-06-28T12:33:41+00:00

News and discussions

Reinier announced the release schedule of Darcs 2.5 (soft freeze July 8th, release August 7th):
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024284.html
Reiner also issued a call for volunteers for fixing unassigned bugs that should be fixed for the next release:
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024407.html
Eric explained how to make the patch reviewing process more efficient:
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024370.html
And the Sumer of Code blog posts of the last two weeks:

Issues resolved in the last week (7)

issue1176 Adolfo Builes

1176: caches interfere with --remote-repo flag
http://bugs.darcs.net/issue1176

issue1277 Eric Kow

percolate repository format errors correctly.
http://bugs.darcs.net/issue1277

issue1389 Reinier Lamers

Change predist pref to point people to use 'cabal sdist'
http://bugs.darcs.net/issue1389

issue1713 Eric Kow

shorter interactive prompts.
http://bugs.darcs.net/issue1713

issue1857 Petr Rockai

Use and update pristine root pointer upon conversion.
http://bugs.darcs.net/issue1857

issue1877 Florent Becker

make pull --dry-run --xml terse
http://bugs.darcs.net/issue1877

issue1879 Eric Kow

notice unexpected commute failure on merge.
http://bugs.darcs.net/issue1879

Patches applied in the last week (53)

See darcs wiki entry for details.

Adolfo Builes: GSoC Report: Week 5

2010-06-26T20:41:53+00:00

In my last post I said I had complete the warm-up phase and that I will start to think about how to handle caches which are no longer available.

The caching mechanism relies in the files _darcs/prefs/sources and ~/.darcs/sources, basically the content of those file is used to generate the cache entries, each of the entries in that file indicate an alternative source to get files. If we want to specify global caches we put that in ~/.Darcs/sources but if we want an alternative repositories to pull from, we specify that in the repository sources file which is in _Darcs/prefs/sources, also each time we do a pull from an external repository it is added to the sources file
automatically.

The problem of expiring caches is given because sometimes it happens that repositories that were available, can become unavailable. For example if I had pulled from 3 different repositories and 2 of them stop being available, it could take up to 2 minutes to get each patch, because Darcs could try to fetch every patch it needs from those 2 not longer available repositories, it tries to establish a connection, an then waits for a time-out or a bad response code. After the problem I just mentioned, the idea is to design a mechanism which can help Darcs to establish which entries should be expired.

We can split the cache entries in two groups: locals and remotes. Dealing with local non-longer reachable repositories is not a big deal since if we don't find the local entries we can assume they don't exist and we can drop them from the cache and stop trying to fetch files from them. Remote repositories are more tricky, for example I can't eliminate an entry just because it gives a time-out when it tries to establish a connection with it, there are other external factors which could interfere with that particular entry in a given moment
(firewalls).

So as we seem this is not an easy task, handling remote repositories is out of our hands, we don't have control over the external sources, we don't have control over the network configuration and so on. So a first approach to this is to mark the entries which are not working and ignore them for the rest of the pulling since we don't want to try to establish a connection with an entry which we know is not available. If we try to establish a connection and fails we can mark it as a bad entry but also I think it could be awkward to wait for a 60 seconds time out, something we could implement is a default time for waiting for a connection to succeed (10-15 seconds maybe) if it doesn't happen between that time we can skip it, mark it as bad, and don't try for the rest of the patches that particular entry. Other approach suggested in the bug tracker was to try to establish a connection with all of the entries and use the one that responds first but then what if all the entries are bad entries?. I have to think more about it, I will discuss on irc and the mailing and then with a clearer idea I will start to code a patch to solve the issue.

Also I sent a first version of a failing test for the case of unreachable entries, but I need to amend, as there are some missing cases.

More of my progress can be found in the wiki.

Alexey Levan: GSoC 2010 Progress Report #1

2010-06-26T09:25:07+00:00

This summer I work on making Darcs over networks faster. My project consists of two parts:

Implement an optimization for getting repository over HTTP. This is done by creating a snapshot of current repository state.
Create a smart server for darcs, which is able to determine the patches needed by client and send them with response. This will be the most effective way to get/pull a repository, since it reduces the number of roundtrips to minimum.

The original complete description of the project can be seen on the wiki. Note, however, that for smart server more priority is given to the CGI frontend, rather than plain HTTP.

Changes so far

I glad to report that the most work on HTTP optimization is complete and patches are on their way to the Darcs repository. Some notes on implementation:

getting an optimized repository results in almost the same copy as for unoptimized one. While inventory files may be split in different ways, semantically resulting repositories remain identical.
there is still a couple of issues with special cases, like working with cache and handling interrupts; I will hopefully resolve them in nearest time.

Next week

At the moment I have started implementing the smart server. Besides the completing of the work on optimize --http issues, on the next week I will refactor the get/pull commands' code, which will result in cleaner API for the server.

Adolfo Builes: GSoC Report: Week 4

2010-06-20T20:02:39+00:00

I have completed my phase of warm up issues which was oriented to allow me to get familiar with the Darcs system, specifically the cache part. I have sent the patches and they have been applied.

During the week I worked in finishing issue 1176, continued with the documentation part, more specifically describing in a higher level how a patch is fetch with a given hash, making easier for someone with a non-technical background understand what is happening, fixing a test which was failing in Windows when it shouldn't and finding out why the IO operations over hashedRepos were put in a different module.

For fixing issue 1176 I did some modifications over some code I had already sent which was about keeping the caches sorted by locality, the initial idea was that anything which was local should be first in the list and then the remote sources, but we realized that between the remotes also exist a "wanted" hierarchy, basically we would prefer to access first http repos over ssh, so the new sorting keeps all the locals first, http in the middle and ssh repos at the end. One of the problems this solve is that weird behaviour of darcs trying to establish a ssh connection when pulling from a http or local repository.

While I was working in the test which was failing we redefine what gets saved in the _darcs/prefs/sources, the global caches were one of those things, it wasn't necessary to have them there because that's why a global cache configuration file exist (~/.darcs/sources), so basically we just drop anything related with global caches before saving the sources file of a repository.

For the next week I will start to work in the problem of how to deal with the unused cache, the idea is by the end of the week to have a work plan and write a test case.

You can check out more of my progress in the wiki.

the Patch-Tag blog: patch-tag gets more ram

2010-06-16T20:38:26+00:00

For free.

Thanks, linode!

Darcs News: darcs weekly news #65

2010-06-12T13:20:40+00:00

News and discussions

Lele Gaifax released a new version of the trac+darcs plugin:
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024281.html
- http://progetti.arstecnica.it/trac+darcs/
We are still looking for a release manager for the release of darcs 2.5. Eric summarized the discussions concerning recruitment and the responsibilities and challenges related to this job:
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024268.html
Roadmap: rebase won't be in 2.5, annotate will be improved:
- http://irclog.perlgeek.de/darcs/2010-06-08#i_2415755
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024227.html

Issues resolved in the last week (2)

issue1210 Adolfo Builes

global cache gets recorded in _darcs/prefs/sources
http://bugs.darcs.net/issue1210

issue1874 Eric Kow

recognise network tests on cabal test command line.
http://bugs.darcs.net/issue1874

Patches applied in the last week (21)

See darcs wiki entry for details.

Jason Dagit: Delimited Continuations and version control: an update

2010-06-10T20:44:01+00:00

Last time I presented the idea that version control and delimited continuations are related. I left off with a question how how to make Darcs fit this model. I think I understand now what I was missing.

I forgot to think about Darcs operations in terms of the intermediate operations that get performed. In Darcs, everything is based on commuting patches, even merging. Therefore, to see how Darcs fits into this model it's important to think about commuting patches in terms of delimited continuations.

Specifically, I now believe that commuting two patches introduces marks that can be shifted to later.

I have several ideas for the next steps of this. One is to start modeling toy versions of svn and darcs in Haskell via delimited continuations. After that, I would like to figure out the correspondence between the delimited continuations that I've identified and their data structure reification as zippers. I hope to have more about that later.

Judging by a paper written by Oleg, there should be a natural way to convert the delimited continuation representation into a zipper. Investigating this model might shed light on the Darcs patch model, or even lead to a more concise formalism.

Darcs News: darcs weekly news #64

2010-06-06T20:45:15+00:00

News and discussions

An alpha release of darcs 2.5 might happen soon:
- http://irclog.perlgeek.de/darcs/2010-06-06#i_2408520
Summer of Code: Alexey Levan sent a first version of patches for darcs optimize --http, and Adolfo Builes sent a patch fixing a bug concerning cache pool choice by darcs:
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024159.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-June/024132.html

Issues resolved in the last week (10)

issue1337 Petr Rockai

mark the test as passing. Likely addressed by noslurps.
http://bugs.darcs.net/issue1337

issue1503 Adolfo Builes

prefer local caches to remote ones
http://bugs.darcs.net/issue1503

issue1610 Petr Rockai

mark the test as passing. Likely fixed by NewSet.
http://bugs.darcs.net/issue1610

issue1817 Petr Rockai

--external-merge broken, by bumping h-s dependency to 0.5.1.
http://bugs.darcs.net/issue1817

issue1839 Florent Becker

k broken in interactive selection
http://bugs.darcs.net/issue1839

issue1843 Florent Becker

interactive 'v' prints double entries
http://bugs.darcs.net/issue1843

issue1848 Florent Becker

Patch.Choices.makeEverythingSooner is incorrect
http://bugs.darcs.net/issue1848

issue1860 Petr Rockai

(Un)applying move patches may lead to incomplete pristine.
http://bugs.darcs.net/issue1860

issue1861 Eric Kow

Fix typo in --no-boring help.
http://bugs.darcs.net/issue1861

issue1864 Florent Becker

w should only skip one patch
http://bugs.darcs.net/issue1864

Patches applied in the last week (18)

See darcs wiki entry for details.

Adolfo Builes: GSoC Week 2

2010-06-06T19:05:41+00:00

After last week's meeting with Eric ( my mentor) we got a better time line written down, and now I have clear goals for each week until the midterms evaluation, for this week my goals were:
* Complete issue1503
* Complete issue1210
* Description of cache usage for each module in Darcs.Repository

I'm happy to say that issue 1503 was closed, and I sent the patch for issue1210 but is waiting for revision, also I noticed that I'm more familiar with the darcs structure ( at least the part in which I'm working ) and I've started to know where I have to search for something each time I need a particular functionality.

When I first sent the patch for issue 1503, Eric and Petr did some comments about design and style which took me to rewrite the original patch, thanks to them I have learn to push myself more into thinking in a bigger view each time I plan to introduce changes somewhere, sometimes you just develop bad practices which you don't see unless someone else point it to you, so I feel more convinced that a good way to learn and become a better programmer is contributing to this kind of projects, where you interact with other people, where they are commenting on you code, making you think better about what you are doing, all this kind of stuff is something you won't learn from the university, but just getting involve in something in the real world.

Other thing that I learnt was that I had a wrong idea of the concept of tests, I thought the test case for a certain issue would be one were it used to fail before applying the changes, but I didn't thought about how should the test behave in case of failing, here thanks again to Eric for explaining it to me :).

For the next week I plan to work on issue1176, start to elaborate a test plan for issue 1599, write the test cases for issues 1503 and 1210 and continue my work in the document of the darcs cache.

You can always check out my advance and know more about my project in the darcs wiki where we are documenting everything.

Jason Dagit: Delimited Continuations and Version Control

2010-06-06T09:41:55+00:00

Delimited continuations give us a way to create markers that we can jump back to. We can construct the future of the computation, work with the computation so far, or abort the current continuation and go down a new path (create a new future).

These primitives have a natural correspondence with version control systems that snapshot the world like
SVN. Focusing on SVN for a moment:

The current continuation is the transformation of repository state that you're working on. It's the diff you're creating.
For commits, each revision created by commit is a marker, so we model this with reset.
Checking out an older revision, or reverting changes, corresponds to a shift. We discard the current continuation and move back to a marker created by a specific reset.
Updating consists of having the client copy learn the current state of the continuation on the server and applying it to the local copy.
Starting a branch corresponds to a reset.
Merging two branches is a bit trickier I suspect. I haven't worked out all the details sufficiently to convince myself I have it right, but here is how I think this case works. The merge first shifts to each of the markers and then combines those two continuations into one future. The part that seems weird to me about this, is that I haven't really seen examples of delimited continuations were the continuation of two different markers (prompts) were combined.

Now, if you accept the above it gives us some intuition to build on. Although my correspondence is terribly informal at the moment, if we took some time to make it formal by working out enough details to model it in, say, Haskell, then we'd have a nice formal backing for how SVN works. I think the above model applies equally well to git, but I'm not confident with git's model.

One insight the above gives, is that the way merging works is not described by the continuations in general. It's up to the exact combining function to determine the merge. We know that an automatic merge can fail in practice due to things like conflicts between the changes in the branches. So, in the SVN implementation considerable work has gone into implementing logic for creating the proper continuation.

Now, in general when a merge, or update, is performed human intervention may be required. This typically happens when the changes are in conflict. What this means for our model is that in general the creation of the continuation requires knowledge outside of our model. What does that correspond to? Well, it's essentially saying that calculating the correct continuation to resolve the merge is non-deterministic!

In other words, this continuation view of version control gives us a rigorous way to talk about our intuition. Of course we can easily tell that merging is going to require human intervention without needing to study delimited continuations, but this framework of reasoning now gives us a more mathematical way to say it.

The next question is: How does the delimited continuation model of vcs apply to Darcs?

So far I'm not sure. I suspect, without working through the details, that in Darcs, every time you record a patch multiple resets happen, instead of just one. The model really breaks down for Darcs because you can seemingly visit points in "history" that did not exist when the patches were created, but they are valid repository states.

For example, imagine a repository that only exists locally. You create a sequence of patches. Now, take a patch in the middle that can be commuted to the end of the patch sequence. Doing so has not created a new repository state; so far this is fine with the above model as no new markers need to be created. Suppose we remove the patch from the end of the patch sequence. This is exactly how darcs unpull works. The funny thing is, we've now created a state that never existed previously. So in the delimited continuation model, what marker did we just shift to?

I don't know the answer yet, but I think it's an interesting question. I suspect there are multiple "correct" answers, but that only some answers will yield elegant and robust models here.

Jason Dagit: Darcs 2 Real-World Push Performance Evaluation

2010-06-06T08:53:20+00:00

Thanks to Eric Kow, Duncan Coutts and Ian Lynagh we have some great timing data for using darcs2 and darcs1 to push patches over ssh.

Eric wrote a script to test three different scenarios of using darcs to push patches:

Scenario l1r1:
Scenario l1r2:
Scenario l2r2:

Next, Duncan and Ian provided us with access to 131 real-world repositories hosted at http://code.haskell.org. We ran the script to push patches to each repository, this gave us a ton of times. Then in Excel we crunched these numbers to see that not only is scenario l2r2 no worse than the other two, it’s actually faster on the time consuming cases!

The one caveat we found is that the minimum start-up time for the first two scenarios is 1 second and in the last scenario it’s 2 seconds. I’m confident we can shave off this 1 second difference in the future.

This is a histogram that shows you how the push times distribute, click on it for a large image. Along the bottom we have how many seconds the push took, and along the vertical axis we have the number of data points in that range. At a glance you can see that most repositories take just a few seconds to push. We can also see that darcs2 is slower on small pushes by about one second. Darcs2 in this chart corresponds to l2r2 and darcs1 corresponds to l1r1.

On a side note, we also tested converting all the repositories to darcs2 repository format and that worked great as well. Converting all the repositories at once takes less than 20 minutes on my laptop without a single error. There were a few warnings, but that’s to be expected as potentially exponential merges are fixed in the new darcs2 format, but darcs emits a warning when fixing them.

For anyone that wants to see the raw numbers click here. The link does work, but not all web browsers are showing the numbers. Opera and FF3 work on some platforms and not others.

Jason Dagit: Type-Correct Changes — A Safe Approach to Version Control Implementation

2010-06-06T08:14:44+00:00

On March 20th, 2009, I successfully defended my Master’s thesis in Computer Science.

Abstract:
Ensuring correctness of real-world software applications is a challenging task. Testing can be used to find many bugs, but is typically not sufficient for proving correctness or even eliminating entire classes of bugs. However, formal proof and verification techniques tend to be very heavy weight and are simply not available for day to day use in many common programming environments.

We demonstrate a form of light-weight proof assistant by using the type checking features of the programming language Haskell with existing extensions. We apply this work to the Open Source version control system Darcs. The properties checked by our approach are derived directly from the data model used by Darcs. This allows us to eliminate entire classes of bugs at compile time. We also examine how these techniques improve the quality of the Darcs codebase and the challenges that arise when applying these techniques in practice.

You can read the full thesis here. The slides from my presentation are located here.

The bottom line is that we used Generalized Algebraic Data Types (GADTs) to enforce proper patch manipulations. In the Darcs implementation, patches are stored in sequences and rearranging those sequences can only be done is very specific ways. Our use of GADTs allowed us to express those constraints using existentially quantified types, phantom types, and witness types. If you’ve ever wondered how to use GADTs in real-world software, this serves as a very illustrative example.

Jason Dagit: Understanding Darcs Commute

2010-06-06T08:14:44+00:00

People often want to understand how commute on patches works. Usually we start by saying:

Given two patches, A and B, if A and B commute then: AB <--> B’ A’, for some B’ and A’.

Naturally people ask, “But what is the relationship between A and A’ or B and B’?” This is a very important question and I’ll provide you with some insight.

Suppose we have a repository with 2 files, a and b. We could then make the following operations:

mv b c
mv a b
mv c a

You can think of each operation as a transformation on the ’state’ of your repository.
Suppose also, that we make an edit to a, and an edit to b.

Let’s name the above, using T for transformation:

T_bc = mv b c
T_ab = mv a b
T_ca = mv c a
T_a = edit a
T_b = edit b

You can imagine that if I gave the diff for T_a and the diff for T_b that you could apply those diffs in either order to your repository and get the same final ’state’. Meaning, a and b are the same whether you update a first or b first.

But, suppose instead that I performed T_bc, T_ab, and then T_ca. This has the effect of swapping a and b by name. Now suppose you applied the diffs T_a and T_b. What would you want the outcome to be?
It turns out, that it matters which operations were created first. If you created the diffs T_a and T_b *before* you did the operations of the swap, then you should expect that after the swap the diff for T_a actually modifies b, whereas T_b should modify a. On the other hand, if you created the diffs T_a and T_b *after* the swap, then you expect T_a to modify a and T_b to modify b.

We have an intuitive idea of ‘context’ now. As in, what is the context that T_a and T_b were created in? Knowing this will tell us how they transform the repository state.

Intuitively, it seems as though we need to remember the ‘context’ in which T_a and T_b were created. So let’s say that the operations performed up to the point where T_a is created is the context of T_a. In other words, the context for T_a is sequence of transformations that existed when T_a was created. Similarly, since T_a is a transformation, creating it results in a new context, which is the old context plus T_a. We could say that T_b has this context. Going a bit further, it seems like we should talk about how T_a has a pre-context and it also has a post-context.

For example, if we created T_a before doing the swap, then the pre-context might include two transformations, one that creates a and another one that creates b. The post-context would then include those two transformations and T_a itself. If we created T_a after doing the swap, the pre-context and post-contexts of T_a would include T_bc, T_ab and T_ca also.

Now a side note about commutative functions. Consider the function created by composing T_a and T_b, let’s write T_a . T_b. Recall, that with function composition parameters start on the right and pass through the sequence to the left. As discussed in the intro, T_a . T_b is equal to T_b . T_a. This is because T_a and T_b are independent of each other. Thus, we would say that the functions T_a and T_b are commutative functions. This means, that changing their order of application does not change the result.

We are saying that:

T_a . T_b = T_b . T_a

Because T_a and T_b are commutative it doesn’t matter which order we compose them. If we restrict our view to just the repository above with only the files a, b and no c, then on this restricted set of repository state how do these two compare?

T_b . T_a
T_a . T_b . (T_ca . T_ab . T_bc)

In plain English, the first one edits a and then b, the second one swaps a and b, edits b and finally edits a.
As far as the mathematics of it is concerned, the first one will edit a and b, while the second one will have T_a editing a different a than the first one and T_b editing a different b than the second one.
Going a bit further, let’s say that T_a and T_b were created without any of T_bc, T_ab or T_ca in their context. So we could have two scenarios.

We could, for example, start with T_b and T_a, swap their order and then do the swap of a and b afterwards. That would give us:

T_b . T_a

and

(T_ca . T_ab . T_bc) . T_a . T_b

Intuitively, it seems like T_a and T_bc are commutative functions, eg., T_bc . T_a = T_a . T_bc. So we could rewrite the second one as this:

T_ca . T_ab . T_a . T_bc . T_b

Now, suppose when we commute the function T_a with T_ab, that we replace T_a with T_a’. T_a’ is like T_a except that T_a’ makes the edits of T_a to b instead of a. After all, this results in T_a’ editing the correct file after the rename. Similarly, when we commute T_b with T_bc, T_b is replaced with T_b’ that edits c instead of b. When we commute T_b’ with T_ca we replace T_b’ with T_b” that edits a instead of c.

So, the above goes through these steps:

T_ca . T_a’ . T_ab . T_bc . T_b (commute T_a to the left)
T_a’ . T_ca . T_ab . T_bc . T_b (commute T_a’ to the left)
T_a’ . T_ca . T_ab . T_b’ . T_bc (commute T_b to the left)
T_a’ . T_ca . T_b’ . T_ab . T_bc (commute T_b’ to the left)
T_a’ . T_b” . T_ca . T_ab . T_bc

The last one will then have T_a’ and T_b” making edits the same file contents as T_a and T_b respectively, even though the names of the files were changed by the swap.

So, if you’ve followed me to this point, then you now have the intuition for what we mean when two patches A and B, commute to B’ and A’, as AB <--> B’ A’. You can think of a patch as being one of the above transformations along with the context of the transformation. You might also notice that commute of patches must be doing something to the context of the patches.

Patch commute has the potential to update the context and transformation the patches it swaps OR it could update the context and leave the state transformations equal to what they were in the input. Patch commute can also fail, but we’re ignoring that case for the moment.

Thinking back to how we arrived at the need for context, you might notice that for each context, that is each sequence of operations, we get one unique repository state. This is a very important property of context. Without it, context wouldn’t really be useful. Also, notice that the opposite is not true, repository state does not determine the context. Which makes sense, because there are lots of operations you can do that get the repository to a particular state, so given a state how do you know what was done?

The next important property we want for commuting patches is that once two patches have been commuted, you can commute them again to undo the commutation. In fact, it turns out the examples above are saying we want contexts to determine the same state if you commute the patches inside the context (again, context is a sequence of patches!).

For R to be an equivalence relation, we need three things:

x R x, is true for all x
if x R y then y R x
if x R y and y R z then x R z

Here, we replace x R y with “the sequencing, or order, of x can be obtained by commuting adjacent elements of y”. Roughly how to prove each:

either claim that 0 commutes satisfies definition of R or check that commute is self-inverting
relies on self-inverting nature, I think
messier but should still be provable

I’m pretty sure both (2) and (3) could be done with a brute force proof that considered all the pairings of patch types in their general cases. Start with all sequences of length 2, then 3 and I think at that point you could make an inductive argument to hit sequences of length n. This would be a lot of work, and I’m not convinced it could be fully automated.

Why would we want to show the above? Showing that R is a relation would tell us that sequences of patches are equivalent under commute. Now, combine this with the idea that context determines the state uniquely and now we know sets of patches uniquely determine your repository.

Jason Dagit: Darcs Hacking Sprint — Summary from Portland Team

2010-06-06T08:14:44+00:00

The weekend of 24-25 October 2008 was an International darcs hacking sprint! The sprint was a lot of fun and we’ll be having more. The sprint provides a very productive atmosphere for hacking.

We had a team in Brighton with posts from Day 1 and Day 2. We also had team in Paris but I don’t have a link for them.

Here are just some of the highlights from the Portland Team:

Adding language pragmas in all files:
- makes the code cleaner when it’s time to drop ghc6.6 support
- all required language extensions are now known
- makes it easier to check for Haskell’ compatibility
Removed OldFastPackedStrings
Replaced FastPackedStrings api in favor of Data.ByteString api
- lots of small optimizations, less pack/unpack, more standard
  ByteString code
- removed a fair bit of C code, new code compiles to same or
  faster assembly (Don checked)
cabalization:
- no autoconf or make needed
- cabal install tested and working on linux / osx, windows testing
  soon to follow
- builds out of the box w/ 6.8 and 6.10
- configure is much faster
module graph (depends on cabalization)
Duncan improved zlib
- soon to be available on hackage
- allows us to replace our own implementation of zlib bindings
  with the main stream one
- will make building on windows easier
- can use lazy bytestrings

Here are some random pictures from the Portland Sprint.
Packages we should consider:

The TODO list we made on the first day:

Checking on Team Brighton:

Duncan and Jason looking at the projector:

Ah, beautiful Portland in fall:

Adolfo Builes: GSoC Week 1: Getting to know you

2010-05-30T20:15:37+00:00

It has been almost a week since hacking started, during the week my activities were focused in getting to know Darcs code, working in the skeleton of a high level document for the cache system, and looking at one of the warm-up issues, I won't say I already know how it all works, but I'm picking up new things each day and that's fine :).

From the Darcs code section, I know most of my work would be done in the module Darcs.Repository where the cache code lives, I still need to go deeply and see why, how, and what for is it used in each of the submodules, and how the cache system fits in all the system.

The first sketch of the skeleton for the high level document can be found in [1], we are putting all the administrative stuff in [4] to help us to keep track of everything.

About one of the first warm-up issues (1503), I found a possible way to fix it, you can read more about it in the last comment in [2], I haven't sent a patch yet, but I hope to have it before Tuesday.

While I was looking at this issue we noticed something weird in the log and we found another issue [3], I have a test left to do, and check if that is a problem specific to the installation in my server or is something from the current branch.

For the coming week I plan to:

- Close issue 1503.
- Pick up other issue.
- Elaborate more in the "Internal" section of the document.

[1]- http://wiki.darcs.net/DarcsInternals/CacheSystem
[2]- http://bugs.darcs.net/issue1503
[3]- http://bugs.darcs.net/issue1854
[4]- http://wiki.darcs.net/GoogleSummerOfCode/2010-Cache

Darcs News: darcs weekly news #63

2010-05-30T10:21:48+00:00

News and discussions

Darcs 2.4.4 was released this week:
- http://lists.osuosl.org/pipermail/darcs-users/2010-May/024057.html
Eric talked about the ongoing work in fixing the bug about non-ASCII filenames:
- http://lists.osuosl.org/pipermail/darcs-users/2010-May/024103.html
Adolfo wrote his first Google Summer of Code blog report:
- http://abuiles.blogspot.com/2010/05/gsoc-week-1-getting-to-know-you.html

Issues resolved in the last week (2)

issue1763 Petr Rockai

use correct filename encoding in conflictors.
http://bugs.darcs.net/issue1763

issue1784 Guillaume Hoffmann

push and pull print remote address right away
http://bugs.darcs.net/issue1784

Patches applied in the last week (14)

See darcs wiki entry for details.

Darcs News: darcs weekly news #62

2010-05-18T06:49:30+00:00

News and discussions

Unexpectedly, we got a second student funded by the Google Summer of Code this year. Adolfo Builes will work on improving the global cache:
- http://lists.osuosl.org/pipermail/darcs-users/2010-May/024019.html
- http://abuiles.blogspot.com/2010/05/improving-darcs-performance-my-gsoc.html
Darcs 2.4.4 is going to be released in one week if no bug is discovered in its current source (most importantly under Windows):
We are looking for a new release manager:
- http://lists.osuosl.org/pipermail/darcs-users/2010-May/024038.html

Issues resolved in the last week (1)

issue1841 Jeremy Cowgar

Apply binary mode to ssh process and patch file handles.
http://bugs.darcs.net/issue1841

Patches applied in the last week (22)

See darcs wiki entry for details.

Adolfo Builes: Improving Darcs Performance - My GSoC Project.

2010-05-12T03:05:11+00:00

I was accepted for this year GSoC, I will be working for Haskell.org and my project will be focus in improvements of Darcs performance, specifically I'll work in the global cache, you can find a full specification of my project here.

I'm pretty excited about it and really looking forward to start working on this, also my mentor would by Eric Kow (a.k.a kowey).

I'll try to keep weekly updates of how my work is going, and let you know about the ups and downs that I find in my way.

Thanks Haskell.org, thanks Darcs and last but not least thanks Google for given us this awesome opportunity.

Darcs News: darcs weekly news #61

2010-05-09T22:42:00+00:00

News and discussions

Eric announced the release of darcs 2.4.3, which fixes critical bugs under Windows and fixes the performance regression of darcs convert:
Simon Michael proposed a cleanup of the repository format names and gathered a few answers and proposals:

http://lists.osuosl.org/pipermail/darcs-users/2010-May/023936.html
http://lists.osuosl.org/pipermail/darcs-users/2010-May/023969.html

Ganesh and Petr explained why it is not recommended anymore to build darcs with GHC 6.8:

http://lists.osuosl.org/pipermail/darcs-users/2010-April/023774.html
http://lists.osuosl.org/pipermail/darcs-users/2010-April/023776.html
http://wiki.darcs.net/Debian

Issues resolved in the last week (7)

issue64 Reinier Lamers

store metadata as UTF-8, autodetect UTF-8, and don't normalize to NFC
http://bugs.darcs.net/issue64

issue1232 Dino Morelli

darcs convert forgets _darcs/prefs/prefs
http://bugs.darcs.net/issue1232

issue1760 Petr Rockai

Fix working directory handling in Commands.Convert.
http://bugs.darcs.net/issue1760

issue1769 Matthias Kilian

add support for --match 'comment ...'
http://bugs.darcs.net/issue1769

issue1814 Petr Rockai

Include contrib/darcs-errors.hlint in release tarball.
http://bugs.darcs.net/issue1814

issue1824 Eric Kow

avoid PACKAGE_VERSION under Windows.
http://bugs.darcs.net/issue1824

issue1837 Petr Rockai

Add readWorking and use it in pristineFromWorking.
http://bugs.darcs.net/issue1837

Patches applied in the last week (115)

See darcs wiki entry for details.

Darcs News: darcs weekly news #60

2010-04-24T09:15:42+00:00

News and discussions

darcs 2.4.1 was released, fixing a couple of bugs of version 2.4.0. However, a serious bug under Windows was discovered, so Windows users should still stick to the 2.3.1 version, while 2.4.2 is not out:
Three students wrote their applications for this year's Summer of Code. Projects members discussed the priorities of darcs: network speed, local speed or UI ?

Issues resolved in the last week (3)

issue1159 Dmitry Kurochkin

smart caches union.
http://bugs.darcs.net/issue1159

issue1645 Eric Kow

bump hashed-storage to 0.4.11; don't follow symlinks.
http://bugs.darcs.net/issue1645

issue1756 Reinier Lamers

bump hashed-storage dependency to 0.4.10
http://bugs.darcs.net/issue1756

Patches applied in the last week (29)

See darcs wiki entry for details.

Max Battcher: musdex 10.04.04

2010-04-05T00:13:29+00:00

Just uploaded the latest musdex (10.04.04) to PyPI and making a quick note here. 10.04.04 is a somewhat significant refactor of musdex, which won't have too many obvious differences without a bit of configuration changing. The new additions is a customizable handler system that will allow for support for other archive formats (beyond just Zip archives), and a customizable way to specify post-extraction "formatters" to make extracted files even more useful under source control. There are no new handlers yet (although I'm considering building an SQLite one for proof-of-concept), but two built-in formatters that can opted in: xmllint and remove_carriage_returns. Details in the documentation including the basics you will need to build a custom formatter or handler.

Darcs News: darcs weekly news #59

2010-04-04T17:48:55+00:00

News and discussions

Lennart Kolmodin announced that darcs 2.4 was available in Gentoo Linux:
- http://lists.osuosl.org/pipermail/darcs-users/2010-March/023513.html
Mark Stosberg talked about darcs handling large repositories, then spawning a discussion on git vs darcs from the UI point of view:
- http://lists.osuosl.org/pipermail/darcs-users/2010-March/023526.html
- http://lists.osuosl.org/pipermail/darcs-users/2010-March/023539.html
If you haven't already read it on this blog, you can have a look at Eric's report from the last darcs hacking sprint:
- http://blog.darcs.net/2010/03/darcs-hacking-sprint-4-report.html

Issues resolved in the last week (4)

issue1208 dixiecko@gmail.com

wish: darcs trackdown --bisect
http://bugs.darcs.net/issue1208

issue1427 Guillaume Hoffmann

accept gzipped patch bundles in darcs apply
http://bugs.darcs.net/issue1427

issue1739 Reinier Lamers

make ColorPrinter handle characters > 255
http://bugs.darcs.net/issue1739

issue1756 Petr Rockai

new h-s handles file removals correctly in Index.
http://bugs.darcs.net/issue1756

issue1765 Reinier Lamers

refuse to remove non-tracked directories recursively
http://bugs.darcs.net/issue1765

Patches applied in the last week (29)

See darcs wiki entry for details.

Darcs News: darcs hacking sprint 4 report

2010-03-29T20:53:57+00:00

Updated 2010-03-29 with more photos (thanks, David Anderson!), a small correction and a note about the SFC

The Fourth Darcs Hacking Sprint took place last weekend (19 to 21 March) as part of the Zurich Haskell Hackathon. We had a very productive sprint, a bit of code written, polished off many key discussions, had a little beer and a lot of fun.

Overview

In this sprint, we worked on finishing some performance work for the upcoming Darcs 2.5 release this summer (hashed storage, patch index, global caches, inventory hashing); planning our work for the Darcs 2.6 release next year (smart servers, cache cleanup, darcs rebase) and working with new users of the Darcs library.

Issues resolved

issue643 darcs send -o output - Guillaume Hoffmann
issue1473 annotate command line - Stefan Wehr
issue1456 portable darcs dist - Guillaume Hoffmann

New Darcs Hackers

We're always happy to work with new Darcs developers. At this sprint, we were joined by four new contributors.

Guillaume Hoffmann

Guillaume has been writing our Darcs Weekly News articles for a year now. Over the weekend he got his first taste of Darcs hacking, knocking out three ProbablyEasy bugs (darcs dist internals, darcs send -o UI, darcs apply with gzipped patch bundles). Guillaume reports that he can see himself doing more of this in the future!

Steven Keuchel

Steven worked on a new feature to display the file contents hashed associated with any patch. This makes it easier for third party tools to inspect the patch files behind Darcs.

Stefan Wehr and David Leuschner

Stefan and David mostly worked on the Darcs Patch Manager, but to warm up, they tackled a couple of ProbablyEasy bugs, particularly a bug in darcs annotate that was affecting Redmine

Hacking continued...

Darcs hackers at work (Saturday)
Photo taken from David Anderson's Picasa site

Bugfix: Darcs on Windows shares

Salvatore tracked down the Windows regression on 2.4 that make Darcs not work on windows shares.

Performance: Fast darcs annotate

Benedikt Schmidt continued his work on the patch index (formerly known as the filecache). The patch index keeps track of which patches affect which files. This index will bring a big boost to darcs annotate performance, particularly for files which are affected by relative small number of patches.

Performance: Global cache

Luca continued his work on breaking up the global cache ($HOME/.darcs/cache) into buckets for faster access. Working with Reinier and Petr, Luca has developed an approach to migrating from old style caches to the new style bucketed ones. He has also improved the implementation to use hard links, to avoid disk space doubling and to preserve backwards compatibility with prior versions of Darcs.

Windows installer

Salvatore put together a nice Windows installer using the bamse package. It looks like we will be able to use this for the planned Darcs 2.5 release this summer. This work will also open the door to nicer integration with Windows tools, for example, using a bundled Tortoise SSH for better experience working with SSH passphrases.

Interactive cherry picking

Florent improved the quality of the Darcs cherry picking code, making it easier to fine tune our user interface and some day support graphical interfaces via the Darcs library. Witnessed list zippers for the win?

Interactive diff

Florent also started work on adding Darcs's interactive cherry picking to darcs diff, making it possible to choose a set of patches to view as a diff.

Performance: Hashed storage completion

Darcs has a representation of file and directory trees called slurpies. Petr polished off his work to replace the slurpies with his more efficient, general purpose hashed-storage library. Slurpies are going away, and Darcs will be faster for it. He and Ganesh also discussed how to gracefully transition from repositories created before the hashed-storage refactor.

Performance: Using tags when writing patches

Petr ported work by David Roundy to solve a scalability regression in hashed repositories. For darcs commands that write out patches, we had a naive hashing operation that does not account for the fact that patches behind tags cannot be modified. Darcs was unnecessarily traversing the entire sequence of patches (ie. O(n) time) when it could easily have been just traversing the sequence since the last tag.

UTF-8 metadata

Reinier continued to improve the encoding of Darcs patch metadata. Darcs is completely agnonstic with respect to the encoding of your files. Unfortunately, this agnostism extends to patch metadata (patch name, patch author), making it difficult for people to collaborate across different locales. To address this problem, Reinier has been working to make Darcs store its patch metadata in a single encoding (UTF-8) while gracefully supporting older patches (with metadata in potentially any encoding).

Discussions

The rebase discussion
Also from David's site

Release process

The Darcs 2.4 release was quite a tricky one to navigate. We found that bugs were only being flushed out on release candidate time and sometimes after the release proper.

We would like to encourage more people to try out Darcs work in progress and give us feedback early in the release process. After chatting about this, Reinier (with Ganesh, Eric and Petr) decided that as Release Manager, he would put out a Darcs alpha every 4 weeks.

In the future we may investigate automatic nightly builds via the buildbot and a platform support policy such as the one used by Tahoe.

Darcs patch index (fast darcs annotate)

Benedikt updated us on the recent status of his ongoing patch index work (formerly known as the filecache). We discussed the things that make the patch index convincing (permanant, repo-local, unique identifiers for files) the interaction between the patch index and the type witnesses and also ways of tuning the patch index performance and keeping it small.

We're looking forward to sharing the new patch index optimisation with you in upcoming releases. Darcs annotate may become a lot more useful in the next couple of releases!

Readable darcs annotate

Fast darcs annotate won't be useful if nobody can read it. Benedikt and Eric worked on designing a better output format darcs annotate. Taking a page from git blame, there will be one line per source file line, with columns for patch identifier, author name, date and finally the line. One of the design questions was how we should best refer to darcs patches, the current best candidate being a prefix of the darcs patch metadata hash.

Fast darcs over networks

Darcs get over networks is slow, painfully slow. Petr has suggested two priorities for improving the performance of network operations. The first would be to introduce a darcs optimize --http feature which would optimise the Darcs repository for fetching over a network (for example, by creating a "snapshot" of the pristine cache to be fetched in one go). The second priority would be develop a smart server that would provide darcs clients with only the files they need and in the optimal number of chunks. The two ideas combined would make an excellent Google Summer of Code project.

Darcs rebase

Prior to the sprint, Ganesh was working on a darcs rebase feature. Rebase will help Darcs users work with long term branches, and other cases where patch commutation by itself is not enough. At the sprint, Ganesh explained his work to everyone interested. Together we settled on a rough plan for the user interface. It looks like our new rebase command will offer a typically Darcs-ish twist: interactive cherry picking.

Darcs library

Ganesh and Florent talked with three teams building software in the Darcs ecosystem (DPM: Stephan Wehr and David Leuschner, Mac Darcs record GUI: Benedikt Huber and David Markvica, DarcsDen: Alex Suraci). There was a surprising degree of commonality.

The conversations have given us a much stronger sense of direction with the Darcs library. In particular, Ganesh is convinced that we should commit to our use witnesses - at the very least getting them completely finished so we can run with them, probably turning them on by default, and quite possibly dropping the non-witnesses builds.

Default switches

We held a quick roundtable discussion to settle some decisions on Darcs default switches that have been hanging in the air. Our decisions for Darcs 2.5:

--no-set-scripts-executable [unchanged]
pull/push/send --no-set-default
send --edit-description
record --no-test
check --no-test

Performance presentations

Petr and Benedkit gave lighting talks, showing some of our recent performance work to the Haskell community. Some exciting numbers from Benedikt's work (notes) include a 6 second darcs annotate on a file in the GHC repository (previously this did not complete within a half hour).

Google Summer of Code

We discussed our priorities for this year's Google Summer of Code. We have decided that we would focus our attention on performance issues. If we had two GSoC students this year, we would be mainly interested in dividing them between

network performance: developing a smart server for much faster darcs get and pull over a network
local performance: performing a comprehensive overhaul of the Darcs hashed file cache handling

We also discussed ways to make the best use of our students' time. The Darcs team has participated in GSoC twice and learning a lot from the experience. This year we would like to see if we could publish some clear guidelines both on what we expect from GSoC students and what they can expect from us. Watch the mailing list for more discussion on this topic.

Budding Ecosystem

We were pleasantly suprised to find ourselves with users of the (still unstable) Darcs API. These new arrivals give us the feeling that the collection of related software is coalescing into a new Darcs ecosystem.

Darcs Patch Manager

David Leuschner and Stefan Wehr worked on an exciting new patch management program for project maintainers. The Darcs Patch Manager (DPM) offers a new way for repository maintainers to keep track of incoming Darcs patches, including their amendements and dependencies.

$ dpm -r MAIN_REPO -s DPM_DB list
very cool feature [State: OPEN]
2481 Tue Mar 16 17:50:23  2010 Dave Devloper <dave@example.com>
State: UNDECIDED, Reviewed: no
added
7861 Tue Mar 16 17:20:45  2010 Dave Devloper <dave@example.com>
State: REJECTED, Reviewed: yes
marked as rejected: one minor bug
some other patch [State: OPEN]
7631 Tue Mar 16 13:15:20  2010 Eric E. <eric@example.com>
State: REJECTED, Reviewed: yes
added
...

Towards the end of the hackathon, Stefan gave a nice short demo of DPM in action and deftly avoided the wrath of the demo Gods.

MacOS X GUI for Darcs record

Benedikt Huber and David Markvica started work on a graphical interface to the Darcs record command. One key twist is that they make use of the Darcs API to get the kind of dependency-tracking interactiveness goodness that Darcs offers. Bendedikt and Huber report that they have spent most of the hackathon getting to grips with the library. Darcs type witnesses were very helpful for avoiding errors, but they also impose a steep learning curve.

Darcsden

Alex Suraci and Simon Michael made several improvements to Darcsden, an open source hosting solution (akin to Github and Patch-tag). Some recent changes were Atom feeds, the ability to view forks of your repository and cherry-pick patches from them (work in progress). Darcsden also makes use of the Darcs API.

Darcsden fork viewer

Want to host Darcs Hacking Sprint 2010-10?

The Darcs Team would like to hold hacking sprints twice a year. These sprints are an important occassion for us to hold design discussions, hack some code, train new Darcs hackers and generally bond as a team.

Do you think you can help? Please get in touch with me if you think you may be able to host a group of around 20 Darcs hackers one of these October or November weekends.

Thanks!

Getting over 75 Haskell hackers into Zürich and having them up and running on arrival (Swiss power plugs notwithstanding) was no easy task! We'd like to thank Johan Tibell, David Anderson and the rest of the Google Crew for their hard work organising this hackathon.

Thanks also to the generous donors who chipped into our 2010 Darcs Travel Fund. We'll be looking forward to using the leftover cash for the upcoming 5th Darcs Hacking Sprint in October or November.

Speaking of donors, we'd particularly like to thank the Software Freedom Conservancy for providing us with the infrastructure (both legal and technical) for accepting donations and holding assets such as the darcs.net domain. Meta projects like the SFC are crucial for the success of volunteer-driven open source projects such as Darcs.

Finally here are some words from happy Darcs hackers:

The sprint was a wonderful social occasion, and it was great meeting most of the Darcs hackers, and also seeing other Haskell hackers interested in working in the Darcs ecosystem. I especially enjoyed teaching them how to use our API. -- Florent
The atmosphere was wonderful and I consider the sprint to have been very productive overall. -- Petr
This is coolest thing I ever did -- Luca

See you in half a year!

Participants

We had ten Darcs hackers in Zürich along with four Haskellers using the Darcs API to do awesome things (plus two more on IRC).

Florent Becker
Guillaume Hoffmann
Eric Kow
Reinier Lamers
Ganesh Sittampalam
Petr Rockai
Salvatore Insalaco
Luca Molteni
Benedikt Schmidt
Steve Keuchel
Benedikt Huber
David Markvica
Stefan Wehr
David Leuschner
Simon Michael [IRC] - Darcsden
Alex Suraci [IRC] - Darcsden

(Group photo from Johan Tibel's ZuriHac summary post)

Darcs News: darcs weekly news #58

2010-03-21T21:08:07+00:00

News and discussions

The fourth Darcs hacking sprint has just ended! A dozen of Darcs developers were present at Zurich for three days. Before a full-length report is posted, you can consult the wiki for a rough outline of the work done:
- http://koweycode.blogspot.com/2010/03/darcs-team-at-zurihac.html
- http://wiki.darcs.net/Sprints/2010-03
GHC version support was discussed in depth, and it was decided that Darcs would only be guaranteed to compile with the last two stable branches of GHC:

Issues resolved in the last week (5)

issue643 Guillaume Hoffmann

darcs send -o outputs remote repo email address
http://bugs.darcs.net/issue643

issue1456 Guillaume Hoffmann

by using Tar and GZip haskell modules for dist
http://bugs.darcs.net/issue1456

issue1473 mail@stefanwehr.de

annotate command accepts "." and treats "" as invalid input
http://bugs.darcs.net/issue1473

issue1750 Eric Kow

uncover help text for darcs show pristine.
http://bugs.darcs.net/issue1750

issue1767 Ganesh Sittampalam

send CRC warnings to stderr
http://bugs.darcs.net/issue1767

Patches applied in the last week (25)

See darcs wiki entry for details.

Eric Kow: darcs team at ZuriHac

2010-03-20T15:19:45+00:00

Just a quick photo showing what happens when you give a bunch of Darcs hackers a flipchart and a marker pen...

(With thanks to David Anderson for gamely taking this photo for our collective memory)

This was the result of a lively discussion on the future darcs rebase feature, which will make maintaining long-term branches in Darcs a lot easier. Perhaps it'll be ready in early 2011. We'll be sure to take our time to get this right...