hub.darcs.net :: Logan-010 -> darcsden2 -> files

Darcsden fork

root

darcsden is a "code forge" for darcs repositories, allowing one or more darcs repos to be published and managed via web or ssh interfaces, online or on your local machine. It provides several executables:

  • darcsden - the web application, for browsing and managing users, repos, issues
  • darcsden-ssh - OpenSSH authorization and restricted darcs push/pull commands
  • darcsden-post-hook - a helper for updating issues from commit messages
  • den - a command line tool for managing a local darcsden instance

darcsden can be configured for local single-user use, or as a shared multi-user server. In single-user mode it stores all data on the filesystem. In multi-user mode it stores metadata and sessions in SQLite, in addition to darcs repos on the filesystem.

http://hackage.haskell.org/package/darcsden http://hub.darcs.net/simon/darcsden

darcs hub

https://hub.darcs.net is a central community instance of darcsden, operated by darcsden's maintainer. You can explore it and use it free of charge. If it meets your needs, there's no need to install darcsden yourself. Otherwise, read on.

Installing

Using Nix

With flakes enabled, build the project with:

nix build
./result/bin/darcsden --help

The result includes darcsden, den, darcsden-post-hook, darcsden-ssh, and the web assets. The flake pins NixOS 26.05 with GHC 9.10.3 and Darcs 2.18.5. Tests are run separately from nix build using the commands below.

Enter the development environment with:

nix develop
cabal build --project-file=cabal.project.flake --offline

The shell includes GHC, cabal-install, Haskell Language Server (built for GHC 9.10.3), ormolu, cabal-fmt, hlint, ghcid, git, pkg-config, darcs, OpenSSH, Python 3, and hpack, together with the project's library, test and build-tool dependencies. The Nix package also supplies Darcs and OpenSSH on the executables' runtime PATH. Use cabal.project.flake to use these prebuilt dependencies offline.

Using Stack or Cabal

For standalone Cabal/Stack builds, use GHC 9.8.4 (Stack LTS 23.28). Nix uses GHC 9.10.3 and includes Darcs's upstream GHC/Cabal compatibility patches; the unpatched Hackage Darcs 2.18.5 release supports GHC through 9.8. Cabal, Stack and Nix use the same application dependency ranges. package.yaml is authoritative; run hpack after editing it and keep the generated darcsden.cabal in sync. The small HSP template runtime is vendored with mtl 2.3 compatibility fixes in submodules/hsp-0.10.0; see its README.darcsden.md. The old submodules/ssh checkout is no longer used.

Installing with stack:

stack install

This will install the executables in ~/.local/bin or the windows equivalent.

Installing with cabal:

cabal build all -w ghc-9.8.4
cabal install -w ghc-9.8.4 exe:darcsden exe:den exe:darcsden-ssh exe:darcsden-post-hook

This will install executables in ~/.cabal/bin or the windows equivalent.

SSH hosting with OpenSSH

darcsden-ssh is now a restricted OpenSSH command handler. OpenSSH handles the transport, modern key exchange, host keys and signature verification. Deploy a dedicated sshd using manifests/sshd_config.example, with absolute paths to the installed executable and configuration. OpenSSH requires the AuthorizedKeysCommand executable and its parent directories to be root-owned and not group/world-writable. Run the command as the Unix service account that owns the repositories and can read and write the SQLite database directory.

All SSH clients connect as that shared Unix account (sshUser = darcsden by default). Application identity comes from the public key registered in the web UI. For example:

SSH_PORT=8901 darcs clone darcsden@example.com:alice/project
SSH_PORT=8901 darcs push --all darcsden@example.com:alice/project
ssh -p 8901 darcsden@example.com initialise new-project 'Project description'

Set sshUser and sshPort in darcsden.conf to match the OpenSSH service; these settings control the SSH instructions displayed by the web UI. Replace old alice@host:project remotes with darcsden@host:alice/project. Each public key must belong to exactly one application account. Ed25519, RSA (using OpenSSH's RSA-SHA2 signatures), ECDSA and security-key public keys are supported. DSA keys must be replaced. Keys are checked again when a command starts, so revoking a key also invalidates an old generated authorized_keys entry. Interactive shells and forwarding are disabled.

Verification

In nix develop:

cabal build all --enable-tests --project-file=cabal.project.flake --offline
cabal test darcsden-test-core darcsden-test-http --project-file=cabal.project.flake --offline
python3 -m unittest discover -s tests -p 'test_*.py'
python3 tests/sqlite-smoke.py "$(cabal list-bin exe:darcsden --project-file=cabal.project.flake)"
DARCSDEN_SSH="$(cabal list-bin exe:darcsden-ssh --project-file=cabal.project.flake)" \
  cabal test darcsden-test-core --project-file=cabal.project.flake --offline --test-show-details=direct

The core suite tests SSH command parsing, stored password compatibility, configuration, and SQLite metadata, revision conflicts, private repository visibility, session expiry, rollback and concurrent updates in temporary databases. Setting DARCSDEN_SSH adds a real unprivileged OpenSSH/Darcs clone/push/clone test with an Ed25519 client key and SQLite, including key revocation and rejected commands. The HTTP suite runs a local server and real Darcs repositories. The Selenium suite (darcsden-test) additionally needs Selenium and a browser; OAuth tests need external accounts.

The browser suite uses temporary SQLite databases and repository directories. With a Selenium 3 server and HTMLUnit driver listening on port 4444, run:

cabal test darcsden-test --project-file=cabal.project.flake --offline --test-show-details=direct

Installing on Windows

The web server depends on OpenSSL, which Stack/Cabal will not install. The post-hook now uses pure-Haskell regex-tdfa, with no PCRE C library. SSH hosting requires an OpenSSH service. If you do not deploy it on Windows, disable its handler with:

stack install --flag darcsden:-ssh

or:

cabal install -f-ssh

Configuration

Darcsden can be configured via a "darcsden.conf" file in the startup directory, or specified with the --config option:

darcsden --config /path/to/darcsden.conf

The configuration file has the following format (includes all currently available options from the default settings). Note that you do not need to set all options, but only those that are different from the default configuration.

# for logs and user repositories, also important for the ssh key locations
homeDir = .
# for serving the page (if not specified, this is automatically detected
# from the cabal install destination)
publicDir = ./public
# log file locations
accessLog = ./log/access.log
errorLog = ./log/error.log

# Enable user registration?
multiuser = True
# Enable optional issue trackers?
issuetrackers = True
# Delete repos from the filesystem when deleted in the web UI?
deletefiles = True
# Support darcs-1 repositories?
supportdarcs1 = True

# resource limits
# granularity is 5s, precision is +/-5s
maxRequestTime = 30
maxFileDisplaySize = 200000
maxPatchDisplaySize = 200000

# Enable github integration, also needs githubAppIdVarName
# and githubAppPwdVarName.
githubOAuth = True

# Enable Google integration, also needs googleAppIdVarName
# and googleAppPwdVarName.
googleOAuth = True

# To configure github integration, register an application at github.
# Store the client id and client secret as environment variables.
githubAppIdVarName = GITHUB_CLIENT_ID
githubAppPwdVarName = GITHUB_CLIENT_SECRET

# To configure google integration, register an application at google.
# Store the client id and client secret as environment variables.
googleAppIdVarName = GOOGLE_CLIENT_ID
googleAppPwdVarName = GOOGLE_CLIENT_SECRET

# send address: you need to be able to send mails from
# this address using the default sendmail executable
sendName = darcdsden
sendEmail = darcsden@localhost

# this name/email is shown to users for contact information
adminName = admin
adminEmail = webmaster@localhost

# canonical public url of the web app
# this is important eg for loading stylesheets and js
# it must end with a slash /
baseUrl = http://localhost:8900/
hostname = localhost
# port on which the http server listens
httpPort = 8900
# port for ssh connections
sshPort = 22
# shared OpenSSH Unix account
sshUser = darcsden

# SQLite file for metadata and sessions (relative to homeDir, or absolute)
sqlitePath = darcsden.sqlite3

Some features require extra configuration to work:

  • OAuth:

    • Create developer applications for the oauth services:
    • Github: https://github.com/settings/applications
    • Google: https://code.google.com/apis/console
    • Set callback urls:
    • Github: same as your baseUrl
    • Google: multiple callbacks:
      • baseUrl ++ "register/google/response"
      • baseUrl ++ "login/google/response"
      • baseUrl ++ "sync/google/response"
    • Enter Client ID and Client secret into environment variables:
    • Github: Add these lines to your shell startup export GITHUB_CLIENT_ID=your_client_id export GITHUB_CLIENT_SECRET=your_client_secret
    • Google: Add these lines to your shell startup export GOOGLE_CLIENT_ID=your_client_id export GOOGLE_CLIENT_SECRET=your_client_secret
  • Forgot Password:
    • Configure your default sendmail to be able to send emails
    • Edit sendEmail, and sendName to the email address you can use
  • Webdriver Tests:
    • Download and start selenium server from http://docs.seleniumhq.org/download/
    • run ./dist/build/darcsden-test/darcsden-test
    • If you wish to run oauth tests:
      • Configure oauth
      • Add a test google and github account by adding the appropriate environment variables:
        • export GOOGLE_USERNAME, GOOGLE_PASSWORD, GITHUB_USERNAME, GITHUB_PASSWORD.
      • run ./dist/build/darcsden-test/darcsden-test --oauth

Web assets

The web app expects darcsden's public/ directory to be accessible as ./public, so run darcsden from within the main source directory. Or..

Web server integration

If you run a web server in front of the darcsden web app, configure it to serve /public/* requests from darcsden's public/ directory.

Some partial web server config examples.

Apache:

<VirtualHost *:80>
ServerName my.host.name
RewriteEngine on
ProxyPreserveHost On
ProxyErrorOverride On
RewriteRule ^/(.*) http://127.0.0.1:8900/$1 [P]
</VirtualHost>

Nginx:

server {
  listen 0.0.0.0:80;
  server_name my.host.name;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://127.0.0.1:8900;
  }
}

Lighttpd:

$HTTP["host"] == "my.host.name" {
    proxy.server = (
        "" => (
            ( "host" => "127.0.0.1", "port" => "8900" )
        )
    )
}

Multi-user setup

Extra configuration needed for running in multi-user mode:

Databases

SQLite stores users, repository metadata, issues, bundles, comments and web sessions in a single file. No database server is needed. Set sqlitePath in darcsden.conf (default: darcsden.sqlite3 under homeDir). Use the same configuration for the web server, SSH handler and post-hook. The service account must be able to write the database and its parent directory, including SQLite's WAL and shared-memory files. Keep the database outside the public web assets. The schema is created automatically; it can also be initialised explicitly:

darcsden --install

Installation is idempotent and preserves existing data. SQLite uses WAL mode and a busy timeout so the web server and SSH processes can share the file. Back up the database with SQLite's backup API or .backup command, together with the users/ repository directory.

Migrating an existing CouchDB installation

Stop the old web server, SSH service and post-hook writers. Export each of the five CouchDB databases using /_all_docs?include_docs=true, saving the responses as users.json, repositories.json, issues.json, bundles.json and comments.json in one directory. For example (add authentication as required):

curl &#39;http://localhost:5984/users/_all_docs?include_docs=true&#39; > exports/users.json
# Repeat for repositories, issues, bundles and comments.
darcsden --config /path/to/darcsden.conf --install
python3 scripts/import-couchdb.py /path/to/darcsden.sqlite3 exports

The destination must be empty. The importer runs in a single transaction, preserves IDs, references, password hashes and public keys, and skips CouchDB design documents. Keep the existing users/ repository directory under homeDir. Configure every service with the same sqlitePath, then start the new application. Redis sessions are not imported; users log in again. The old CouchDB/Redis host, port and credential settings are no longer used.

User account

You should have a "darcsden" user account for running the app and owning the files:

adduser --system --home /home/darcsden --shell /bin/bash --group --gecos DarcsDen darcsden

OpenSSH manages the server host keys; no private SSH key is required in the application's home directory.

To avoid obscure errors due to mixed file ownership in ~/.darcs/cache, always run darcsden as the darcsden user, with $HOME set properly. So if using sudo, include the -H flag, like this:

sudo -Hu darcsden COMMAND...

Running

Running in single-user mode

From the main darcsden source directory (with ./public/), run:

darcsden

and open your web browser to http://localhost:8900 (or other url configured in Settings.hs).

Running in multi-user mode

The web server should run as the darcsden user. OpenSSH starts the darcsden-ssh handler as that account for each authenticated session.

For persistent operation, you could run them manually inside persistent dtach or screen sessions:

sudo -HE -u darcsden darcsden

Start a dedicated OpenSSH daemon using the configuration described above:

sudo /path/to/sshd -D -e -f /etc/ssh/darcsden_sshd_config

Or better, configure them as daemons, eg using systemd or supervisord. Sample supervisord.conf entries:

[program:darcsden]
command=sudo -EHu darcsden /path/to/darcsden
directory=/home/darcsden
priority=3
redirect_stderr=true
autostart=true
autorestart=true
environment=LANG="en_US.UTF-8"

[program:darcsden-sshd]
command=/path/to/sshd -D -e -f /etc/ssh/darcsden_sshd_config
directory=/home/darcsden
priority=3
redirect_stderr=true
autostart=true
autorestart=true
environment=LANG="en_US.UTF-8"

Contributing

darcsden is a small, relatively clean codebase that is fun to hack on. Discussion takes place on the #darcs IRC channel, and useful changes will quickly be deployed at hub.darcs.net, providing a tight dogfooding/feedback loop. Here's how to contribute a patch there:

  1. register at hub.darcs.net
  2. add your ssh key in settings so you can push
  3. fork your own branch: http://hub.darcs.net/simon/darcsden , fork
  4. copy to your machine: darcs get http://hub.darcs.net/yourname/darcsden
  5. make changes, darcs record
  6. push to hub: darcs push yourname@hub.darcs.net:darcsden --set-default
  7. your change will appear at http://hub.darcs.net/simon/darcsden/patches
  8. discuss on #darcs, or ping me (sm, simon@joyful.com) to merge it

Credits

Alex Suraci created darcsden.

Simon Michael is the current maintainer of darcsden, and operator of darcs hub.

Recent contributors include Alp Mestanogullari, Jeffrey Chu, Ganesh Sittampalam, BSRK Aditya (sponsored by Google's Summer of Code), and Bertram Felgenhauer.

darcsden depends many fine FOSS projects and libraries, including Darcs, Snap, GHC, Twitter Bootstrap, JQuery, and more.

  • darcsden 1.2 beta built Jun 23 201810:48:19
  • report problems to simon