diff options
author | Angel Abad <angel@debian.org> | 2021-01-31 01:48:31 -0800 |
---|---|---|
committer | Angel Abad <angel@debian.org> | 2021-01-31 01:48:31 -0800 |
commit | 60ecf9d1b071e704c88b2e96c54674331ef450ea (patch) | |
tree | 37d40d92887dfcaa77e45eb88d992dee63cef819 | |
parent | 72ea8cf6e6566c0355549912a87b2eae24cdd27f (diff) | |
parent | ce323c2bcee31fe78150187cb014b200387eb83f (diff) |
Record yadm (3.0.2-1) in archive suite sid
72 files changed, 2461 insertions, 1608 deletions
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 4977e4b..392a24d 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -14,6 +14,7 @@ areas. To contribute, you can: +* Star the yadm repo, the star count helps others discover yadm. * Report [bugs](#reporting-a-bug) * Request [features/enhancements](#suggesting-a-feature-or-enhancement) * Contribute changes to [code, tests](#contributing-code), and [documentation](#improving-documentation) @@ -206,7 +207,7 @@ these principles when making changes. ``` 4. Verify you can run the test harness. _(This will require dependencies: - `make`, `docker`, and `docker-compose`)_. + `make` and `docker`)_. ```text $ make test diff --git a/.github/workflows/schedule.yml b/.github/workflows/schedule.yml new file mode 100644 index 0000000..6f1e267 --- /dev/null +++ b/.github/workflows/schedule.yml @@ -0,0 +1,20 @@ +--- +name: Scheduled Site Tests +on: # yamllint disable-line rule:truthy + schedule: + - cron: "0 0 1 * *" # Monthly +jobs: + Tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + with: + ref: gh-pages + - run: >- + docker create -t + --name yadm-website + --entrypoint test/validate + yadm/jekyll:2019-10-17; + docker cp ./ yadm-website:/srv/jekyll + - name: Test Site + run: docker start yadm-website -a diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..1dae7cf --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,13 @@ +--- +name: Tests +on: # yamllint disable-line rule:truthy + - push + - pull_request + - workflow_dispatch +jobs: + Tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Tests + run: make test @@ -2,5 +2,6 @@ .jekyll-metadata .pytest_cache .sass-cache +.testyadm _site testenv diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index e340cd2..0000000 --- a/.travis.yml +++ /dev/null @@ -1,8 +0,0 @@ ---- -language: minimal -services: - - docker -before_install: - - docker pull yadm/testbed:2020-01-20 -script: - - docker run -t --rm -v "$PWD:/yadm:ro" yadm/testbed:2020-01-20 @@ -1,3 +1,37 @@ +3.0.2 + * Fix parsing by sh (#299) + +3.0.1 + * Improve handling of submodules at upgrade (#284, #285, #293) + * Improve Zsh completions (#292, #298) + * Use stderr for error messages (#297) + +3.0.0 + * Support encryption with OpenSSL (#138) + * Support "include" directive in built-in template processor (#255) + * Support extensions for alternate files and templates (#257) + * Improve support for default branches (#231, #232) + * Add --version and --help as yadm internal commands (#267) + * Improve support for XDG base directory specification + * Use XDG_DATA_HOME used for encrypted data and repository (#208) + * Default repo is now ~/.local/share/yadm/repo.git + * Default encrypted archive is now ~/.local/share/yadm/archive + * Improve shell completions (#238, #274, #275) + * Remove support for YADM_COMPATIBILITY=1 (#242) + * Remove deprecated option cygwin-copy + * Fix template mode inheritance on FreeBSD (#243, #246) + * Fix hook execution under MinGW (#150) + * Improve compatibility with Oil shell (#210) + +2.5.0 + * Support for transcrypt (#197) + * Support ESH templates (#220) + * Preserve file mode of template (#193) + * Fish shell completions (#224) + * Fix alt processing when worktree is `/` (#198) + * Assert config directory if missing (#226, #227) + * Documentation improvements (#229) + 2.4.0 * Support multiple keys in `yadm.gpg-recipient` (#139) * Ensure all templates are written atomically (#142) @@ -7,6 +41,7 @@ * Improve identification of WSL (#196) * Fix troff warnings emitted by man page (#195) * Write encrypt-based exclusions during decrypt + 2.3.0 * Support git-crypt (#168) * Support specifying a command after `yadm enter` diff --git a/CONTRIBUTORS b/CONTRIBUTORS index 319b463..ad97610 100644 --- a/CONTRIBUTORS +++ b/CONTRIBUTORS @@ -1,29 +1,39 @@ CONTRIBUTORS Tim Byrne +Erik Flodin Martin Zuther -Ross Smith II +Jan Schulz +Jonathan Daigle +Luis López +Tin Lai Espen Henriksen Cameron Eagans Klas Mellbourn +Ross Smith II +Tomas Cernaj +jonasc +Chad Wade Day, Jr +Sébastien Gross David Mandelberg Daniel Gray -Jan Schulz +Paraplegic Racehorse +japm48 Siôn Le Roux -Stig Palmquist -Sébastien Gross -Thomas Luzat -Tomas Cernaj +Mateusz Piotrowski Uroš Golja -con-f-use -japm48 -Brayden Banks -jonasc -Daniel Wagenknecht +Satoshi Ohki Franciszek Madej -Mateusz Piotrowski -Paraplegic Racehorse +Daniel Wagenknecht +Stig Palmquist Patrick Hof -Russ Allbery -Satoshi Ohki +con-f-use +Travis A. Everett Sheng Yang +Adam Jimerson +addshore +Tim Condit +Thomas Luzat +Russ Allbery +Brayden Banks +Alexandre GV diff --git a/Dockerfile b/Dockerfile deleted file mode 100644 index 469fe54..0000000 --- a/Dockerfile +++ /dev/null @@ -1,50 +0,0 @@ -FROM ubuntu:18.04 -MAINTAINER Tim Byrne <sultan@locehilios.com> - -# No input during build -ENV DEBIAN_FRONTEND noninteractive - -# UTF8 locale -RUN apt-get update && apt-get install -y locales -RUN locale-gen en_US.UTF-8 -ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8' - -# Convenience settings for the testbed's root account -RUN echo 'set -o vi' >> /root/.bashrc - -# Install prerequisites -RUN \ - apt-get update && \ - apt-get install -y \ - curl \ - expect \ - git \ - gnupg \ - lsb-release \ - make \ - man \ - python3-pip \ - shellcheck=0.4.6-1 \ - vim \ - ; -RUN pip3 install \ - envtpl \ - j2cli \ - flake8==3.7.8 \ - pylint==2.4.1 \ - pytest==5.1.3 \ - yamllint==1.17.0 \ - ; - -# Create a flag to identify when running inside the yadm testbed -RUN touch /.yadmtestbed - -# /yadm will be the work directory for all tests -# docker commands should mount the local yadm project as /yadm -WORKDIR /yadm - -# Create a Makefile to be used if no /yadm volume is mounted -RUN echo "test:\n\t@echo 'The yadm project must be mounted at /yadm'\n\t@echo 'Try using a docker parameter like -v \"\$\$PWD:/yadm:ro\"'\n\t@false" > /yadm/Makefile - -# By default, run all tests defined -CMD make test @@ -1,4 +1,5 @@ PYTESTS = $(wildcard test/test_*.py) +IMAGE = yadm/testbed:2020-12-29 .PHONY: all all: @@ -22,7 +23,7 @@ usage: @echo @echo ' make testhost [version=VERSION]' @echo ' - Create an ephemeral container for doing adhoc yadm testing. The' - @echo ' HEAD revision of yadm will be used unless "version" is' + @echo ' working copy version of yadm will be used unless "version" is' @echo ' specified. "version" can be set to any commit, branch, tag, etc.' @echo ' The targeted "version" will be retrieved from the repo, and' @echo ' linked into the container as a local volume.' @@ -32,7 +33,7 @@ usage: @echo ' exiting the shell, a log of the commands used to illustrate the' @echo ' problem will be written to the file "script.txt". This file can' @echo ' be useful to developers to make a repeatable test for the' - @echo ' problem.' + @echo ' problem. The version parameter works as for "testhost" above.' @echo @echo 'LINTING' @echo @@ -81,7 +82,7 @@ usage: # Make it possible to run make specifying a py.test test file .PHONY: $(PYTESTS) $(PYTESTS): - @$(MAKE) test testargs="-k $@ $(testargs)" + @$(MAKE) test testargs="$@ $(testargs)" %.py: @$(MAKE) test testargs="-k $@ $(testargs)" @@ -92,65 +93,75 @@ test: cd /yadm && \ py.test -v $(testargs); \ else \ - if command -v "docker-compose" &> /dev/null; then \ - docker-compose run --rm testbed make test testargs="$(testargs)"; \ - else \ - echo "Sorry, this make test requires docker-compose to be installed."; \ - false; \ - fi \ + $(MAKE) -s require-docker && \ + docker run \ + --rm -t$(shell test -t 0 && echo i) \ + -v "$(CURDIR):/yadm:ro" \ + $(IMAGE) \ + make test testargs="$(testargs)"; \ + fi + +.PHONY: .testyadm +.testyadm: version ?= local +.testyadm: + @rm -f $@ + @if [ "$(version)" = "local" ]; then \ + ln -sf yadm $@; \ + echo "Using local yadm ($$(git describe --tags --dirty))"; \ + else \ + git show $(version):yadm > $@; \ + echo "Using yadm version $$(git describe --tags $(version))"; \ fi + @chmod a+x $@ .PHONY: testhost -testhost: require-docker - @version=HEAD - @rm -rf /tmp/testhost - @git show $(version):yadm > /tmp/testhost - @chmod a+x /tmp/testhost - @echo Starting testhost version=\"$$version\" +testhost: require-docker .testyadm + @echo "Starting testhost" @docker run \ -w /root \ --hostname testhost \ --rm -it \ - -v "/tmp/testhost:/bin/yadm:ro" \ - yadm/testbed:2020-01-20 \ + -v "$(CURDIR)/.testyadm:/bin/yadm:ro" \ + $(IMAGE) \ bash -l .PHONY: scripthost -scripthost: require-docker - @version=HEAD - @rm -rf /tmp/testhost - @git show $(version):yadm > /tmp/testhost - @chmod a+x /tmp/testhost - @echo Starting scripthost version=\"$$version\" \(recording script\) +scripthost: require-docker .testyadm + @echo "Starting scripthost \(recording script\)" @printf '' > script.gz @docker run \ -w /root \ --hostname scripthost \ --rm -it \ - -v "$$PWD/script.gz:/script.gz:rw" \ - -v "/tmp/testhost:/bin/yadm:ro" \ - yadm/testbed:2020-01-20 \ + -v "$(CURDIR)/script.gz:/script.gz:rw" \ + -v "$(CURDIR)/.testyadm:/bin/yadm:ro" \ + $(IMAGE) \ bash -c "script /tmp/script -q -c 'bash -l'; gzip < /tmp/script > /script.gz" @echo - @echo "Script saved to $$PWD/script.gz" + @echo "Script saved to $(CURDIR)/script.gz" .PHONY: testenv testenv: @echo 'Creating a local virtual environment in "testenv/"' @echo + @rm -rf testenv python3 -m venv --clear testenv testenv/bin/pip3 install --upgrade pip setuptools - testenv/bin/pip3 install --upgrade \ - flake8==3.7.8 \ - pylint==2.4.1 \ - pytest==5.1.3 \ - yamllint==1.17.0 \ - ; + testenv/bin/pip3 install --upgrade -r test/requirements.txt; + @for v in $$(sed -En -e 's:.*/yadm-([0-9.]+)$$:\1:p' test/Dockerfile); do \ + git show $$v:yadm > testenv/bin/yadm-$$v; \ + chmod +x testenv/bin/yadm-$$v; \ + done @echo @echo 'To activate this test environment type:' @echo ' source testenv/bin/activate' +.PHONY: image +image: + @docker build -f test/Dockerfile . -t "$(IMAGE)" + + .PHONY: man man: @groff -man -Tascii ./yadm.1 | less @@ -167,9 +178,14 @@ yadm.md: yadm.1 @groff -man -Tascii ./yadm.1 | col -bx | sed 's/^[A-Z]/## &/g' | sed '/yadm(1)/d' > yadm.md .PHONY: contrib +contrib: SHELL = /bin/bash contrib: - @echo "CONTRIBUTORS\n" > CONTRIBUTORS - @git shortlog -ns master gh-pages develop dev-pages | cut -f2 >> CONTRIBUTORS + @echo -e "CONTRIBUTORS\n" > CONTRIBUTORS + @IFS=$$'\n'; for author in $$(git shortlog -ns master gh-pages develop dev-pages | cut -f2); do \ + git log master gh-pages develop dev-pages \ + --author="$$author" --format=tformat: --numstat | \ + awk "{sum += \$$1 + \$$2} END {print sum \"\t\" \"$$author\"}"; \ + done | sort -nr | cut -f2 >> CONTRIBUTORS .PHONY: install install: @@ -192,7 +208,7 @@ sync-clock: .PHONY: require-docker require-docker: - @if ! command -v "docker" &> /dev/null; then \ + @if ! command -v "docker" > /dev/null 2>&1; then \ echo "Sorry, this make target requires docker to be installed."; \ false; \ fi @@ -8,45 +8,80 @@ [![Master Update][master-date]][master-commits] [![Develop Update][develop-date]][develop-commits] [![Website Update][website-date]][website-commits]<br /> -[![Master Status][master-badge]][travis-ci] -[![Develop Status][develop-badge]][travis-ci] -[![GH Pages Status][gh-pages-badge]][travis-ci] -[![Dev Pages Status][dev-pages-badge]][travis-ci] +[![Master Status][master-badge]][workflow-master] +[![Develop Status][develop-badge]][workflow-develop] +[![GH Pages Status][gh-pages-badge]][workflow-gh-pages] +[![Dev Pages Status][dev-pages-badge]][workflow-dev-pages] [https://yadm.io/][website-link] -[**yadm**][website-link] is a tool for managing [dotfiles][]. +**yadm** is a tool for managing [dotfiles][]. * Based on [Git][], with full range of Git's features -* Supports system-specific alternative files -* Encryption of private data using [GnuPG][] +* Supports system-specific alternative files or templated files +* Encryption of private data using [GnuPG][], [OpenSSL][], [transcrypt][], or + [git-crypt][] * Customizable initialization (bootstrapping) +* Customizable hooks for before and after any operation -Features, usage, examples and installation instructions can be found on the -[website][website-link]. +Complete features, usage, examples and installation instructions can be found on +the [yadm.io][website-link] website. + +## A very quick tour + + # Initialize a new repository + yadm init + + # Clone an existing repository + yadm clone <url> + + # Add files/changes + yadm add <important file> + yadm commit + + # Encrypt your ssh key + echo '.ssh/id_rsa' > ~/.config/yadm/encrypt + yadm encrypt + + # Later, decrypt your ssh key + yadm decrypt + + # Create different files for Linux vs MacOS + yadm add path/file.cfg##os.Linux + yadm add path/file.cfg##os.Darwin + +If you enjoy using yadm, consider adding a star to the repository on GitHub. +The star count helps others discover yadm. [Git]: https://git-scm.com/ [GnuPG]: https://gnupg.org/ -[aur-badge]: https://img.shields.io/aur/version/yadm-git.svg -[aur-link]: https://aur.archlinux.org/packages/yadm-git -[dev-pages-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/dev-pages.svg?label=dev-pages -[develop-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/develop.svg?label=develop +[OpenSSL]: https://www.openssl.org/ +[aur-badge]: https://img.shields.io/aur/version/yadm.svg +[aur-link]: https://aur.archlinux.org/packages/yadm +[dev-pages-badge]: https://img.shields.io/github/workflow/status/TheLocehiliosan/yadm/Test%20Site/dev-pages?label=dev-pages +[develop-badge]: https://img.shields.io/github/workflow/status/TheLocehiliosan/yadm/Tests/develop?label=develop [develop-commits]: https://github.com/TheLocehiliosan/yadm/commits/develop [develop-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/develop.svg?label=develop [dotfiles]: https://en.wikipedia.org/wiki/Hidden_file_and_hidden_directory -[gh-pages-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/gh-pages.svg?label=gh-pages +[gh-pages-badge]: https://img.shields.io/github/workflow/status/TheLocehiliosan/yadm/Test%20Site/gh-pages?label=gh-pages +[git-crypt]: https://github.com/AGWA/git-crypt [homebrew-badge]: https://img.shields.io/homebrew/v/yadm.svg [homebrew-link]: https://formulae.brew.sh/formula/yadm [license-badge]: https://img.shields.io/github/license/TheLocehiliosan/yadm.svg [license-link]: https://github.com/TheLocehiliosan/yadm/blob/master/LICENSE -[master-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/master.svg?label=master +[master-badge]: https://img.shields.io/github/workflow/status/TheLocehiliosan/yadm/Tests/master?label=master [master-commits]: https://github.com/TheLocehiliosan/yadm/commits/master [master-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/master.svg?label=master -[obs-badge]: https://img.shields.io/badge/OBS-v2.4.0-blue +[obs-badge]: https://img.shields.io/badge/OBS-v3.0.2-blue [obs-link]: https://software.opensuse.org//download.html?project=home%3ATheLocehiliosan%3Ayadm&package=yadm [releases-badge]: https://img.shields.io/github/tag/TheLocehiliosan/yadm.svg?label=latest+release [releases-link]: https://github.com/TheLocehiliosan/yadm/releases -[travis-ci]: https://travis-ci.org/TheLocehiliosan/yadm/branches +[transcrypt]: https://github.com/elasticdog/transcrypt +[travis-ci]: https://travis-ci.com/TheLocehiliosan/yadm/branches [website-commits]: https://github.com/TheLocehiliosan/yadm/commits/gh-pages [website-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/gh-pages.svg?label=website [website-link]: https://yadm.io/ +[workflow-dev-pages]: https://github.com/thelocehiliosan/yadm/actions?query=workflow%3a%22test+site%22+branch%3adev-pages +[workflow-develop]: https://github.com/TheLocehiliosan/yadm/actions?query=workflow%3ATests+branch%3Adevelop +[workflow-gh-pages]: https://github.com/thelocehiliosan/yadm/actions?query=workflow%3a%22test+site%22+branch%3agh-pages +[workflow-master]: https://github.com/TheLocehiliosan/yadm/actions?query=workflow%3ATests+branch%3Amaster diff --git a/completion/README.md b/completion/README.md index 69dae8b..1edd861 100644 --- a/completion/README.md +++ b/completion/README.md @@ -1,36 +1,47 @@ # Installation -## Bash completions ### Prerequisites -**yadm** completion only works if Git completions are also enabled. -### Homebrew -If using `homebrew` to install **yadm**, completions should automatically be handled if you also install `brew install bash-completion`. This might require you to include the main completion script in your own bashrc file like this: -``` +Bash and Zsh completion only works if Git completions are also enabled. + +## Homebrew + +If using `homebrew` to install yadm, Bash, Zsh, and Fish completions should +automatically be installed. For Bash and Zsh, you also must install +`bash-completion` or `zsh-completions`. This might require you to include the +main completion script in your own shell configuration like this: + +```bash [ -f /usr/local/etc/bash_completion ] && source /usr/local/etc/bash_completion ``` -### Manual installation +## Bash (manual installation) + Copy the completion script locally, and add this to you bashrc: -``` -[ -f /full/path/to/yadm.bash_completion ] && source /full/path/to/yadm.bash_completion + +```bash +[ -f /path/to/yadm/completion/bash/yadm ] && source /path/to/yadm/completion/bash/yadm ``` -## Zsh completions -### Homebrew -If using `homebrew` to install **yadm**, completions should handled automatically. +## Zsh (manual installation) -### Manual installation -Copy the completion script `yadm.zsh_completion` locally, rename it to `_yadm`, and add the containing folder to `$fpath` in `.zshrc`: -``` -fpath=(/path/to/folder/containing_yadm $fpath) +Add the `completion/zsh` folder to `$fpath` in `.zshrc`: + +```zsh +fpath=(/path/to/yadm/completion/zsh $fpath) autoload -U compinit compinit ``` -### Installation using [zplug](https://github.com/b4b4r07/zplug) +## Zsh (using [zplug](https://github.com/b4b4r07/zplug)) + Load `_yadm` as a plugin in your `.zshrc`: -``` + +```zsh fpath=("$ZPLUG_HOME/bin" $fpath) -zplug "TheLocehiliosan/yadm", rename-to:_yadm, use:"completion/yadm.zsh_completion", as:command, defer:2 +zplug "TheLocehiliosan/yadm", use:"completion/zsh/_yadm", as:command, defer:2 ``` + +## Fish (manual installation) + +Copy the completion script `yadm.fish` to any folder within `$fish_complete_path`. For example, for local installation, you can copy it to `$HOME/.config/fish/completions/` and it will be loaded when `yadm` is invoked. diff --git a/completion/yadm.bash_completion b/completion/bash/yadm index f8cfe87..f8cfe87 100644 --- a/completion/yadm.bash_completion +++ b/completion/bash/yadm diff --git a/completion/fish/yadm.fish b/completion/fish/yadm.fish new file mode 100644 index 0000000..ffb9067 --- /dev/null +++ b/completion/fish/yadm.fish @@ -0,0 +1,77 @@ +#!/usr/bin/fish + +function __fish_yadm_universial_optspecs + string join \n 'a-yadm-dir=' 'b-yadm-repo=' 'c-yadm-config=' \ + 'd-yadm-encrypt=' 'e-yadm-archive=' 'f-yadm-bootstrap=' +end + +function __fish_yadm_needs_command + # Figure out if the current invocation already has a command. + set -l cmd (commandline -opc) + set -e cmd[1] + argparse -s (__fish_yadm_universial_optspecs) -- $cmd 2>/dev/null + or return 0 + if set -q argv[1] + echo $argv[1] + return 1 + end + return 0 +end + +function __fish_yadm_using_command + set -l cmd (__fish_yadm_needs_command) + test -z "$cmd" + and return 1 + contains -- $cmd $argv + and return 0 +end + +# yadm's specific autocomplete +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'clone' -d 'Clone an existing repository' +complete -F -c yadm -n '__fish_yadm_using_command clone' -s w -d 'work-tree to use (default: $HOME)' +complete -f -c yadm -n '__fish_yadm_using_command clone' -s b -d 'branch to clone' +complete -x -c yadm -n '__fish_yadm_using_command clone' -s f -d 'force to overwrite' +complete -x -c yadm -n '__fish_yadm_using_command clone' -l bootstrap -d 'force bootstrap to run' +complete -x -c yadm -n '__fish_yadm_using_command clone' -l no-bootstrap -d 'prevent bootstrap from beingrun' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'alt' -d 'Create links for alternates' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'bootstrap' -d 'Execute $HOME/.config/yadm/bootstrap' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'perms' -d 'Fix perms for private files' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'enter' -d 'Run sub-shell with GIT variables set' +complete -c yadm -n '__fish_yadm_needs_command' -a 'git-crypt' -d 'Run git-crypt commands for the yadm repo' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'help' -d 'Print a summary of yadm commands' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'upgrade' -d 'Upgrade to version 2 of yadm directory structure' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'version' -d 'Print the version of yadm' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'init' -d 'Initialize an empty repository' +complete -x -c yadm -n '__fish_yadm_using_command init' -s f -d 'force to overwrite' +complete -F -c yadm -n '__fish_yadm_using_command init' -s w -d 'set work-tree (default: $HOME)' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'list' -d 'List tracked files at current directory' +complete -x -c yadm -n '__fish_yadm_using_command list' -s a -d 'list all managed files instead' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'encrypt' -d 'Encrypt files' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'decrypt' -d 'Decrypt files' +complete -x -c yadm -n '__fish_yadm_using_command decrypt' -s l -d 'list the files stored without extracting' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'introspect' -d 'Report internal yadm data' +complete -x -c yadm -n '__fish_yadm_using_command introspect' -a (printf -- '%s\n' 'commands configs repo switches') -d 'category' + +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'gitconfig' -d 'Pass options to the git config command' +complete -x -c yadm -n '__fish_yadm_needs_command' -a 'config' -d 'Configure a setting' +for name in (yadm introspect configs) + complete -x -c yadm -n '__fish_yadm_using_command config' -a '$name' -d 'yadm config' +end + +# yadm universial options +complete --force-files -c yadm -s Y -l yadm-dir -d 'Override location of yadm directory' +complete --force-files -c yadm -l yadm-repo -d 'Override location of yadm repository' +complete --force-files -c yadm -l yadm-config -d 'Override location of yadm configuration file' +complete --force-files -c yadm -l yadm-encrypt -d 'Override location of yadm encryption configuration' +complete --force-files -c yadm -l yadm-archive -d 'Override location of yadm encrypted files archive' +complete --force-files -c yadm -l yadm-bootstrap -d 'Override location of yadm bootstrap program' + +# wraps git's autocomplete +set -l GIT_DIR (yadm introspect repo) +# setup the correct git-dir by appending it to git's argunment +complete -c yadm -w "git --git-dir=$GIT_DIR" diff --git a/completion/yadm.zsh_completion b/completion/yadm.zsh_completion deleted file mode 100644 index fa79c01..0000000 --- a/completion/yadm.zsh_completion +++ /dev/null @@ -1,46 +0,0 @@ -#compdef yadm -_yadm(){ - local -a _1st_arguments - _1st_arguments=( - 'help:Display yadm command help' - 'init:Initialize an empty repository' - 'config:Configure a setting' - 'list:List tracked files' - 'alt:Create links for alternates' - 'bootstrap:Execute $HOME/.config/yadm/bootstrap' - 'encrypt:Encrypt files' - 'decrypt:Decrypt files' - 'perms:Fix perms for private files' - 'add:git add' - 'push:git push' - 'pull:git pull' - 'diff:git diff' - 'checkout:git checkout' - 'co:git co' - 'commit:git commit' - 'ci:git ci' - 'status:git status' - 'st:git st' - 'reset:git reset' - 'log:git log' - ) - - local context state line expl - local -A opt_args - - _arguments '*:: :->subcmds' && return 0 - - if (( CURRENT == 1 )); then - _describe -t commands "yadm commands" _1st_arguments -V1 - return - fi - - case "$words[1]" in - *) - _arguments ':filenames:_files' - ;; - esac - -} - -_yadm "$@" diff --git a/completion/zsh/_yadm b/completion/zsh/_yadm new file mode 100644 index 0000000..a1997a1 --- /dev/null +++ b/completion/zsh/_yadm @@ -0,0 +1,164 @@ +#compdef yadm + +# This completion tries to fallback to git's completion for git commands. + +zstyle -T ':completion:*:yadm:argument-1:descriptions:' format && \ + zstyle ':completion:*:yadm:argument-1:descriptions' format '%d:' +zstyle -T ':completion:*:yadm:*:yadm' group-name && \ + zstyle ':completion:*:yadm:*:yadm' group-name '' + +_yadm-alt() { + return 0 +} + +_yadm-bootstrap() { + return 0 +} + +_yadm-clone() { + _arguments \ + '(--bootstrap --no-bootstrap)--bootstrap[force bootstrap, without prompt]' \ + '(--bootstrap --no-bootstrap)--no-bootstrap[prevent bootstrap, without prompt]' \ + '-b[branch name]:' \ + '-f[force overwrite of existing repository]' \ + '-w[work tree path]: :_files -/' \ + '*:' +} + +_yadm-config() { + # TODO: complete config names +} + +_yadm-decrypt() { + _arguments \ + '-l[list files]' +} + +_yadm-encrypt() { + return 0 +} + +_yadm-enter() { + _arguments \ + ':command: _command_names -e' \ + '*::arguments: _normal' +} + +_yadm-git-crypt() { + # TODO: complete git-crypt options +} + +_yadm-help() { + return 0 +} + +_yadm-init() { + _arguments \ + '-f[force overwrite of existing repository]' \ + '-w[work tree path]: :_files -/' +} + +_yadm-list() { + _arguments \ + '-a[list all tracked files]' +} + +_yadm-perms() { + return 0 +} + +_yadm-transcrypt() { + integer _ret=1 + _call_function _ret _transcrypt + return _ret +} + +_yadm-upgrade() { + _arguments \ + '-f[force deinit of submodules]' \ + ': ' +} + +_yadm-version() { + return 0 +} + +_yadm_commands() { + local -a commands=( + alt:'create links for alternates' + bootstrap:'execute bootstrap' + clone:'clone an existing yadm repository' + config:'configure an yadm setting' + decrypt:'decrypt files' + encrypt:'encrypt files' + enter:'run sub-shell with GIT variables set' + git-crypt:'run git-crypt commands for the yadm repository' + gitconfig:'run the git config command' + help:'display yadm help information' + init:'initialize an empty yadm repository' + list:'list files tracked by yadm' + perms:'fix perms for private files' + transcrypt:'run transcrypt commands for the yadm repository' + upgrade:'upgrade legacy yadm paths' + version:'show yadm version' + ) + + local oldcontext="$curcontext" + local curcontext="${curcontext%:*:*}:git:" + + words=("git" "${words[-1]}") CURRENT=2 service=git _git + + curcontext="$oldcontext" + _describe -t yadm "yadm commands" commands + + return 0 +} + +_yadm() { + local curcontext=$curcontext state state_descr line + declare -A opt_args + + _arguments -C \ + '(-Y --yadm-dir)'{-Y,--yadm-dir}'[override the standard yadm directory]: :_files -/' \ + '--yadm-data[override the standard yadm data directory]: :_files -/' \ + '--yadm-repo[override the standard repo path]: :_files -/' \ + '--yadm-config[override the standard config path]: :_files -/' \ + '--yadm-encrypt[override the standard encrypt path]: :_files -/' \ + '--yadm-archive[override the standard archive path]: :_files -/' \ + '--yadm-bootstrap[override the standard bootstrap path]: :_files' \ + '--help[display yadm help information]' \ + '--version[show yadm version]' \ + '(-): :->command' \ + '(-)*:: :->option-or-argument' && return + + local -a repo_args + (( $+opt_args[--yadm-repo] )) && repo_args+=(--yadm-repo "$opt_args[--yadm-repo]") + (( $+opt_args[--yadm-data] )) && repo_args+=(--yadm-data "$opt_args[--yadm-data]") + local -x GIT_DIR="$(_call_program gitdir yadm "${repo_args[@]}" introspect repo)" + [[ -z "$GIT_DIR" ]] && return 1 + + integer _ret=1 + case $state in + (command) + _yadm_commands && _ret=0 + ;; + (option-or-argument) + curcontext=${curcontext%:*:*}:yadm-${words[1]}: + if ! _call_function _ret _yadm-${words[1]}; then + + # Translate gitconfig to use the regular completion for config + [[ ${words[1]} = "gitconfig" ]] && words[1]=config + + words=("git" "${(@)words}") + CURRENT=$(( CURRENT + 1 )) + + curcontext=${curcontext%:*:*}:git: + service=git _git && _ret=0 + fi + ;; + esac + + return _ret +} + +(( $+functions[_git] )) && _yadm diff --git a/contrib/bootstrap/bootstrap-in-dir b/contrib/bootstrap/bootstrap-in-dir new file mode 100755 index 0000000..91b75f7 --- /dev/null +++ b/contrib/bootstrap/bootstrap-in-dir @@ -0,0 +1,24 @@ +#!/bin/bash + +# Save this file as ~/.config/yadm/bootstrap and make it executable. It will +# execute all executable files (excluding templates and editor backups) in the +# ~/.config/yadm/bootstrap.d directory when run. + +set -eu + +# Directory to look for bootstrap executables in +BOOTSTRAP_D="${BASH_SOURCE[0]}.d" + +if [[ ! -d "$BOOTSTRAP_D" ]]; then + echo "Error: bootstrap directory '$BOOTSTRAP_D' not found" >&2 + exit 1 +fi + +find "$BOOTSTRAP_D" -type f | sort | while IFS= read -r bootstrap; do + if [[ -x "$bootstrap" && ! "$bootstrap" =~ "##" && ! "$bootstrap" =~ "~$" ]]; then + if ! "$bootstrap"; then + echo "Error: bootstrap '$bootstrap' failed" >&2 + exit 1 + fi + fi +done diff --git a/contrib/hooks/encrypt_with_checksums/post_encrypt b/contrib/hooks/encrypt_with_checksums/post_encrypt index 78935dd..5bb7cde 100755 --- a/contrib/hooks/encrypt_with_checksums/post_encrypt +++ b/contrib/hooks/encrypt_with_checksums/post_encrypt @@ -1,7 +1,7 @@ #!/usr/bin/env bash # yadm - Yet Another Dotfiles Manager -# Copyright (C) 2015-2020 Tim Byrne and Martin Zuther +# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/contrib/hooks/encrypt_with_checksums/post_list b/contrib/hooks/encrypt_with_checksums/post_list index 0a76751..e4c57df 100755 --- a/contrib/hooks/encrypt_with_checksums/post_list +++ b/contrib/hooks/encrypt_with_checksums/post_list @@ -1,7 +1,7 @@ #!/usr/bin/env bash # yadm - Yet Another Dotfiles Manager -# Copyright (C) 2015-2020 Tim Byrne and Martin Zuther +# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/contrib/hooks/encrypt_with_checksums/post_status b/contrib/hooks/encrypt_with_checksums/post_status index de2a650..d8778c1 100755 --- a/contrib/hooks/encrypt_with_checksums/post_status +++ b/contrib/hooks/encrypt_with_checksums/post_status @@ -1,7 +1,7 @@ #!/usr/bin/env bash # yadm - Yet Another Dotfiles Manager -# Copyright (C) 2015-2020 Tim Byrne and Martin Zuther +# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/debian/NEWS b/debian/NEWS index e9ec983..137de92 100644 --- a/debian/NEWS +++ b/debian/NEWS @@ -1,3 +1,27 @@ +yadm (3.0.2-1) unstable; urgency=medium + + yadm now supports openssl in addition to gpg for encryption. Along with this + change, the encrypted archive is now called archive instead of the old name + files.gpg. + + yadm now uses the XDG Base Directory Specification to find its data. For the + majority of users, this means data will now be in $HOME/.local/share/yadm/ + instead of the old location of $HOME/.config/yadm/. + + This location is used for the local repository and encrypted archive. + + The easiest way to adopt these new paths is to use the yadm upgrade command. + This command will move your existing repo and encrypted archive to the new + directory. Upgrading will also re-initialize all submodules you have added + (otherwise they will be broken when the repo moves). + + For more information about the necessary changes, please see the + following link: + + - https://yadm.io/docs/upgrade_from_2# + + -- Angel Abad <angel@debian.org> Sun, 31 Jan 2021 10:45:01 +0100 + yadm (2.3.0-1) unstable; urgency=medium Beginning with version 2.0.0, yadm introduced a few major changes which diff --git a/debian/changelog b/debian/changelog index de8cc38..4951cd9 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,28 @@ +yadm (3.0.2-1) unstable; urgency=medium + + * New upstream version 3.0.2. + * debian/watch: Upgrade to version 4. + * debian/NEWS: Update breaking changes info. + * debian/salsa-ci.yml: Add salsa ci basic config. + * debian/gbp.conf: Add gbp configuration file. + * Update shell completion install files. + * Declare compliance with Debian Policy 4.5.1. + + -- Angel Abad <angel@debian.org> Sun, 31 Jan 2021 10:48:31 +0100 + +yadm (2.5.0-2) unstable; urgency=medium + + * Source only upload + + -- Angel Abad <angel@debian.org> Thu, 20 Aug 2020 09:17:04 +0200 + +yadm (2.5.0-1) unstable; urgency=medium + + * New upstream version 2.5.0 + * debian/rules: Use execute_after_dh_auto_install + + -- Angel Abad <angel@debian.org> Wed, 19 Aug 2020 16:42:24 +0200 + yadm (2.4.0-2) unstable; urgency=medium * Update to debhelper compatibility level V13. diff --git a/debian/control b/debian/control index 71cc6ab..f9349ab 100644 --- a/debian/control +++ b/debian/control @@ -9,7 +9,7 @@ Uploaders: Build-Depends: debhelper-compat (= 13), Rules-Requires-Root: no -Standards-Version: 4.5.0 +Standards-Version: 4.5.1 Homepage: https://github.com/TheLocehiliosan/yadm Vcs-Browser: https://salsa.debian.org/debian/yadm Vcs-Git: https://salsa.debian.org/debian/yadm.git diff --git a/debian/gbp.conf b/debian/gbp.conf new file mode 100644 index 0000000..aa8a019 --- /dev/null +++ b/debian/gbp.conf @@ -0,0 +1,4 @@ +[DEFAULT] +debian-branch = debian/master +[import-orig] +pristine-tar = True diff --git a/debian/rules b/debian/rules index 7fa32cf..9343dbd 100755 --- a/debian/rules +++ b/debian/rules @@ -10,8 +10,7 @@ override_dh_auto_build: # Remove some files installed by upstream that are installed by the # packaging logic in different ways. -override_dh_auto_install: - dh_auto_install +execute_after_dh_auto_install: rm debian/yadm/usr/share/doc/yadm/CHANGES rm debian/yadm/usr/share/doc/yadm/LICENSE rm -r debian/yadm/usr/share/doc/yadm/contrib diff --git a/debian/salsa-ci.yml b/debian/salsa-ci.yml new file mode 100644 index 0000000..8424db4 --- /dev/null +++ b/debian/salsa-ci.yml @@ -0,0 +1,3 @@ +--- +include: + - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml diff --git a/debian/watch b/debian/watch index 86ff2a4..1a53157 100644 --- a/debian/watch +++ b/debian/watch @@ -1,6 +1,3 @@ -# You must remove unused comment lines for the released package. -version=3 - -opts=filenamemangle=s/.+\/v?(\d\S*)\.tar\.gz/$1\.tar\.gz/ \ - https://github.com/TheLocehiliosan/yadm/tags .*/v?(\d\S*)\.tar\.gz - +version=4 +opts=filenamemangle=s/.+\/v?(\d\S+)\.tar\.gz/yadm-$1\.tar\.gz/ \ + https://github.com/TheLocehiliosan/yadm/tags .*/v?(\d\S+)\.tar\.gz diff --git a/debian/yadm.install b/debian/yadm.install index b23fc10..25882ff 100644 --- a/debian/yadm.install +++ b/debian/yadm.install @@ -1,2 +1,2 @@ -completion/yadm.bash_completion usr/share/yadm/completion -completion/yadm.zsh_completion usr/share/yadm/completion +completion/bash/yadm usr/share/yadm/completion/bash +completion/zsh/_yadm usr/share/yadm/completion/zsh diff --git a/debian/yadm.links b/debian/yadm.links index 879201c..a001ad0 100644 --- a/debian/yadm.links +++ b/debian/yadm.links @@ -1,2 +1,2 @@ -usr/share/yadm/completion/yadm.bash_completion usr/share/bash-completion/completions/yadm.bash -usr/share/yadm/completion/yadm.zsh_completion usr/share/zsh/vendor-completions/_yadm +usr/share/yadm/completion/bash/yadm usr/share/bash-completion/completions/yadm.bash +usr/share/yadm/completion/zsh/_yadm usr/share/zsh/vendor-completions/_yadm diff --git a/docker-compose.yml b/docker-compose.yml deleted file mode 100644 index 831a062..0000000 --- a/docker-compose.yml +++ /dev/null @@ -1,7 +0,0 @@ ---- -version: '3' -services: - testbed: - volumes: - - .:/yadm:ro - image: yadm/testbed:2020-01-20 diff --git a/test/Dockerfile b/test/Dockerfile new file mode 100644 index 0000000..3e5a299 --- /dev/null +++ b/test/Dockerfile @@ -0,0 +1,73 @@ +FROM ubuntu:18.04 +MAINTAINER Tim Byrne <sultan@locehilios.com> + +# Shellcheck and esh versions +ARG SC_VER=0.7.1 +ARG ESH_VER=0.3.0 + +# Install prerequisites and configure UTF-8 locale +RUN \ + echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \ + && apt-get update \ + && DEBIAN_FRONTEND=noninteractive \ + apt-get install -y --no-install-recommends \ + expect \ + git \ + gnupg \ + locales \ + lsb-release \ + make \ + man \ + python3-pip \ + vim-tiny \ + xz-utils \ + && rm -rf /var/lib/apt/lists/* \ + && update-locale LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8' + +ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8' + +# Convenience settings for the testbed's root account +RUN echo 'set -o vi' >> /root/.bashrc + +# Create a flag to identify when running inside the yadm testbed +RUN touch /.yadmtestbed + +# Install shellcheck +ADD https://github.com/koalaman/shellcheck/releases/download/v$SC_VER/shellcheck-v$SC_VER.linux.x86_64.tar.xz /opt +RUN cd /opt \ + && tar xf shellcheck-v$SC_VER.linux.x86_64.tar.xz \ + && rm -f shellcheck-v$SC_VER.linux.x86_64.tar.xz \ + && ln -s /opt/shellcheck-v$SC_VER/shellcheck /usr/local/bin + +# Upgrade pip3 and install requirements +COPY test/requirements.txt /tmp/requirements.txt +RUN python3 -m pip install --upgrade pip setuptools \ + && python3 -m pip install --upgrade -r /tmp/requirements.txt \ + && rm -f /tmp/requirements + +# Install esh +ADD https://raw.githubusercontent.com/jirutka/esh/v$ESH_VER/esh /usr/local/bin +RUN chmod +x /usr/local/bin/esh + +# Create workdir and dummy Makefile to be used if no /yadm volume is mounted +RUN mkdir /yadm \ + && echo "test:" > /yadm/Makefile \ + && echo "\t@echo 'The yadm project must be mounted at /yadm'" >> /yadm/Makefile \ + && echo "\t@echo 'Try using a docker parameter like -v \"\$\$PWD:/yadm:ro\"'" >> /yadm/Makefile \ + && echo "\t@false" >> /yadm/Makefile + +# Include released versions of yadm to test upgrades +ADD https://raw.githubusercontent.com/TheLocehiliosan/yadm/1.12.0/yadm /usr/local/bin/yadm-1.12.0 +ADD https://raw.githubusercontent.com/TheLocehiliosan/yadm/2.5.0/yadm /usr/local/bin/yadm-2.5.0 +RUN chmod +x /usr/local/bin/yadm-* + +# Configure git to make it easier to test yadm manually +RUN git config --system user.email "test@yadm.io" \ + && git config --system user.name "Yadm Test" + +# /yadm will be the work directory for all tests +# docker commands should mount the local yadm project as /yadm +WORKDIR /yadm + +# By default, run all tests defined +CMD make test diff --git a/test/conftest.py b/test/conftest.py index 31d872b..38228a1 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -25,25 +25,25 @@ def pytest_addoption(parser): @pytest.fixture(scope='session') def shellcheck_version(): """Version of shellcheck supported""" - return '0.4.6' + return '0.7.1' @pytest.fixture(scope='session') def pylint_version(): """Version of pylint supported""" - return '2.4.1' + return '2.6.0' @pytest.fixture(scope='session') def flake8_version(): """Version of flake8 supported""" - return '3.7.8' + return '3.8.4' @pytest.fixture(scope='session') def yamllint_version(): """Version of yamllint supported""" - return '1.17.0' + return '1.25.0' @pytest.fixture(scope='session') @@ -96,6 +96,7 @@ def supported_commands(): 'introspect', 'list', 'perms', + 'transcrypt', 'upgrade', 'version', ] @@ -117,10 +118,14 @@ def supported_configs(): 'yadm.auto-exclude', 'yadm.auto-perms', 'yadm.auto-private-dirs', + 'yadm.cipher', 'yadm.git-program', 'yadm.gpg-perms', 'yadm.gpg-program', 'yadm.gpg-recipient', + 'yadm.openssl-ciphername', + 'yadm.openssl-old', + 'yadm.openssl-program', 'yadm.ssh-perms', ] @@ -135,6 +140,7 @@ def supported_switches(): '--yadm-archive', '--yadm-bootstrap', '--yadm-config', + '--yadm-data', '--yadm-dir', '--yadm-encrypt', '--yadm-repo', @@ -174,6 +180,10 @@ class Runner(): self.command = ' '.join([str(cmd) for cmd in command]) else: self.command = command + if env is None: + env = {} + merged_env = os.environ.copy() + merged_env.update(env) self.inp = inp self.wrap(expect) process = Popen( @@ -183,7 +193,7 @@ class Runner(): stderr=PIPE, shell=shell, cwd=cwd, - env=env, + env=merged_env, ) input_bytes = self.inp if self.inp: @@ -274,13 +284,17 @@ def yadm(): @pytest.fixture() def paths(tmpdir, yadm): """Function scoped test paths""" + dir_root = tmpdir.mkdir('root') + dir_remote = dir_root.mkdir('remote') dir_work = dir_root.mkdir('work') - dir_yadm = dir_root.mkdir('yadm') - dir_repo = dir_yadm.mkdir('repo.git') + dir_xdg_data = dir_root.mkdir('xdg_data') + dir_xdg_home = dir_root.mkdir('xdg_home') + dir_data = dir_xdg_data.mkdir('yadm') + dir_yadm = dir_xdg_home.mkdir('yadm') dir_hooks = dir_yadm.mkdir('hooks') - dir_remote = dir_root.mkdir('remote') - file_archive = dir_yadm.join('files.gpg') + dir_repo = dir_data.mkdir('repo.git') + file_archive = dir_data.join('archive') file_bootstrap = dir_yadm.join('bootstrap') file_config = dir_yadm.join('config') file_encrypt = dir_yadm.join('encrypt') @@ -288,24 +302,32 @@ def paths(tmpdir, yadm): 'Paths', [ 'pgm', 'root', + 'remote', 'work', + 'xdg_data', + 'xdg_home', + 'data', 'yadm', - 'repo', 'hooks', - 'remote', + 'repo', 'archive', 'bootstrap', 'config', 'encrypt', ]) + os.environ['XDG_CONFIG_HOME'] = str(dir_xdg_home) + os.environ['XDG_DATA_HOME'] = str(dir_xdg_data) return paths( yadm, dir_root, + dir_remote, dir_work, + dir_xdg_data, + dir_xdg_home, + dir_data, dir_yadm, - dir_repo, dir_hooks, - dir_remote, + dir_repo, file_archive, file_bootstrap, file_config, @@ -314,11 +336,11 @@ def paths(tmpdir, yadm): @pytest.fixture() -def yadm_y(paths): +def yadm_cmd(paths): """Generate custom command_list function""" def command_list(*args): """Produce params for running yadm with -Y""" - return [paths.pgm, '-Y', str(paths.yadm)] + list(args) + return [paths.pgm] + list(args) return command_list diff --git a/test/requirements.txt b/test/requirements.txt new file mode 100644 index 0000000..30da6ae --- /dev/null +++ b/test/requirements.txt @@ -0,0 +1,6 @@ +envtpl +flake8==3.8.4 +j2cli +pylint==2.6.0 +pytest==6.2.1 +yamllint==1.25.0 diff --git a/test/test_alt.py b/test/test_alt.py index 359f32d..b18e6cb 100644 --- a/test/test_alt.py +++ b/test/test_alt.py @@ -22,12 +22,12 @@ def test_alt_source( tracked, encrypt, exclude, yadm_alt): """Test yadm alt operates on all expected sources of alternates""" - yadm_dir = setup_standard_yadm_dir(paths) + yadm_dir, yadm_data = setup_standard_yadm_dir(paths) utils.create_alt_files( paths, '##default', tracked=tracked, encrypt=encrypt, exclude=exclude, yadm_alt=yadm_alt, yadm_dir=yadm_dir) - run = runner([paths.pgm, '-Y', yadm_dir, 'alt']) + run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt']) assert run.success assert run.err == '' linked = utils.parse_alt_output(run.out) @@ -57,12 +57,12 @@ def test_alt_source( @pytest.mark.parametrize('yadm_alt', [True, False], ids=['alt', 'worktree']) def test_relative_link(runner, paths, yadm_alt): """Confirm links created are relative""" - yadm_dir = setup_standard_yadm_dir(paths) + yadm_dir, yadm_data = setup_standard_yadm_dir(paths) utils.create_alt_files( paths, '##default', tracked=True, encrypt=False, exclude=False, yadm_alt=yadm_alt, yadm_dir=yadm_dir) - run = runner([paths.pgm, '-Y', yadm_dir, 'alt']) + run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt']) assert run.success assert run.err == '' @@ -81,6 +81,7 @@ def test_relative_link(runner, paths, yadm_alt): @pytest.mark.usefixtures('ds1_copy') @pytest.mark.parametrize('suffix', [ '##default', + '##default,e.txt', '##default,extension.txt', '##o.$tst_sys', '##os.$tst_sys', '##d.$tst_distro', '##distro.$tst_distro', '##c.$tst_class', '##class.$tst_class', @@ -91,7 +92,7 @@ def test_alt_conditions( runner, paths, tst_sys, tst_distro, tst_host, tst_user, suffix): """Test conditions supported by yadm alt""" - yadm_dir = setup_standard_yadm_dir(paths) + yadm_dir, yadm_data = setup_standard_yadm_dir(paths) # set the class tst_class = 'testclass' @@ -106,7 +107,7 @@ def test_alt_conditions( ) utils.create_alt_files(paths, suffix) - run = runner([paths.pgm, '-Y', yadm_dir, 'alt']) + run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt']) assert run.success assert run.err == '' linked = utils.parse_alt_output(run.out) @@ -126,18 +127,18 @@ def test_alt_conditions( @pytest.mark.usefixtures('ds1_copy') @pytest.mark.parametrize( - 'kind', ['default', '', None, 'envtpl', 'j2cli', 'j2']) + 'kind', ['default', '', None, 'envtpl', 'j2cli', 'j2', 'esh']) @pytest.mark.parametrize('label', ['t', 'template', 'yadm', ]) def test_alt_templates( runner, paths, kind, label): """Test templates supported by yadm alt""" - yadm_dir = setup_standard_yadm_dir(paths) + yadm_dir, yadm_data = setup_standard_yadm_dir(paths) suffix = f'##{label}.{kind}' if kind is None: suffix = f'##{label}' utils.create_alt_files(paths, suffix) - run = runner([paths.pgm, '-Y', yadm_dir, 'alt']) + run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt']) assert run.success assert run.err == '' created = utils.parse_alt_output(run.out, linked=False) @@ -152,15 +153,15 @@ def test_alt_templates( @pytest.mark.usefixtures('ds1_copy') @pytest.mark.parametrize('autoalt', [None, 'true', 'false']) -def test_auto_alt(runner, yadm_y, paths, autoalt): +def test_auto_alt(runner, yadm_cmd, paths, autoalt): """Test auto alt""" # set the value of auto-alt if autoalt: - os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt))) + os.system(' '.join(yadm_cmd('config', 'yadm.auto-alt', autoalt))) utils.create_alt_files(paths, '##default') - run = runner(yadm_y('status')) + run = runner(yadm_cmd('status')) assert run.success assert run.err == '' linked = utils.parse_alt_output(run.out) @@ -185,7 +186,7 @@ def test_auto_alt(runner, yadm_y, paths, autoalt): @pytest.mark.usefixtures('ds1_copy') -def test_stale_link_removal(runner, yadm_y, paths): +def test_stale_link_removal(runner, yadm_cmd, paths): """Stale links to alternative files are removed This test ensures that when an already linked alternative becomes invalid @@ -200,7 +201,7 @@ def test_stale_link_removal(runner, yadm_y, paths): utils.create_alt_files(paths, f'##class.{tst_class}') # run alt to trigger linking - run = runner(yadm_y('alt')) + run = runner(yadm_cmd('alt')) assert run.success assert run.err == '' linked = utils.parse_alt_output(run.out) @@ -222,7 +223,7 @@ def test_stale_link_removal(runner, yadm_y, paths): utils.set_local(paths, 'class', 'changedclass') # run alt to trigger linking - run = runner(yadm_y('alt')) + run = runner(yadm_cmd('alt')) assert run.success assert run.err == '' linked = utils.parse_alt_output(run.out) @@ -235,7 +236,7 @@ def test_stale_link_removal(runner, yadm_y, paths): @pytest.mark.usefixtures('ds1_repo_copy') -def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys): +def test_template_overwrite_symlink(runner, yadm_cmd, paths, tst_sys): """Remove symlinks before processing a template If a symlink is in the way of the output of a template, the target of the @@ -252,7 +253,7 @@ def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys): template = paths.work.join('test_link##template.default') template.write('test-data') - run = runner(yadm_y('add', target, template)) + run = runner(yadm_cmd('add', target, template)) assert run.success assert run.err == '' assert run.out == '' @@ -265,12 +266,13 @@ def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys): @pytest.mark.parametrize('style', ['symlink', 'template']) def test_ensure_alt_path(runner, paths, style): """Test that directories are created before making alternates""" - yadm_dir = setup_standard_yadm_dir(paths) + yadm_dir, yadm_data = setup_standard_yadm_dir(paths) suffix = 'default' if style == 'symlink' else 'template' filename = 'a/b/c/file' source = yadm_dir.join(f'alt/{filename}##{suffix}') source.write('test-data', ensure=True) - run = runner([paths.pgm, '-Y', yadm_dir, 'add', source]) + run = runner([ + paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'add', source]) assert run.success assert run.err == '' assert run.out == '' @@ -280,6 +282,7 @@ def test_ensure_alt_path(runner, paths, style): def setup_standard_yadm_dir(paths): """Configure a yadm home within the work tree""" std_yadm_dir = paths.work.mkdir('.config').mkdir('yadm') - std_yadm_dir.join('repo.git').mksymlinkto(paths.repo, absolute=1) + std_yadm_data = paths.work.mkdir('.local').mkdir('share').mkdir('yadm') + std_yadm_data.join('repo.git').mksymlinkto(paths.repo, absolute=1) std_yadm_dir.join('encrypt').mksymlinkto(paths.encrypt, absolute=1) - return std_yadm_dir + return std_yadm_dir, std_yadm_data diff --git a/test/test_alt_copy.py b/test/test_alt_copy.py index c808348..fa8e09c 100644 --- a/test/test_alt_copy.py +++ b/test/test_alt_copy.py @@ -5,10 +5,6 @@ import pytest @pytest.mark.parametrize( - 'cygwin', - [pytest.param(True, marks=pytest.mark.deprecated), False], - ids=['cygwin', 'no-cygwin']) -@pytest.mark.parametrize( 'setting, expect_link, pre_existing', [ (None, True, None), (True, False, None), @@ -25,15 +21,12 @@ import pytest ]) @pytest.mark.usefixtures('ds1_copy') def test_alt_copy( - runner, yadm_y, paths, tst_sys, - setting, expect_link, pre_existing, - cygwin): + runner, yadm_cmd, paths, tst_sys, + setting, expect_link, pre_existing): """Test yadm.alt-copy""" - option = 'yadm.cygwin-copy' if cygwin else 'yadm.alt-copy' - if setting is not None: - os.system(' '.join(yadm_y('config', option, str(setting)))) + os.system(' '.join(yadm_cmd('config', 'yadm.alt-copy', str(setting)))) expected_content = f'test_alt_copy##os.{tst_sys}' @@ -43,7 +36,7 @@ def test_alt_copy( elif pre_existing == 'file': alt_path.write('wrong content') - run = runner(yadm_y('alt')) + run = runner(yadm_cmd('alt')) assert run.success assert run.err == '' assert 'Linking' in run.out diff --git a/test/test_assert_private_dirs.py b/test/test_assert_private_dirs.py index 606012f..bfd55ac 100644 --- a/test/test_assert_private_dirs.py +++ b/test/test_assert_private_dirs.py @@ -9,7 +9,7 @@ PRIVATE_DIRS = ['.gnupg', '.ssh'] @pytest.mark.parametrize('home', [True, False], ids=['home', 'not-home']) -def test_pdirs_missing(runner, yadm_y, paths, home): +def test_pdirs_missing(runner, yadm_cmd, paths, home): """Private dirs (private dirs missing) When a git command is run @@ -29,7 +29,7 @@ def test_pdirs_missing(runner, yadm_y, paths, home): env['HOME'] = paths.work # run status - run = runner(command=yadm_y('status'), env=env) + run = runner(command=yadm_cmd('status'), env=env) assert run.success assert run.err == '' assert 'On branch master' in run.out @@ -53,7 +53,7 @@ def test_pdirs_missing(runner, yadm_y, paths, home): run.out, re.DOTALL), 'directories created before command is run' -def test_pdirs_missing_apd_false(runner, yadm_y, paths): +def test_pdirs_missing_apd_false(runner, yadm_cmd, paths): """Private dirs (private dirs missing / yadm.auto-private-dirs=false) When a git command is run @@ -70,11 +70,11 @@ def test_pdirs_missing_apd_false(runner, yadm_y, paths): assert not path.exists() # set configuration - os.system(' '.join(yadm_y( + os.system(' '.join(yadm_cmd( 'config', '--bool', 'yadm.auto-private-dirs', 'false'))) # run status - run = runner(command=yadm_y('status')) + run = runner(command=yadm_cmd('status')) assert run.success assert run.err == '' assert 'On branch master' in run.out @@ -84,7 +84,7 @@ def test_pdirs_missing_apd_false(runner, yadm_y, paths): assert not paths.work.join(pdir).exists() -def test_pdirs_exist_apd_false(runner, yadm_y, paths): +def test_pdirs_exist_apd_false(runner, yadm_cmd, paths): """Private dirs (private dirs exist / yadm.auto-perms=false) When a git command is run @@ -102,11 +102,11 @@ def test_pdirs_exist_apd_false(runner, yadm_y, paths): assert oct(path.stat().mode).endswith('77'), 'Directory is secure.' # set configuration - os.system(' '.join(yadm_y( + os.system(' '.join(yadm_cmd( 'config', '--bool', 'yadm.auto-perms', 'false'))) # run status - run = runner(command=yadm_y('status')) + run = runner(command=yadm_cmd('status')) assert run.success assert run.err == '' assert 'On branch master' in run.out diff --git a/test/test_bootstrap.py b/test/test_bootstrap.py index 2adbe33..4865ece 100644 --- a/test/test_bootstrap.py +++ b/test/test_bootstrap.py @@ -14,7 +14,7 @@ import pytest 'executable', ]) def test_bootstrap( - runner, yadm_y, paths, exists, executable, code, expect): + runner, yadm_cmd, paths, exists, executable, code, expect): """Test bootstrap command""" if exists: paths.bootstrap.write('') @@ -25,7 +25,11 @@ def test_bootstrap( f'exit {code}\n' ) paths.bootstrap.chmod(0o775) - run = runner(command=yadm_y('bootstrap')) + run = runner(command=yadm_cmd('bootstrap')) assert run.code == code - assert run.err == '' - assert expect in run.out + if exists and executable: + assert run.err == '' + assert expect in run.out + else: + assert expect in run.err + assert run.out == '' diff --git a/test/test_clean.py b/test/test_clean.py index 9a2221a..39e7e54 100644 --- a/test/test_clean.py +++ b/test/test_clean.py @@ -1,11 +1,11 @@ """Test clean""" -def test_clean_command(runner, yadm_y): +def test_clean_command(runner, yadm_cmd): """Run with clean command""" - run = runner(command=yadm_y('clean')) + run = runner(command=yadm_cmd('clean')) # do nothing, this is a dangerous Git command when managing dot files # report the command as disabled and exit as a failure assert run.failure - assert run.err == '' - assert 'disabled' in run.out + assert run.out == '' + assert 'disabled' in run.err diff --git a/test/test_clone.py b/test/test_clone.py index a6df6d0..b024e9c 100644 --- a/test/test_clone.py +++ b/test/test_clone.py @@ -24,7 +24,7 @@ BOOTSTRAP_MSG = 'Bootstrap successful' 'conflicts', ]) def test_clone( - runner, paths, yadm_y, repo_config, ds1, + runner, paths, yadm_cmd, repo_config, ds1, good_remote, repo_exists, force, conflicts): """Test basic clone operation""" @@ -53,23 +53,26 @@ def test_clone( if force: args += ['-f'] args += [remote_url] - run = runner(command=yadm_y(*args)) + run = runner(command=yadm_cmd(*args)) if not good_remote: # clone should fail assert run.failure - assert run.err != '' - assert 'Unable to fetch origin' in run.out + assert run.out != '' + assert 'Unable to fetch origin' in run.err assert not paths.repo.exists() elif repo_exists and not force: # can't overwrite data assert run.failure - assert run.err == '' - assert 'Git repo already exists' in run.out + assert run.out == '' + assert 'Git repo already exists' in run.err else: # clone should succeed, and repo should be configured properly assert successful_clone(run, paths, repo_config) + # these clones should have master as HEAD + verify_head(paths, 'master') + # ensure conflicts are handled properly if conflicts: assert 'NOTE' in run.out @@ -88,20 +91,21 @@ def test_clone( if conflicts: # test to see if the work tree is actually "clean" run = runner( - command=yadm_y('status', '-uno', '--porcelain'), + command=yadm_cmd('status', '-uno', '--porcelain'), cwd=paths.work) assert run.success assert run.err == '' assert run.out == '', 'worktree has unexpected changes' # test to see if the conflicts are stashed - run = runner(command=yadm_y('stash', 'list'), cwd=paths.work) + run = runner(command=yadm_cmd('stash', 'list'), cwd=paths.work) assert run.success assert run.err == '' assert 'Conflicts preserved' in run.out, 'conflicts not stashed' # verify content of the stashed conflicts - run = runner(command=yadm_y('stash', 'show', '-p'), cwd=paths.work) + run = runner( + command=yadm_cmd('stash', 'show', '-p'), cwd=paths.work) assert run.success assert run.err == '' assert '\n+conflict' in run.out, 'conflicts not stashed' @@ -130,7 +134,7 @@ def test_clone( 'existing, answer y', ]) def test_clone_bootstrap( - runner, paths, yadm_y, repo_config, bs_exists, bs_param, answer): + runner, paths, yadm_cmd, repo_config, bs_exists, bs_param, answer): """Test bootstrap clone features""" # establish a bootstrap @@ -144,7 +148,7 @@ def test_clone_bootstrap( expect = [] if answer: expect.append(('Would you like to execute it now', answer)) - run = runner(command=yadm_y(*args), expect=expect) + run = runner(command=yadm_cmd(*args), expect=expect) if answer: assert 'Would you like to execute it now' in run.out @@ -161,6 +165,7 @@ def test_clone_bootstrap( assert BOOTSTRAP_MSG not in run.out assert successful_clone(run, paths, repo_config, expected_code) + verify_head(paths, 'master') if not bs_exists: assert BOOTSTRAP_MSG not in run.out @@ -197,7 +202,7 @@ def create_bootstrap(paths, exists): 'missing gnupg, tracked', ]) def test_clone_perms( - runner, yadm_y, paths, repo_config, + runner, yadm_cmd, paths, repo_config, private_type, in_repo, in_work): """Test clone permission-related functions""" @@ -224,11 +229,12 @@ def test_clone_perms( env = {'HOME': paths.work} run = runner( - yadm_y('clone', '-d', '-w', paths.work, f'file://{paths.remote}'), + yadm_cmd('clone', '-d', '-w', paths.work, f'file://{paths.remote}'), env=env ) assert successful_clone(run, paths, repo_config) + verify_head(paths, 'master') if in_work: # private directories which already exist, should be left as they are, # which in this test is "insecure". @@ -259,8 +265,9 @@ def test_clone_perms( @pytest.mark.usefixtures('remote') -@pytest.mark.parametrize('branch', ['master', 'valid', 'invalid']) -def test_alternate_branch(runner, paths, yadm_y, repo_config, branch): +@pytest.mark.parametrize( + 'branch', ['master', 'default', 'valid', 'invalid']) +def test_alternate_branch(runner, paths, yadm_cmd, repo_config, branch): """Test cloning a branch other than master""" # add a "valid" branch to the remote @@ -268,6 +275,12 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch): os.system( f'GIT_DIR="{paths.remote}" git commit ' f'--allow-empty -m "This branch is valid"') + if branch != 'default': + # When branch == 'default', the "default" branch of the remote repo + # will remain "valid" to validate identification the correct default + # branch by inspecting the repo. Otherwise it will be set back to + # "master" + os.system(f'GIT_DIR="{paths.remote}" git checkout master') # clear out the work path paths.work.remove() @@ -277,15 +290,15 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch): # run the clone command args = ['clone', '-w', paths.work] - if branch != 'master': + if branch not in ['master', 'default']: args += ['-b', branch] args += [remote_url] - run = runner(command=yadm_y(*args)) + run = runner(command=yadm_cmd(*args)) if branch == 'invalid': assert run.failure - assert 'ERROR: Clone failed' in run.out - assert f"'origin/{branch}' does not exist in {remote_url}" in run.out + assert 'ERROR: Clone failed' in run.err + assert f"'origin/{branch}' does not exist in {remote_url}" in run.err else: assert successful_clone(run, paths, repo_config) @@ -296,11 +309,13 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch): assert run.success assert run.err == '' assert f'origin\t{remote_url}' in run.out - run = runner(command=yadm_y('show')) - if branch == 'valid': - assert 'This branch is valid' in run.out - else: + run = runner(command=yadm_cmd('show')) + if branch == 'master': assert 'Initial commit' in run.out + verify_head(paths, 'master') + else: + assert 'This branch is valid' in run.out + verify_head(paths, 'valid') def successful_clone(run, paths, repo_config, expected_code=0): @@ -323,3 +338,16 @@ def remote(paths, ds1_repo_copy): # cannot be applied to another fixture. paths.remote.remove() paths.repo.move(paths.remote) + + +def test_no_repo(runner, yadm_cmd, ): + """Test cloning without specifying a repo""" + run = runner(command=yadm_cmd('clone')) + assert run.failure + assert run.out == '' + assert 'ERROR: No repository provided' in run.err + + +def verify_head(paths, branch): + """Assert the local repo has the correct head branch""" + assert paths.repo.join('HEAD').read() == f'ref: refs/heads/{branch}\n' diff --git a/test/test_compat_alt.py b/test/test_compat_alt.py deleted file mode 100644 index da7a8cf..0000000 --- a/test/test_compat_alt.py +++ /dev/null @@ -1,453 +0,0 @@ -"""Test alt""" - -import os -import string -import py -import pytest -import utils - -# These tests are for the alternate processing in YADM_COMPATIBILITY=1 mode -pytestmark = pytest.mark.deprecated - -# These test IDs are broken. During the writing of these tests, problems have -# been discovered in the way yadm orders matching files. -BROKEN_TEST_IDS = [ - 'test_wild[tracked-##C.S.H.U-C-S%-H%-U]', - 'test_wild[tracked-##C.S.H.U-C-S-H%-U]', - 'test_wild[encrypted-##C.S.H.U-C-S%-H%-U]', - 'test_wild[encrypted-##C.S.H.U-C-S-H%-U]', - ] - -PRECEDENCE = [ - '##', - '##$tst_sys', - '##$tst_sys.$tst_host', - '##$tst_sys.$tst_host.$tst_user', - '##$tst_class', - '##$tst_class.$tst_sys', - '##$tst_class.$tst_sys.$tst_host', - '##$tst_class.$tst_sys.$tst_host.$tst_user', - ] - -WILD_TEMPLATES = [ - '##$tst_class', - '##$tst_class.$tst_sys', - '##$tst_class.$tst_sys.$tst_host', - '##$tst_class.$tst_sys.$tst_host.$tst_user', - ] - -TEST_PATHS = [utils.ALT_FILE1, utils.ALT_FILE2, utils.ALT_DIR] - -WILD_TESTED = set() - - -@pytest.mark.parametrize('precedence_index', range(len(PRECEDENCE))) -@pytest.mark.parametrize( - 'tracked, encrypt, exclude', [ - (False, False, False), - (True, False, False), - (False, True, False), - (False, True, True), - ], ids=[ - 'untracked', - 'tracked', - 'encrypted', - 'excluded', - ]) -@pytest.mark.usefixtures('ds1_copy') -def test_alt(runner, yadm_y, paths, - tst_sys, tst_host, tst_user, - tracked, encrypt, exclude, - precedence_index): - """Test alternate linking - - This test is done by iterating for the number of templates in PRECEDENCE. - With each iteration, another file is left off the list. So with each - iteration, the template with the "highest precedence" is left out. The file - using the highest precedence should be the one linked. - """ - - # set the class - tst_class = 'testclass' - utils.set_local(paths, 'class', tst_class) - - # process the templates in PRECEDENCE - precedence = list() - for template in PRECEDENCE: - precedence.append( - string.Template(template).substitute( - tst_class=tst_class, - tst_host=tst_host, - tst_sys=tst_sys, - tst_user=tst_user, - ) - ) - - # create files using a subset of files - for suffix in precedence[0:precedence_index+1]: - utils.create_alt_files(paths, suffix, tracked=tracked, - encrypt=encrypt, exclude=exclude) - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + precedence[precedence_index] - if tracked or (encrypt and not exclude): - assert paths.work.join(file_path).islink() - target = py.path.local( - os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert not paths.work.join(file_path).exists() - assert str(paths.work.join(source_file)) not in linked - - -def short_template(template): - """Translate template into something short for test IDs""" - return string.Template(template).substitute( - tst_class='C', - tst_host='H', - tst_sys='S', - tst_user='U', - ) - - -@pytest.mark.parametrize('wild_user', [True, False], ids=['U%', 'U']) -@pytest.mark.parametrize('wild_host', [True, False], ids=['H%', 'H']) -@pytest.mark.parametrize('wild_sys', [True, False], ids=['S%', 'S']) -@pytest.mark.parametrize('wild_class', [True, False], ids=['C%', 'C']) -@pytest.mark.parametrize('template', WILD_TEMPLATES, ids=short_template) -@pytest.mark.parametrize( - 'tracked, encrypt', [ - (True, False), - (False, True), - ], ids=[ - 'tracked', - 'encrypted', - ]) -@pytest.mark.usefixtures('ds1_copy') -def test_wild(request, runner, yadm_y, paths, - tst_sys, tst_host, tst_user, - tracked, encrypt, - wild_class, wild_host, wild_sys, wild_user, - template): - """Test wild linking - - These tests are done by creating permutations of the possible files using - WILD_TEMPLATES. Each case is then tested (while skipping the already tested - permutations for efficiency). - """ - - if request.node.name in BROKEN_TEST_IDS: - pytest.xfail( - 'This test is known to be broken. ' - 'This bug only affects deprecated features.') - - tst_class = 'testclass' - - # determine the "wild" version of the suffix - str_class = '%' if wild_class else tst_class - str_host = '%' if wild_host else tst_host - str_sys = '%' if wild_sys else tst_sys - str_user = '%' if wild_user else tst_user - wild_suffix = string.Template(template).substitute( - tst_class=str_class, - tst_host=str_host, - tst_sys=str_sys, - tst_user=str_user, - ) - - # determine the "standard" version of the suffix - std_suffix = string.Template(template).substitute( - tst_class=tst_class, - tst_host=tst_host, - tst_sys=tst_sys, - tst_user=tst_user, - ) - - # skip over duplicate tests (this seems to be the simplest way to cover the - # permutations of tests, while skipping duplicates.) - test_key = f'{tracked}{encrypt}{wild_suffix}{std_suffix}' - if test_key in WILD_TESTED: - return - WILD_TESTED.add(test_key) - - # set the class - utils.set_local(paths, 'class', tst_class) - - # create files using the wild suffix - utils.create_alt_files(paths, wild_suffix, tracked=tracked, - encrypt=encrypt, exclude=False) - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + wild_suffix - assert paths.work.join(file_path).islink() - target = py.path.local(os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - - # create files using the standard suffix - utils.create_alt_files(paths, std_suffix, tracked=tracked, - encrypt=encrypt, exclude=False) - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + std_suffix - assert paths.work.join(file_path).islink() - target = py.path.local(os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - - -@pytest.mark.usefixtures('ds1_copy') -def test_local_override(runner, yadm_y, paths, - tst_sys, tst_host, tst_user): - """Test local overrides""" - - # define local overrides - utils.set_local(paths, 'class', 'or-class') - utils.set_local(paths, 'hostname', 'or-hostname') - utils.set_local(paths, 'os', 'or-os') - utils.set_local(paths, 'user', 'or-user') - - # create files, the first would normally be the most specific version - # however, the second is the overridden version which should be preferred. - utils.create_alt_files( - paths, f'##or-class.{tst_sys}.{tst_host}.{tst_user}') - utils.create_alt_files( - paths, '##or-class.or-os.or-hostname.or-user') - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + '##or-class.or-os.or-hostname.or-user' - assert paths.work.join(file_path).islink() - target = py.path.local(os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - - -@pytest.mark.parametrize('suffix', ['AAA', 'ZZZ', 'aaa', 'zzz']) -@pytest.mark.usefixtures('ds1_copy') -def test_class_case(runner, yadm_y, paths, tst_sys, suffix): - """Test range of class cases""" - - # set the class - utils.set_local(paths, 'class', suffix) - - # create files - endings = [suffix] - if tst_sys == 'Linux': - # Only create all of these side-by-side on Linux, which is - # unquestionably case-sensitive. This would break tests on - # case-insensitive systems. - endings = ['AAA', 'ZZZ', 'aaa', 'zzz'] - for ending in endings: - utils.create_alt_files(paths, f'##{ending}') - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + f'##{suffix}' - assert paths.work.join(file_path).islink() - target = py.path.local(os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - - -@pytest.mark.parametrize('autoalt', [None, 'true', 'false']) -@pytest.mark.usefixtures('ds1_copy') -def test_auto_alt(runner, yadm_y, paths, autoalt): - """Test setting auto-alt""" - - # set the value of auto-alt - if autoalt: - os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt))) - - # create file - suffix = '##' - utils.create_alt_files(paths, suffix) - - # run status to possibly trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('status'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + suffix - if autoalt == 'false': - assert not paths.work.join(file_path).exists() - else: - assert paths.work.join(file_path).islink() - target = py.path.local( - os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - # no linking output when run via auto-alt - assert str(paths.work.join(source_file)) not in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - # no linking output when run via auto-alt - assert str(paths.work.join(source_file)) not in linked - - -@pytest.mark.parametrize('delimiter', ['.', '_']) -@pytest.mark.usefixtures('ds1_copy') -def test_delimiter(runner, yadm_y, paths, - tst_sys, tst_host, tst_user, delimiter): - """Test delimiters used""" - - suffix = '##' + delimiter.join([tst_sys, tst_host, tst_user]) - - # create file - utils.create_alt_files(paths, suffix) - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - # only a delimiter of '.' is valid - for file_path in TEST_PATHS: - source_file = file_path + suffix - if delimiter == '.': - assert paths.work.join(file_path).islink() - target = py.path.local( - os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert not paths.work.join(file_path).exists() - assert str(paths.work.join(source_file)) not in linked - - -@pytest.mark.usefixtures('ds1_copy') -def test_invalid_links_removed(runner, yadm_y, paths): - """Links to invalid alternative files are removed - - This test ensures that when an already linked alternative becomes invalid - due to a change in class, the alternate link is removed. - """ - - # set the class - tst_class = 'testclass' - utils.set_local(paths, 'class', tst_class) - - # create files which match the test class - utils.create_alt_files(paths, f'##{tst_class}') - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the proper linking has occurred - for file_path in TEST_PATHS: - source_file = file_path + '##' + tst_class - assert paths.work.join(file_path).islink() - target = py.path.local(os.path.realpath(paths.work.join(file_path))) - if target.isfile(): - assert paths.work.join(file_path).read() == source_file - assert str(paths.work.join(source_file)) in linked - else: - assert paths.work.join(file_path).join( - utils.CONTAINED).read() == source_file - assert str(paths.work.join(source_file)) in linked - - # change the class so there are no valid alternates - utils.set_local(paths, 'class', 'changedclass') - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - linked = utils.parse_alt_output(run.out) - - # assert the linking is removed - for file_path in TEST_PATHS: - source_file = file_path + '##' + tst_class - assert not paths.work.join(file_path).exists() - assert str(paths.work.join(source_file)) not in linked diff --git a/test/test_compat_jinja.py b/test/test_compat_jinja.py deleted file mode 100644 index 7e2b766..0000000 --- a/test/test_compat_jinja.py +++ /dev/null @@ -1,198 +0,0 @@ -"""Test jinja""" - -import os -import pytest -import utils - -# These tests are for the template processing in YADM_COMPATIBILITY=1 mode -pytestmark = pytest.mark.deprecated - - -@pytest.fixture(scope='module') -def envtpl_present(runner): - """Is envtpl present and working?""" - try: - run = runner(command=['envtpl', '-h']) - if run.success: - return True - except OSError: - pass - return False - - -@pytest.mark.usefixtures('ds1_copy') -def test_local_override(runner, yadm_y, paths, - tst_distro, envtpl_present): - """Test local overrides""" - if not envtpl_present: - pytest.skip('Unable to test without envtpl.') - - # define local overrides - utils.set_local(paths, 'class', 'or-class') - utils.set_local(paths, 'hostname', 'or-hostname') - utils.set_local(paths, 'os', 'or-os') - utils.set_local(paths, 'user', 'or-user') - - template = ( - 'j2-{{ YADM_CLASS }}-' - '{{ YADM_OS }}-{{ YADM_HOSTNAME }}-' - '{{ YADM_USER }}-{{ YADM_DISTRO }}' - '-{%- ' - f"include '{utils.INCLUDE_FILE}'" - ' -%}' - ) - expected = ( - f'j2-or-class-or-os-or-hostname-or-user-{tst_distro}' - f'-{utils.INCLUDE_CONTENT}' - ) - - utils.create_alt_files(paths, '##yadm.j2', content=template, - includefile=True) - - # os.system(f'find {paths.work}' + ' -name *j2 -ls -exec cat \'{}\' ";"') - # os.system(f'find {paths.work}') - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - created = utils.parse_alt_output(run.out, linked=False) - - # assert the proper creation has occurred - for file_path in (utils.ALT_FILE1, utils.ALT_FILE2): - source_file = file_path + '##yadm.j2' - assert paths.work.join(file_path).isfile() - lines = paths.work.join(file_path).readlines(cr=False) - assert lines[0] == source_file - assert lines[1] == expected - assert str(paths.work.join(source_file)) in created - - -@pytest.mark.parametrize('autoalt', [None, 'true', 'false']) -@pytest.mark.usefixtures('ds1_copy') -def test_auto_alt(runner, yadm_y, paths, autoalt, tst_sys, - envtpl_present): - """Test setting auto-alt""" - - if not envtpl_present: - pytest.skip('Unable to test without envtpl.') - - # set the value of auto-alt - if autoalt: - os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt))) - - # create file - jinja_suffix = '##yadm.j2' - utils.create_alt_files(paths, jinja_suffix, content='{{ YADM_OS }}') - - # run status to possibly trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('status'), env=env) - assert run.success - assert run.err == '' - created = utils.parse_alt_output(run.out, linked=False) - - # assert the proper creation has occurred - for file_path in (utils.ALT_FILE1, utils.ALT_FILE2): - source_file = file_path + jinja_suffix - if autoalt == 'false': - assert not paths.work.join(file_path).exists() - else: - assert paths.work.join(file_path).isfile() - lines = paths.work.join(file_path).readlines(cr=False) - assert lines[0] == source_file - assert lines[1] == tst_sys - # no created output when run via auto-alt - assert str(paths.work.join(source_file)) not in created - - -@pytest.mark.usefixtures('ds1_copy') -def test_jinja_envtpl_missing(runner, paths): - """Test operation when envtpl is missing""" - - script = f""" - YADM_TEST=1 source {paths.pgm} - process_global_args -Y "{paths.yadm}" - set_operating_system - configure_paths - YADM_COMPATIBILITY=1 - ENVTPL_PROGRAM='envtpl_missing' main alt - """ - - utils.create_alt_files(paths, '##yadm.j2') - - run = runner(command=['bash'], inp=script) - assert run.success - assert run.err == '' - assert f'envtpl not available, not creating' in run.out - - -@pytest.mark.parametrize( - 'tracked, encrypt, exclude', [ - (False, False, False), - (True, False, False), - (False, True, False), - (False, True, True), - ], ids=[ - 'untracked', - 'tracked', - 'encrypted', - 'excluded', - ]) -@pytest.mark.usefixtures('ds1_copy') -def test_jinja(runner, yadm_y, paths, - tst_sys, tst_host, tst_user, tst_distro, - tracked, encrypt, exclude, - envtpl_present): - """Test jinja processing""" - - if not envtpl_present: - pytest.skip('Unable to test without envtpl.') - - jinja_suffix = '##yadm.j2' - - # set the class - tst_class = 'testclass' - utils.set_local(paths, 'class', tst_class) - - template = ( - 'j2-{{ YADM_CLASS }}-' - '{{ YADM_OS }}-{{ YADM_HOSTNAME }}-' - '{{ YADM_USER }}-{{ YADM_DISTRO }}' - '-{%- ' - f"include '{utils.INCLUDE_FILE}'" - ' -%}' - ) - expected = ( - f'j2-{tst_class}-' - f'{tst_sys}-{tst_host}-' - f'{tst_user}-{tst_distro}' - f'-{utils.INCLUDE_CONTENT}' - ) - - utils.create_alt_files(paths, jinja_suffix, content=template, - tracked=tracked, encrypt=encrypt, exclude=exclude, - includefile=True) - - # run alt to trigger linking - env = os.environ.copy() - env['YADM_COMPATIBILITY'] = '1' - run = runner(yadm_y('alt'), env=env) - assert run.success - assert run.err == '' - created = utils.parse_alt_output(run.out, linked=False) - - # assert the proper creation has occurred - for file_path in (utils.ALT_FILE1, utils.ALT_FILE2): - source_file = file_path + jinja_suffix - if tracked or (encrypt and not exclude): - assert paths.work.join(file_path).isfile() - lines = paths.work.join(file_path).readlines(cr=False) - assert lines[0] == source_file - assert lines[1] == expected - assert str(paths.work.join(source_file)) in created - else: - assert not paths.work.join(file_path).exists() - assert str(paths.work.join(source_file)) not in created diff --git a/test/test_config.py b/test/test_config.py index 4e44b1c..d364128 100644 --- a/test/test_config.py +++ b/test/test_config.py @@ -10,7 +10,7 @@ TEST_VALUE = 'testvalue' TEST_FILE = f'[{TEST_SECTION}]\n\t{TEST_ATTRIBUTE} = {TEST_VALUE}' -def test_config_no_params(runner, yadm_y, supported_configs): +def test_config_no_params(runner, yadm_cmd, supported_configs): """No parameters Display instructions @@ -18,7 +18,7 @@ def test_config_no_params(runner, yadm_y, supported_configs): Exit with 0 """ - run = runner(yadm_y('config')) + run = runner(yadm_cmd('config')) assert run.success assert run.err == '' @@ -27,21 +27,21 @@ def test_config_no_params(runner, yadm_y, supported_configs): assert config in run.out -def test_config_read_missing(runner, yadm_y): +def test_config_read_missing(runner, yadm_cmd): """Read missing attribute Display an empty value Exit with 0 """ - run = runner(yadm_y('config', TEST_KEY)) + run = runner(yadm_cmd('config', TEST_KEY)) assert run.success assert run.err == '' assert run.out == '' -def test_config_write(runner, yadm_y, paths): +def test_config_write(runner, yadm_cmd, paths): """Write attribute Display no output @@ -49,7 +49,7 @@ def test_config_write(runner, yadm_y, paths): Exit with 0 """ - run = runner(yadm_y('config', TEST_KEY, TEST_VALUE)) + run = runner(yadm_cmd('config', TEST_KEY, TEST_VALUE)) assert run.success assert run.err == '' @@ -57,7 +57,7 @@ def test_config_write(runner, yadm_y, paths): assert paths.config.read().strip() == TEST_FILE -def test_config_read(runner, yadm_y, paths): +def test_config_read(runner, yadm_cmd, paths): """Read attribute Display value @@ -65,14 +65,14 @@ def test_config_read(runner, yadm_y, paths): """ paths.config.write(TEST_FILE) - run = runner(yadm_y('config', TEST_KEY)) + run = runner(yadm_cmd('config', TEST_KEY)) assert run.success assert run.err == '' assert run.out.strip() == TEST_VALUE -def test_config_update(runner, yadm_y, paths): +def test_config_update(runner, yadm_cmd, paths): """Update attribute Display no output @@ -82,7 +82,7 @@ def test_config_update(runner, yadm_y, paths): paths.config.write(TEST_FILE) - run = runner(yadm_y('config', TEST_KEY, TEST_VALUE + 'extra')) + run = runner(yadm_cmd('config', TEST_KEY, TEST_VALUE + 'extra')) assert run.success assert run.err == '' @@ -92,7 +92,7 @@ def test_config_update(runner, yadm_y, paths): @pytest.mark.usefixtures('ds1_repo_copy') -def test_config_local_read(runner, yadm_y, paths, supported_local_configs): +def test_config_local_read(runner, yadm_cmd, paths, supported_local_configs): """Read local attribute Display value from the repo config @@ -107,14 +107,14 @@ def test_config_local_read(runner, yadm_y, paths, supported_local_configs): # run yadm config for config in supported_local_configs: - run = runner(yadm_y('config', config)) + run = runner(yadm_cmd('config', config)) assert run.success assert run.err == '' assert run.out.strip() == f'value_of_{config}' @pytest.mark.usefixtures('ds1_repo_copy') -def test_config_local_write(runner, yadm_y, paths, supported_local_configs): +def test_config_local_write(runner, yadm_cmd, paths, supported_local_configs): """Write local attribute Display no output @@ -124,7 +124,7 @@ def test_config_local_write(runner, yadm_y, paths, supported_local_configs): # run yadm config for config in supported_local_configs: - run = runner(yadm_y('config', config, f'value_of_{config}')) + run = runner(yadm_cmd('config', config, f'value_of_{config}')) assert run.success assert run.err == '' assert run.out == '' @@ -137,3 +137,27 @@ def test_config_local_write(runner, yadm_y, paths, supported_local_configs): assert run.success assert run.err == '' assert run.out.strip() == f'value_of_{config}' + + +def test_config_without_parent_directory(runner, yadm_cmd, paths): + """Write and read attribute to/from config file with non-existent parent dir + + Update configuration file + Display value + Exit with 0 + """ + + config_file = paths.root + '/folder/does/not/exist/config' + + run = runner( + yadm_cmd('--yadm-config', config_file, 'config', TEST_KEY, TEST_VALUE)) + + assert run.success + assert run.err == '' + assert run.out == '' + + run = runner(yadm_cmd('--yadm-config', config_file, 'config', TEST_KEY)) + + assert run.success + assert run.err == '' + assert run.out.strip() == TEST_VALUE diff --git a/test/test_default_remote_branch.py b/test/test_default_remote_branch.py new file mode 100644 index 0000000..6405417 --- /dev/null +++ b/test/test_default_remote_branch.py @@ -0,0 +1,27 @@ +"""Unit tests: _default_remote_branch()""" +import pytest + + +@pytest.mark.parametrize('condition', ['found', 'missing']) +def test(runner, paths, condition): + """Test _default_remote_branch()""" + test_branch = 'test/branch' + output = f'ref: refs/heads/{test_branch}\\tHEAD\\n' + if condition == 'missing': + output = 'output that is missing ref' + script = f""" + YADM_TEST=1 source {paths.pgm} + function git() {{ + printf '{output}'; + printf 'mock stderr\\n' 1>&2 + }} + _default_remote_branch URL + """ + print(condition) + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + if condition == 'found': + assert run.out.strip() == test_branch + else: + assert run.out.strip() == 'master' diff --git a/test/test_encryption.py b/test/test_encryption.py index b0d00de..fea2ff0 100644 --- a/test/test_encryption.py +++ b/test/test_encryption.py @@ -61,7 +61,7 @@ def asymmetric_key(runner, gnupg): @pytest.fixture -def encrypt_targets(yadm_y, paths): +def encrypt_targets(yadm_cmd, paths): """Fixture for setting up data to encrypt This fixture: @@ -78,7 +78,7 @@ def encrypt_targets(yadm_y, paths): """ # init empty yadm repo - os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f'))) + os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f'))) expected = [] @@ -186,7 +186,7 @@ def decrypt_targets(tmpdir_factory, runner, gnupg): 'overwrite', [False, True], ids=['clean', 'overwrite']) def test_symmetric_encrypt( - runner, yadm_y, paths, encrypt_targets, + runner, yadm_cmd, paths, encrypt_targets, gnupg, bad_phrase, overwrite, missing_encrypt): """Test symmetric encryption""" @@ -203,7 +203,7 @@ def test_symmetric_encrypt( env = os.environ.copy() env['GNUPGHOME'] = gnupg.home - run = runner(yadm_y('encrypt'), env=env) + run = runner(yadm_cmd('encrypt'), env=env) if missing_encrypt or bad_phrase: assert run.failure @@ -212,7 +212,7 @@ def test_symmetric_encrypt( assert run.err == '' if missing_encrypt: - assert 'does not exist' in run.out + assert 'does not exist' in run.err elif bad_phrase: assert 'Invalid passphrase' in run.err else: @@ -230,12 +230,12 @@ def test_symmetric_encrypt( 'dolist', [False, True], ids=['decrypt', 'list']) def test_symmetric_decrypt( - runner, yadm_y, paths, decrypt_targets, gnupg, + runner, yadm_cmd, paths, decrypt_targets, gnupg, dolist, archive_exists, bad_phrase): """Test decryption""" # init empty yadm repo - os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f'))) + os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f'))) if bad_phrase: gnupg.pw('') @@ -256,7 +256,7 @@ def test_symmetric_decrypt( if dolist: args.append('-l') - run = runner(yadm_y('decrypt') + args, env=env) + run = runner(yadm_cmd('decrypt') + args, env=env) if archive_exists and not bad_phrase: assert run.success @@ -284,16 +284,16 @@ def test_symmetric_decrypt( 'overwrite', [False, True], ids=['clean', 'overwrite']) def test_asymmetric_encrypt( - runner, yadm_y, paths, encrypt_targets, gnupg, + runner, yadm_cmd, paths, encrypt_targets, gnupg, overwrite, key_exists, ask): """Test asymmetric encryption""" # specify encryption recipient if ask: - os.system(' '.join(yadm_y('config', 'yadm.gpg-recipient', 'ASK'))) + os.system(' '.join(yadm_cmd('config', 'yadm.gpg-recipient', 'ASK'))) expect = [('Enter the user ID', KEY_NAME), ('Enter the user ID', '')] else: - os.system(' '.join(yadm_y('config', 'yadm.gpg-recipient', KEY_NAME))) + os.system(' '.join(yadm_cmd('config', 'yadm.gpg-recipient', KEY_NAME))) expect = [] if overwrite: @@ -305,7 +305,7 @@ def test_asymmetric_encrypt( env = os.environ.copy() env['GNUPGHOME'] = gnupg.home - run = runner(yadm_y('encrypt'), env=env, expect=expect) + run = runner(yadm_cmd('encrypt'), env=env, expect=expect) if key_exists: assert run.success @@ -313,7 +313,7 @@ def test_asymmetric_encrypt( runner, gnupg, paths.archive, encrypt_targets) else: assert run.failure - assert 'Unable to write' in run.out + assert 'Unable to write' in run.out if expect else run.err if ask: assert 'Enter the user ID' in run.out @@ -321,17 +321,17 @@ def test_asymmetric_encrypt( @pytest.mark.usefixtures('asymmetric_key') @pytest.mark.usefixtures('encrypt_targets') -def test_multi_key(runner, yadm_y, gnupg): +def test_multi_key(runner, yadm_cmd, gnupg): """Test multiple recipients""" # specify two encryption recipient - os.system(' '.join(yadm_y( + os.system(' '.join(yadm_cmd( 'config', 'yadm.gpg-recipient', f'"{KEY_NAME} second-key"'))) env = os.environ.copy() env['GNUPGHOME'] = gnupg.home - run = runner(yadm_y('encrypt'), env=env) + run = runner(yadm_cmd('encrypt'), env=env) assert run.failure assert 'second-key: skipped: No public key' in run.err @@ -345,12 +345,12 @@ def test_multi_key(runner, yadm_y, gnupg): 'dolist', [False, True], ids=['decrypt', 'list']) def test_asymmetric_decrypt( - runner, yadm_y, paths, decrypt_targets, gnupg, + runner, yadm_cmd, paths, decrypt_targets, gnupg, dolist, key_exists): """Test decryption""" # init empty yadm repo - os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f'))) + os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f'))) decrypt_targets['asymmetric'].copy(paths.archive) @@ -366,7 +366,7 @@ def test_asymmetric_decrypt( args.append('-l') env = os.environ.copy() env['GNUPGHOME'] = gnupg.home - run = runner(yadm_y('decrypt') + args, env=env) + run = runner(yadm_cmd('decrypt') + args, env=env) if key_exists: assert run.success @@ -380,7 +380,7 @@ def test_asymmetric_decrypt( assert paths.work.join(filename).read() == filename else: assert run.failure - assert 'Unable to extract encrypted files' in run.out + assert 'Unable to extract encrypted files' in run.err @pytest.mark.parametrize( @@ -388,7 +388,7 @@ def test_asymmetric_decrypt( [False, 'y', 'n'], ids=['tracked', 'untracked_answer_y', 'untracked_answer_n']) def test_offer_to_add( - runner, yadm_y, paths, encrypt_targets, gnupg, untracked): + runner, yadm_cmd, paths, encrypt_targets, gnupg, untracked): """Test offer to add encrypted archive All the other encryption tests use an archive outside of the work tree. @@ -408,10 +408,10 @@ def test_offer_to_add( expect.append(('add it now', untracked)) else: worktree_archive.write('exists') - os.system(' '.join(yadm_y('add', str(worktree_archive)))) + os.system(' '.join(yadm_cmd('add', str(worktree_archive)))) run = runner( - yadm_y('encrypt', '--yadm-archive', str(worktree_archive)), + yadm_cmd('encrypt', '--yadm-archive', str(worktree_archive)), env=env, expect=expect ) @@ -422,7 +422,7 @@ def test_offer_to_add( runner, gnupg, worktree_archive, encrypt_targets) run = runner( - yadm_y('status', '--porcelain', '-uall', str(worktree_archive))) + yadm_cmd('status', '--porcelain', '-uall', str(worktree_archive))) assert run.success assert run.err == '' @@ -438,7 +438,7 @@ def test_offer_to_add( @pytest.mark.usefixtures('ds1_copy') -def test_encrypt_added_to_exclude(runner, yadm_y, paths, gnupg): +def test_encrypt_added_to_exclude(runner, yadm_cmd, paths, gnupg): """Confirm that .config/yadm/encrypt is added to exclude""" gnupg.pw(PASSPHRASE) @@ -450,7 +450,7 @@ def test_encrypt_added_to_exclude(runner, yadm_y, paths, gnupg): paths.work.join('test-encrypt-data').write('') exclude_file.write('original-data', ensure=True) - run = runner(yadm_y('encrypt'), env=env) + run = runner(yadm_cmd('encrypt'), env=env) assert 'test-encrypt-data' in paths.repo.join('info/exclude').read() assert 'original-data' in paths.repo.join('info/exclude').read() diff --git a/test/test_enter.py b/test/test_enter.py index d1f65d0..f5ea2d8 100644 --- a/test/test_enter.py +++ b/test/test_enter.py @@ -17,7 +17,7 @@ import pytest 'shell-noexec', ]) @pytest.mark.usefixtures('ds1_copy') -def test_enter(runner, yadm_y, paths, shell, success): +def test_enter(runner, yadm_cmd, paths, shell, success): """Enter tests""" env = os.environ.copy() if shell == 'delete': @@ -33,15 +33,15 @@ def test_enter(runner, yadm_y, paths, shell, success): else: env['SHELL'] = shell - run = runner(command=yadm_y('enter'), env=env) + run = runner(command=yadm_cmd('enter'), env=env) assert run.success == success - assert run.err == '' prompt = f'yadm shell ({paths.repo})' if success: assert run.out.startswith('Entering yadm repo') assert run.out.rstrip().endswith('Leaving yadm repo') - if not success: - assert 'does not refer to an executable' in run.out + assert run.err == '' + else: + assert 'does not refer to an executable' in run.err if 'env' in shell: assert f'GIT_DIR={paths.repo}' in run.out assert f'GIT_WORK_TREE={paths.work}' in run.out @@ -63,8 +63,12 @@ def test_enter(runner, yadm_y, paths, shell, success): 'cmd', [False, 'cmd', 'cmd-bad-exit'], ids=['no-cmd', 'cmd', 'cmd-bad-exit']) +@pytest.mark.parametrize( + 'term', ['', 'dumb'], + ids=['term-empty', 'term-dumb']) @pytest.mark.usefixtures('ds1_copy') -def test_enter_shell_ops(runner, yadm_y, paths, shell, opts, path, cmd): +def test_enter_shell_ops(runner, yadm_cmd, paths, shell, + opts, path, cmd, term): """Enter tests for specific shell options""" change_exit = '\nfalse' if cmd == 'cmd-bad-exit' else '' @@ -83,9 +87,13 @@ def test_enter_shell_ops(runner, yadm_y, paths, shell, opts, path, cmd): enter_cmd += test_cmd env = os.environ.copy() + env['TERM'] = term env['SHELL'] = custom_shell - run = runner(command=yadm_y(*enter_cmd), env=env) + if shell == 'zsh' and term == 'dumb': + opts += ' --no-zle' + + run = runner(command=yadm_cmd(*enter_cmd), env=env) if cmd == 'cmd-bad-exit': assert run.failure else: diff --git a/test/test_git_crypt.py b/test/test_ext_crypt.py index 6b92de9..cb74afc 100644 --- a/test/test_git_crypt.py +++ b/test/test_ext_crypt.py @@ -1,4 +1,4 @@ -"""Test git-crypt""" +"""Test external encryption commands""" import pytest @@ -8,15 +8,21 @@ import pytest [False, 'installed', 'installed-but-failed'], ids=['not-installed', 'installed', 'installed-but-failed'] ) -def test_git_crypt(runner, yadm, paths, tmpdir, crypt): - """git-crypt tests""" +@pytest.mark.parametrize( + 'cmd,var', [ + ['git_crypt', 'GIT_CRYPT_PROGRAM'], + ['transcrypt', 'TRANSCRYPT_PROGRAM'], + ], + ids=['git-crypt', 'transcrypt']) +def test_ext_encryption(runner, yadm, paths, tmpdir, crypt, cmd, var): + """External encryption tests""" paths.repo.ensure(dir=True) bindir = tmpdir.mkdir('bin') - pgm = bindir.join('test-git-crypt') + pgm = bindir.join('test-ext-crypt') if crypt: - pgm.write(f'#!/bin/sh\necho git-crypt ran\n') + pgm.write('#!/bin/sh\necho ext-crypt ran\n') pgm.chmod(0o775) if crypt == 'installed-but-failed': pgm.write('false\n', mode='a') @@ -24,8 +30,8 @@ def test_git_crypt(runner, yadm, paths, tmpdir, crypt): script = f""" YADM_TEST=1 source {yadm} YADM_REPO={paths.repo} - GIT_CRYPT_PROGRAM="{pgm}" - git_crypt "param1" + {var}="{pgm}" + {cmd} "param1" """ run = runner(command=['bash'], inp=script) @@ -35,8 +41,8 @@ def test_git_crypt(runner, yadm, paths, tmpdir, crypt): assert run.failure else: assert run.success - assert run.out.strip() == 'git-crypt ran' + assert run.out.strip() == 'ext-crypt ran' + assert run.err == '' else: assert run.failure - assert f"command '{pgm}' cannot be located" in run.out - assert run.err == '' + assert f"command '{pgm}' cannot be located" in run.err diff --git a/test/test_git.py b/test/test_git.py index 427c54a..76eccab 100644 --- a/test/test_git.py +++ b/test/test_git.py @@ -5,7 +5,7 @@ import pytest @pytest.mark.usefixtures('ds1_copy') -def test_git(runner, yadm_y, paths): +def test_git(runner, yadm_cmd, paths): """Test series of passthrough git commands Passthru unknown commands to Git @@ -17,14 +17,14 @@ def test_git(runner, yadm_y, paths): """ # passthru unknown commands to Git - run = runner(command=yadm_y('bogus')) + run = runner(command=yadm_cmd('bogus')) assert run.failure assert "git: 'bogus' is not a git command." in run.err assert "See 'git --help'" in run.err assert run.out == '' # git command 'add' - badfile - run = runner(command=yadm_y('add', '-v', 'does_not_exist')) + run = runner(command=yadm_cmd('add', '-v', 'does_not_exist')) assert run.code == 128 assert "pathspec 'does_not_exist' did not match any files" in run.err assert run.out == '' @@ -32,19 +32,19 @@ def test_git(runner, yadm_y, paths): # git command 'add' newfile = paths.work.join('test_git') newfile.write('test_git') - run = runner(command=yadm_y('add', '-v', str(newfile))) + run = runner(command=yadm_cmd('add', '-v', str(newfile))) assert run.success assert run.err == '' assert "add 'test_git'" in run.out # git command 'status' - run = runner(command=yadm_y('status')) + run = runner(command=yadm_cmd('status')) assert run.success assert run.err == '' assert re.search(r'new file:\s+test_git', run.out) # git command 'commit' - run = runner(command=yadm_y('commit', '-m', 'Add test_git')) + run = runner(command=yadm_cmd('commit', '-m', 'Add test_git')) assert run.success assert run.err == '' assert '1 file changed' in run.out @@ -52,7 +52,7 @@ def test_git(runner, yadm_y, paths): assert re.search(r'create mode .+ test_git', run.out) # git command 'log' - run = runner(command=yadm_y('log', '--oneline')) + run = runner(command=yadm_cmd('log', '--oneline')) assert run.success assert run.err == '' assert 'Add test_git' in run.out diff --git a/test/test_help.py b/test/test_help.py index 79a7652..0d8f2c3 100644 --- a/test/test_help.py +++ b/test/test_help.py @@ -1,17 +1,19 @@ """Test help""" +import pytest -def test_missing_command(runner, yadm_y): +def test_missing_command(runner, yadm_cmd): """Run without any command""" - run = runner(command=yadm_y()) + run = runner(command=yadm_cmd()) assert run.failure assert run.err == '' assert run.out.startswith('Usage: yadm') -def test_help_command(runner, yadm_y): +@pytest.mark.parametrize('cmd', ['--help', 'help']) +def test_help_command(runner, yadm_cmd, cmd): """Run with help command""" - run = runner(command=yadm_y('help')) + run = runner(command=yadm_cmd(cmd)) assert run.failure assert run.err == '' assert run.out.startswith('Usage: yadm') diff --git a/test/test_hooks.py b/test/test_hooks.py index 36f3d98..704636a 100644 --- a/test/test_hooks.py +++ b/test/test_hooks.py @@ -21,8 +21,9 @@ import pytest 'pre-post-success', 'pre-post-fail', ]) +@pytest.mark.parametrize('cmd', ['--version', 'version']) def test_hooks( - runner, yadm_y, paths, + runner, yadm_cmd, paths, cmd, pre, pre_code, post, post_code): """Test pre/post hook""" @@ -33,7 +34,7 @@ def test_hooks( create_hook(paths, 'post_version', post_code) # run yadm - run = runner(yadm_y('version')) + run = runner(yadm_cmd(cmd)) # when a pre hook fails, yadm should exit with the hook's code assert run.code == pre_code assert run.err == '' @@ -53,7 +54,7 @@ def test_hooks( # repo fixture is needed to test the population of YADM_HOOK_WORK @pytest.mark.usefixtures('ds1_repo_copy') -def test_hook_env(runner, yadm_y, paths): +def test_hook_env(runner, yadm_cmd, paths): """Test hook environment""" # test will be done with a non existent "git" passthru command @@ -65,7 +66,7 @@ def test_hook_env(runner, yadm_y, paths): hook.write('#!/bin/bash\nenv\ndeclare\n') hook.chmod(0o755) - run = runner(yadm_y(cmd, 'extra_args')) + run = runner(yadm_cmd(cmd, 'extra_args')) # expect passthru to fail assert run.failure @@ -78,7 +79,7 @@ def test_hook_env(runner, yadm_y, paths): assert f'YADM_HOOK_FULL_COMMAND={cmd} extra_args\n' in run.out assert f'YADM_HOOK_REPO={paths.repo}\n' in run.out assert f'YADM_HOOK_WORK={paths.work}\n' in run.out - assert f'YADM_ENCRYPT_INCLUDE_FILES=\n' in run.out + assert 'YADM_ENCRYPT_INCLUDE_FILES=\n' in run.out # verify the hook environment contains certain exported functions for func in [ @@ -103,7 +104,7 @@ def test_hook_env(runner, yadm_y, paths): assert 'YADM_ENCRYPT_INCLUDE_FILES=a\nb\nc\n' in run.out -def test_escaped(runner, yadm_y, paths): +def test_escaped(runner, yadm_cmd, paths): """Test escaped values in YADM_HOOK_FULL_COMMAND""" # test will be done with a non existent "git" passthru command @@ -115,7 +116,7 @@ def test_escaped(runner, yadm_y, paths): hook.write('#!/bin/bash\nenv\n') hook.chmod(0o755) - run = runner(yadm_y(cmd, 'a b', 'c\td', 'e\\f')) + run = runner(yadm_cmd(cmd, 'a b', 'c\td', 'e\\f')) # expect passthru to fail assert run.failure @@ -126,6 +127,39 @@ def test_escaped(runner, yadm_y, paths): 'a\\ b c\\\td e\\\\f\n') in run.out +@pytest.mark.parametrize('condition', ['exec', 'no-exec', 'mingw']) +def test_executable(runner, paths, condition): + """Verify hook must be exectuable""" + cmd = 'version' + hook = paths.hooks.join(f'pre_{cmd}') + hook.write('#!/bin/sh\necho HOOK\n') + hook.chmod(0o644) + if condition == 'exec': + hook.chmod(0o755) + + mingw = 'OPERATING_SYSTEM="MINGWx"' if condition == 'mingw' else '' + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_HOOKS="{paths.hooks}" + HOOK_COMMAND="{cmd}" + {mingw} + invoke_hook "pre" + """ + run = runner(command=['bash'], inp=script) + + if condition != 'mingw': + assert run.success + assert run.err == '' + else: + assert run.failure + assert 'Permission denied' in run.err + + if condition == 'exec': + assert 'HOOK' in run.out + elif condition == 'no-exec': + assert 'HOOK' not in run.out + + def create_hook(paths, name, code): """Create hook""" hook = paths.hooks.join(name) diff --git a/test/test_init.py b/test/test_init.py index 1519b38..c738a02 100644 --- a/test/test_init.py +++ b/test/test_init.py @@ -19,7 +19,7 @@ import pytest ]) @pytest.mark.usefixtures('ds1_work_copy') def test_init( - runner, yadm_y, paths, repo_config, alt_work, repo_present, force): + runner, yadm_cmd, paths, repo_config, alt_work, repo_present, force): """Test init Repos should have attribs: @@ -51,16 +51,16 @@ def test_init( args.append('-f') # run init - run = runner(yadm_y(*args), env={'HOME': home}) - assert run.err == '' + run = runner(yadm_cmd(*args), env={'HOME': home}) if repo_present and not force: assert run.failure - assert 'repo already exists' in run.out + assert 'repo already exists' in run.err assert old_repo.isfile(), 'Missing original repo' else: assert run.success assert 'Initialized empty shared Git repository' in run.out + assert run.err == '' if repo_present: assert not old_repo.isfile(), 'Original repo still exists' diff --git a/test/test_introspect.py b/test/test_introspect.py index fcadf14..b292bd4 100644 --- a/test/test_introspect.py +++ b/test/test_introspect.py @@ -13,13 +13,13 @@ import pytest 'switches', ]) def test_introspect_category( - runner, yadm_y, paths, name, + runner, yadm_cmd, paths, name, supported_commands, supported_configs, supported_switches): """Validate introspection category""" if name: - run = runner(command=yadm_y('introspect', name)) + run = runner(command=yadm_cmd('introspect', name)) else: - run = runner(command=yadm_y('introspect')) + run = runner(command=yadm_cmd('introspect')) assert run.success assert run.err == '' diff --git a/test/test_list.py b/test/test_list.py index c2d8631..dcfe500 100644 --- a/test/test_list.py +++ b/test/test_list.py @@ -11,7 +11,7 @@ import pytest 'subdir', ]) @pytest.mark.usefixtures('ds1_copy') -def test_list(runner, yadm_y, paths, ds1, location): +def test_list(runner, yadm_cmd, paths, ds1, location): """List tests""" if location == 'work': run_dir = paths.work @@ -23,7 +23,7 @@ def test_list(runner, yadm_y, paths, ds1, location): with run_dir.as_cwd(): # test with '-a' # should get all tracked files, relative to the work path - run = runner(command=yadm_y('list', '-a')) + run = runner(command=yadm_cmd('list', '-a')) assert run.success assert run.err == '' returned_files = set(run.out.splitlines()) @@ -33,7 +33,7 @@ def test_list(runner, yadm_y, paths, ds1, location): # should get all tracked files, relative to the work path unless in a # subdir, then those should be a limited set of files, relative to the # subdir - run = runner(command=yadm_y('list')) + run = runner(command=yadm_cmd('list')) assert run.success assert run.err == '' returned_files = set(run.out.splitlines()) diff --git a/test/test_perms.py b/test/test_perms.py index 0eb8add..4f052bd 100644 --- a/test/test_perms.py +++ b/test/test_perms.py @@ -6,12 +6,13 @@ import pytest @pytest.mark.parametrize('autoperms', ['notest', 'unset', 'true', 'false']) @pytest.mark.usefixtures('ds1_copy') -def test_perms(runner, yadm_y, paths, ds1, autoperms): +def test_perms(runner, yadm_cmd, paths, ds1, autoperms): """Test perms""" # set the value of auto-perms if autoperms != 'notest': if autoperms != 'unset': - os.system(' '.join(yadm_y('config', 'yadm.auto-perms', autoperms))) + os.system(' '.join( + yadm_cmd('config', 'yadm.auto-perms', autoperms))) # privatepaths will hold all paths that should become secured privatepaths = [paths.work.join('.ssh'), paths.work.join('.gnupg')] @@ -38,7 +39,7 @@ def test_perms(runner, yadm_y, paths, ds1, autoperms): cmd = 'perms' if autoperms != 'notest': cmd = 'status' - run = runner(yadm_y(cmd), env={'HOME': paths.work}) + run = runner(yadm_cmd(cmd), env={'HOME': paths.work}) assert run.success assert run.err == '' if cmd == 'perms': @@ -62,15 +63,15 @@ def test_perms(runner, yadm_y, paths, ds1, autoperms): @pytest.mark.parametrize('sshperms', [None, 'true', 'false']) @pytest.mark.parametrize('gpgperms', [None, 'true', 'false']) @pytest.mark.usefixtures('ds1_copy') -def test_perms_control(runner, yadm_y, paths, ds1, sshperms, gpgperms): +def test_perms_control(runner, yadm_cmd, paths, ds1, sshperms, gpgperms): """Test fine control of perms""" # set the value of ssh-perms if sshperms: - os.system(' '.join(yadm_y('config', 'yadm.ssh-perms', sshperms))) + os.system(' '.join(yadm_cmd('config', 'yadm.ssh-perms', sshperms))) # set the value of gpg-perms if gpgperms: - os.system(' '.join(yadm_y('config', 'yadm.gpg-perms', gpgperms))) + os.system(' '.join(yadm_cmd('config', 'yadm.gpg-perms', gpgperms))) # privatepaths will hold all paths that should become secured privatepaths = [paths.work.join('.ssh'), paths.work.join('.gnupg')] @@ -81,7 +82,7 @@ def test_perms_control(runner, yadm_y, paths, ds1, sshperms, gpgperms): assert not oct(private.stat().mode).endswith('00'), ( 'Path started secured') - run = runner(yadm_y('perms'), env={'HOME': paths.work}) + run = runner(yadm_cmd('perms'), env={'HOME': paths.work}) assert run.success assert run.err == '' assert run.out == '' diff --git a/test/test_unit_configure_paths.py b/test/test_unit_configure_paths.py index 332277d..1a6fea9 100644 --- a/test/test_unit_configure_paths.py +++ b/test/test_unit_configure_paths.py @@ -2,19 +2,21 @@ import pytest -ARCHIVE = 'files.gpg' +ARCHIVE = 'archive' BOOTSTRAP = 'bootstrap' CONFIG = 'config' ENCRYPT = 'encrypt' HOME = '/testhome' REPO = 'repo.git' YDIR = '.config/yadm' +YDATA = '.local/share/yadm' @pytest.mark.parametrize( 'override, expect', [ (None, {}), - ('-Y', {}), + ('-Y', {'yadm': 'YADM_DIR'}), + ('--yadm-data', {'data': 'YADM_DATA'}), ('--yadm-repo', {'repo': 'YADM_REPO', 'git': 'GIT_DIR'}), ('--yadm-config', {'config': 'YADM_CONFIG'}), ('--yadm-encrypt', {'encrypt': 'YADM_ENCRYPT'}), @@ -23,6 +25,7 @@ YDIR = '.config/yadm' ], ids=[ 'default', 'override yadm dir', + 'override yadm data', 'override repo', 'override config', 'override encrypt', @@ -36,6 +39,8 @@ def test_config(runner, paths, override, expect): args = [] if override == '-Y': matches = match_map('/' + opath) + if override == '--yadm-data': + matches = match_map(None, '/' + opath) if override: args = [override, '/' + opath] @@ -49,18 +54,20 @@ def test_config(runner, paths, override, expect): run_test(runner, paths, args, matches.values(), 0) -def match_map(yadm_dir=None): +def match_map(yadm_dir=None, yadm_data=None): """Create a dictionary of matches, relative to yadm_dir""" if not yadm_dir: yadm_dir = '/'.join([HOME, YDIR]) + if not yadm_data: + yadm_data = '/'.join([HOME, YDATA]) return { 'yadm': f'YADM_DIR="{yadm_dir}"', - 'repo': f'YADM_REPO="{yadm_dir}/{REPO}"', + 'repo': f'YADM_REPO="{yadm_data}/{REPO}"', 'config': f'YADM_CONFIG="{yadm_dir}/{CONFIG}"', 'encrypt': f'YADM_ENCRYPT="{yadm_dir}/{ENCRYPT}"', - 'archive': f'YADM_ARCHIVE="{yadm_dir}/{ARCHIVE}"', + 'archive': f'YADM_ARCHIVE="{yadm_data}/{ARCHIVE}"', 'bootstrap': f'YADM_BOOTSTRAP="{yadm_dir}/{BOOTSTRAP}"', - 'git': f'GIT_DIR="{yadm_dir}/{REPO}"', + 'git': f'GIT_DIR="{yadm_data}/{REPO}"', } @@ -70,12 +77,15 @@ def run_test(runner, paths, args, expected_matches, expected_code=0): script = f""" YADM_TEST=1 HOME="{HOME}" source {paths.pgm} process_global_args {argstring} - HOME="{HOME}" set_yadm_dir + XDG_CONFIG_HOME= + XDG_DATA_HOME= + HOME="{HOME}" set_yadm_dirs configure_paths declare -p | grep -E '(YADM|GIT)_' """ run = runner(command=['bash'], inp=script) assert run.code == expected_code - assert run.err == '' + assert run.success == (run.code == 0) + assert (run.err if run.success else run.out) == '' for match in expected_matches: - assert match in run.out + assert match in run.out if run.success else run.err diff --git a/test/test_unit_copy_perms.py b/test/test_unit_copy_perms.py new file mode 100644 index 0000000..3c79768 --- /dev/null +++ b/test/test_unit_copy_perms.py @@ -0,0 +1,53 @@ +"""Unit tests: copy_perms""" +import os +import pytest + +OCTAL = '7654' +NON_OCTAL = '9876' + + +@pytest.mark.parametrize( + 'stat_broken', [True, False], ids=['normal', 'stat broken']) +def test_copy_perms(runner, yadm, tmpdir, stat_broken): + """Test function copy_perms""" + src_mode = 0o754 + dst_mode = 0o644 + source = tmpdir.join('source') + source.write('test', ensure=True) + source.chmod(src_mode) + + dest = tmpdir.join('dest') + dest.write('test', ensure=True) + dest.chmod(dst_mode) + + override_stat = '' + if stat_broken: + override_stat = 'function stat() { echo broken; }' + script = f""" + YADM_TEST=1 source {yadm} + {override_stat} + copy_perms "{source}" "{dest}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert run.out == '' + expected = dst_mode if stat_broken else src_mode + assert oct(os.stat(dest).st_mode)[-3:] == oct(expected)[-3:] + + +@pytest.mark.parametrize( + 'stat_output', [OCTAL, NON_OCTAL], ids=['octal', 'non-octal']) +def test_get_mode(runner, yadm, stat_output): + """Test function get_mode""" + script = f""" + YADM_TEST=1 source {yadm} + function stat() {{ echo {stat_output}; }} + mode=$(get_mode abc) + echo "MODE:$mode" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + expected = OCTAL if stat_output == OCTAL else "" + assert f'MODE:{expected}\n' in run.out diff --git a/test/test_unit_encryption.py b/test/test_unit_encryption.py new file mode 100644 index 0000000..ab03c62 --- /dev/null +++ b/test/test_unit_encryption.py @@ -0,0 +1,135 @@ +"""Unit tests: encryption functions""" + +import pytest + + +@pytest.mark.parametrize('condition', ['default', 'override']) +def test_get_cipher(runner, paths, condition): + """Test _get_cipher()""" + + if condition == 'override': + paths.config.write('[yadm]\n\tcipher = override-cipher') + + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_DIR="{paths.yadm}" + set_yadm_dirs + configure_paths + _get_cipher test-archive + echo "output_archive:$output_archive" + echo "yadm_cipher:$yadm_cipher" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert 'output_archive:test-archive' in run.out + if condition == 'override': + assert 'yadm_cipher:override-cipher' in run.out + else: + assert 'yadm_cipher:gpg' in run.out + + +@pytest.mark.parametrize('cipher', ['gpg', 'openssl', 'bad']) +@pytest.mark.parametrize('mode', ['_encrypt_to', '_decrypt_from']) +def test_encrypt_decrypt(runner, paths, cipher, mode): + """Test _encrypt_to() & _decrypt_from""" + + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_DIR="{paths.yadm}" + set_yadm_dirs + configure_paths + function mock_openssl() {{ echo openssl $*; }} + function mock_gpg() {{ echo gpg $*; }} + function _get_cipher() {{ + output_archive="$1" + yadm_cipher="{cipher}" + }} + OPENSSL_PROGRAM=mock_openssl + GPG_PROGRAM=mock_gpg + {mode} {paths.archive} + """ + run = runner(command=['bash'], inp=script) + + if cipher != 'bad': + assert run.success + assert run.out.startswith(cipher) + assert str(paths.archive) in run.out + assert run.err == '' + else: + assert run.failure + assert 'Unknown cipher' in run.err + + +@pytest.mark.parametrize('condition', ['default', 'override']) +def test_get_openssl_ciphername(runner, paths, condition): + """Test _get_openssl_ciphername()""" + + if condition == 'override': + paths.config.write('[yadm]\n\topenssl-ciphername = override-cipher') + + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_DIR="{paths.yadm}" + set_yadm_dirs + configure_paths + result=$(_get_openssl_ciphername) + echo "result:$result" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + if condition == 'override': + assert run.out.strip() == 'result:override-cipher' + else: + assert run.out.strip() == 'result:aes-256-cbc' + + +@pytest.mark.parametrize('condition', ['old', 'not-old']) +def test_set_openssl_options(runner, paths, condition): + """Test _set_openssl_options()""" + + if condition == 'old': + paths.config.write('[yadm]\n\topenssl-old = true') + + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_DIR="{paths.yadm}" + set_yadm_dirs + configure_paths + function _get_openssl_ciphername() {{ echo "testcipher"; }} + _set_openssl_options + echo "result:${{OPENSSL_OPTS[@]}}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + if condition == 'old': + assert '-testcipher -salt -md md5' in run.out + else: + assert '-testcipher -salt -pbkdf2 -iter 100000 -md sha512' in run.out + + +@pytest.mark.parametrize('recipient', ['ASK', 'present', '']) +def test_set_gpg_options(runner, paths, recipient): + """Test _set_gpg_options()""" + + paths.config.write(f'[yadm]\n\tgpg-recipient = {recipient}') + + script = f""" + YADM_TEST=1 source {paths.pgm} + YADM_DIR="{paths.yadm}" + set_yadm_dirs + configure_paths + _set_gpg_options + echo "result:${{GPG_OPTS[@]}}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + if recipient == 'ASK': + assert run.out.strip() == 'result:--no-default-recipient -e' + elif recipient != '': + assert run.out.strip() == f'result:-e -r {recipient}' + else: + assert run.out.strip() == 'result:-c' diff --git a/test/test_unit_issue_legacy_path_warning.py b/test/test_unit_issue_legacy_path_warning.py index 3f5cd6f..e43228b 100644 --- a/test/test_unit_issue_legacy_path_warning.py +++ b/test/test_unit_issue_legacy_path_warning.py @@ -6,36 +6,36 @@ import pytest 'legacy_path', [ None, 'repo.git', - 'config', - 'encrypt', 'files.gpg', - 'bootstrap', - 'hooks/pre_command', - 'hooks/post_command', ], ) @pytest.mark.parametrize( + 'override', [True, False], ids=['override', 'no-override']) +@pytest.mark.parametrize( 'upgrade', [True, False], ids=['upgrade', 'no-upgrade']) -def test_legacy_warning(tmpdir, runner, yadm, upgrade, legacy_path): +def test_legacy_warning(tmpdir, runner, yadm, upgrade, override, legacy_path): """Use issue_legacy_path_warning""" home = tmpdir.mkdir('home') if legacy_path: - home.mkdir(f'.yadm').ensure(legacy_path) + home.ensure(f'.config/yadm/{str(legacy_path)}') + override = 'YADM_OVERRIDE_REPO=override' if override else '' main_args = 'MAIN_ARGS=("upgrade")' if upgrade else '' script = f""" + XDG_CONFIG_HOME= + XDG_DATA_HOME= HOME={home} YADM_TEST=1 source {yadm} {main_args} + {override} + set_yadm_dirs issue_legacy_path_warning - echo "LWI:$LEGACY_WARNING_ISSUED" """ run = runner(command=['bash'], inp=script) assert run.success - assert run.err == '' - if legacy_path and not upgrade: - assert 'Legacy configuration paths have been detected' in run.out - assert 'LWI:1' in run.out + assert run.out == '' + if legacy_path and (not upgrade) and (not override): + assert 'Legacy paths have been detected' in run.err else: - assert run.out.rstrip() == 'LWI:0' + assert 'Legacy paths have been detected' not in run.err diff --git a/test/test_unit_record_score.py b/test/test_unit_record_score.py index 525c967..78596e1 100644 --- a/test/test_unit_record_score.py +++ b/test/test_unit_record_score.py @@ -112,3 +112,30 @@ def test_existing_template(runner, yadm): assert 'SCORES:1\n' in run.out assert 'TARGETS:testtgt\n' in run.out assert 'SOURCES:\n' in run.out + + +def test_config_first(runner, yadm): + """Verify YADM_CONFIG is always processed first""" + + config = 'yadm_config_file' + script = f""" + YADM_TEST=1 source {yadm} + {INIT_VARS} + YADM_CONFIG={config} + record_score "1" "tgt_before" "src_before" + record_template "tgt_tmp" "cmd_tmp" "src_tmp" + record_score "2" "{config}" "src_config" + record_score "3" "tgt_after" "src_after" + {REPORT_RESULTS} + echo "CMD_VALUE:${{alt_template_cmds[@]}}" + echo "CMD_INDEX:${{!alt_template_cmds[@]}}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert 'SIZE:3\n' in run.out + assert 'SCORES:2 1 3\n' in run.out + assert f'TARGETS:{config} tgt_before tgt_tmp tgt_after\n' in run.out + assert 'SOURCES:src_config src_before src_tmp src_after\n' in run.out + assert 'CMD_VALUE:cmd_tmp\n' in run.out + assert 'CMD_INDEX:2\n' in run.out diff --git a/test/test_unit_report_invalid_alts.py b/test/test_unit_report_invalid_alts.py index 7aa93bb..8730d61 100644 --- a/test/test_unit_report_invalid_alts.py +++ b/test/test_unit_report_invalid_alts.py @@ -2,38 +2,29 @@ import pytest -@pytest.mark.parametrize( - 'condition', [ - 'compat', - 'previous-message', - 'invalid-alts', - 'no-invalid-alts', - ]) -def test_report_invalid_alts(runner, yadm, condition): +@pytest.mark.parametrize('valid', [True, False], ids=['valid', 'no_valid']) +@pytest.mark.parametrize('previous', [True, False], ids=['prev', 'no_prev']) +def test_report_invalid_alts(runner, yadm, valid, previous): """Use report_invalid_alts""" - compat = '' - previous = '' + lwi = '' alts = 'INVALID_ALT=()' - if condition == 'compat': - compat = 'YADM_COMPATIBILITY=1' - if condition == 'previous-message': - previous = 'LEGACY_WARNING_ISSUED=1' - if condition == 'invalid-alts': + if previous: + lwi = 'LEGACY_WARNING_ISSUED=1' + if not valid: alts = 'INVALID_ALT=("file##invalid")' script = f""" YADM_TEST=1 source {yadm} - {compat} - {previous} + {lwi} {alts} report_invalid_alts """ run = runner(command=['bash'], inp=script) assert run.success - assert run.err == '' - if condition == 'invalid-alts': - assert 'WARNING' in run.out - assert 'file##invalid' in run.out + assert run.out == '' + if not valid and not previous: + assert 'WARNING' in run.err + assert 'file##invalid' in run.err else: - assert run.out == '' + assert run.err == '' diff --git a/test/test_unit_score_file.py b/test/test_unit_score_file.py index 679229f..450c154 100644 --- a/test/test_unit_score_file.py +++ b/test/test_unit_score_file.py @@ -196,6 +196,28 @@ def test_score_values( assert run.out == expected +@pytest.mark.parametrize('ext', [None, 'e', 'extension']) +def test_extensions(runner, yadm, ext): + """Verify extensions do not effect scores""" + local_user = 'testuser' + filename = f'filename##u.{local_user}' + if ext: + filename += f',{ext}.xyz' + expected = '' + script = f""" + YADM_TEST=1 source {yadm} + score=0 + local_user={local_user} + score_file "{filename}" + echo "$score" + """ + expected = f'{1000 + CONDITION["user"]["modifier"]}\n' + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert run.out == expected + + def test_score_values_templates(runner, yadm): """Test score results""" local_class = 'testclass' @@ -260,19 +282,3 @@ def test_template_recording(runner, yadm, cmd_generated): assert run.success assert run.err == '' assert run.out.rstrip() == expected - - -def test_invalid(runner, yadm): - """Verify invalid alternates are noted in INVALID_ALT""" - - invalid_file = "file##invalid" - - script = f""" - YADM_TEST=1 source {yadm} - score_file "{invalid_file}" - echo "INVALID:${{INVALID_ALT[@]}}" - """ - run = runner(command=['bash'], inp=script) - assert run.success - assert run.err == '' - assert run.out.rstrip() == f'INVALID:{invalid_file}' diff --git a/test/test_unit_set_local_alt_values.py b/test/test_unit_set_local_alt_values.py index d3d2447..e49d055 100644 --- a/test/test_unit_set_local_alt_values.py +++ b/test/test_unit_set_local_alt_values.py @@ -26,7 +26,7 @@ def test_set_local_alt_values( script = f""" YADM_TEST=1 source {yadm} && set_operating_system && - YADM_DIR={paths.yadm} configure_paths && + YADM_DIR={paths.yadm} YADM_DATA={paths.data} configure_paths && set_local_alt_values echo "class='$local_class'" echo "os='$local_system'" @@ -52,12 +52,12 @@ def test_set_local_alt_values( assert f"os='{tst_sys}'" in run.out if override == 'hostname': - assert f"host='override'" in run.out + assert "host='override'" in run.out else: assert f"host='{tst_host}'" in run.out if override == 'user': - assert f"user='override'" in run.out + assert "user='override'" in run.out else: assert f"user='{tst_user}'" in run.out @@ -67,6 +67,7 @@ def test_distro(runner, yadm): script = f""" YADM_TEST=1 source {yadm} + function config() {{ echo "$1"; }} function query_distro() {{ echo "testdistro"; }} set_local_alt_values echo "distro='$local_distro'" diff --git a/test/test_unit_set_yadm_dir.py b/test/test_unit_set_yadm_dir.py index 65459f8..32af8bf 100644 --- a/test/test_unit_set_yadm_dir.py +++ b/test/test_unit_set_yadm_dir.py @@ -1,35 +1,48 @@ -"""Unit tests: set_yadm_dir""" +"""Unit tests: set_yadm_dirs""" import pytest @pytest.mark.parametrize( - 'condition', - ['basic', 'override', 'xdg_config_home', 'legacy'], + 'condition', [ + 'basic', + 'override', + 'override_data', + 'xdg_config_home', + 'xdg_data_home' + ], ) -def test_set_yadm_dir(runner, yadm, condition): - """Test set_yadm_dir""" +def test_set_yadm_dirs(runner, yadm, condition): + """Test set_yadm_dirs""" setup = '' if condition == 'override': setup = 'YADM_DIR=/override' + elif condition == 'override_data': + setup = 'YADM_DATA=/override' elif condition == 'xdg_config_home': setup = 'XDG_CONFIG_HOME=/xdg' - elif condition == 'legacy': - setup = 'YADM_COMPATIBILITY=1' + elif condition == 'xdg_data_home': + setup = 'XDG_DATA_HOME=/xdg' script = f""" HOME=/testhome YADM_TEST=1 source {yadm} + XDG_CONFIG_HOME= + XDG_DATA_HOME= {setup} - set_yadm_dir - echo "$YADM_DIR" + set_yadm_dirs + echo "YADM_DIR=$YADM_DIR" + echo "YADM_DATA=$YADM_DATA" """ run = runner(command=['bash'], inp=script) assert run.success assert run.err == '' if condition == 'basic': - assert run.out.rstrip() == '/testhome/.config/yadm' + assert 'YADM_DIR=/testhome/.config/yadm' in run.out + assert 'YADM_DATA=/testhome/.local/share/yadm' in run.out elif condition == 'override': - assert run.out.rstrip() == '/override' + assert 'YADM_DIR=/override' in run.out + elif condition == 'override_data': + assert 'YADM_DATA=/override' in run.out elif condition == 'xdg_config_home': - assert run.out.rstrip() == '/xdg/yadm' - elif condition == 'legacy': - assert run.out.rstrip() == '/testhome/.yadm' + assert 'YADM_DIR=/xdg/yadm' in run.out + elif condition == 'xdg_data_home': + assert 'YADM_DATA=/xdg/yadm' in run.out diff --git a/test/test_unit_template_default.py b/test/test_unit_template_default.py index 42464b8..639cb29 100644 --- a/test/test_unit_template_default.py +++ b/test/test_unit_template_default.py @@ -1,4 +1,7 @@ """Unit tests: template_default""" +import os + +FILE_MODE = 0o754 # these values are also testing the handling of bizarre characters LOCAL_CLASS = "default_Test+@-!^Class" @@ -85,12 +88,43 @@ Included section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again) end of template ''' +INCLUDE_BASIC = 'basic\n' +INCLUDE_VARIABLES = '''\ +included <{{ yadm.class }}> file + +empty line above +''' +INCLUDE_NESTED = 'no newline at the end' + +TEMPLATE_INCLUDE = '''\ +The first line +{% include empty %} +An empty file removes the line above +{%include basic%} +{% include "./variables.{{ yadm.os }}" %} +{% include dir/nested %} +Include basic again: +{% include basic %} +''' +EXPECTED_INCLUDE = f'''\ +The first line +An empty file removes the line above +basic +included <{LOCAL_CLASS}> file + +empty line above +no newline at the end +Include basic again: +basic +''' + def test_template_default(runner, yadm, tmpdir): """Test template_default""" input_file = tmpdir.join('input') input_file.write(TEMPLATE, ensure=True) + input_file.chmod(FILE_MODE) output_file = tmpdir.join('output') script = f""" @@ -107,6 +141,7 @@ def test_template_default(runner, yadm, tmpdir): assert run.success assert run.err == '' assert output_file.read() == EXPECTED + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode def test_source(runner, yadm, tmpdir): @@ -114,6 +149,7 @@ def test_source(runner, yadm, tmpdir): input_file = tmpdir.join('input') input_file.write('{{yadm.source}}', ensure=True) + input_file.chmod(FILE_MODE) output_file = tmpdir.join('output') script = f""" @@ -125,3 +161,38 @@ def test_source(runner, yadm, tmpdir): assert run.success assert run.err == '' assert output_file.read().strip() == str(input_file) + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode + + +def test_include(runner, yadm, tmpdir): + """Test include""" + + empty_file = tmpdir.join('empty') + empty_file.write('', ensure=True) + + basic_file = tmpdir.join('basic') + basic_file.write(INCLUDE_BASIC) + + variables_file = tmpdir.join(f'variables.{LOCAL_SYSTEM}') + variables_file.write(INCLUDE_VARIABLES) + + nested_file = tmpdir.join('dir').join('nested') + nested_file.write(INCLUDE_NESTED, ensure=True) + + input_file = tmpdir.join('input') + input_file.write(TEMPLATE_INCLUDE) + input_file.chmod(FILE_MODE) + output_file = tmpdir.join('output') + + script = f""" + YADM_TEST=1 source {yadm} + set_awk + local_class="{LOCAL_CLASS}" + local_system="{LOCAL_SYSTEM}" + template_default "{input_file}" "{output_file}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert output_file.read() == EXPECTED_INCLUDE + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode diff --git a/test/test_unit_template_esh.py b/test/test_unit_template_esh.py new file mode 100644 index 0000000..e975152 --- /dev/null +++ b/test/test_unit_template_esh.py @@ -0,0 +1,121 @@ +"""Unit tests: template_esh""" +import os + +FILE_MODE = 0o754 + +LOCAL_CLASS = "esh_Test+@-!^Class" +LOCAL_SYSTEM = "esh_Test+@-!^System" +LOCAL_HOST = "esh_Test+@-!^Host" +LOCAL_USER = "esh_Test+@-!^User" +LOCAL_DISTRO = "esh_Test+@-!^Distro" +TEMPLATE = f''' +start of template +esh class = ><%=$YADM_CLASS%>< +esh os = ><%=$YADM_OS%>< +esh host = ><%=$YADM_HOSTNAME%>< +esh user = ><%=$YADM_USER%>< +esh distro = ><%=$YADM_DISTRO%>< +<% if [ "$YADM_CLASS" = "wrongclass1" ]; then -%> +wrong class 1 +<% fi -%> +<% if [ "$YADM_CLASS" = "{LOCAL_CLASS}" ]; then -%> +Included section for class = <%=$YADM_CLASS%> (<%=$YADM_CLASS%> repeated) +<% fi -%> +<% if [ "$YADM_CLASS" = "wrongclass2" ]; then -%> +wrong class 2 +<% fi -%> +<% if [ "$YADM_OS" = "wrongos1" ]; then -%> +wrong os 1 +<% fi -%> +<% if [ "$YADM_OS" = "{LOCAL_SYSTEM}" ]; then -%> +Included section for os = <%=$YADM_OS%> (<%=$YADM_OS%> repeated) +<% fi -%> +<% if [ "$YADM_OS" = "wrongos2" ]; then -%> +wrong os 2 +<% fi -%> +<% if [ "$YADM_HOSTNAME" = "wronghost1" ]; then -%> +wrong host 1 +<% fi -%> +<% if [ "$YADM_HOSTNAME" = "{LOCAL_HOST}" ]; then -%> +Included section for host = <%=$YADM_HOSTNAME%> (<%=$YADM_HOSTNAME%> again) +<% fi -%> +<% if [ "$YADM_HOSTNAME" = "wronghost2" ]; then -%> +wrong host 2 +<% fi -%> +<% if [ "$YADM_USER" = "wronguser1" ]; then -%> +wrong user 1 +<% fi -%> +<% if [ "$YADM_USER" = "{LOCAL_USER}" ]; then -%> +Included section for user = <%=$YADM_USER%> (<%=$YADM_USER%> repeated) +<% fi -%> +<% if [ "$YADM_USER" = "wronguser2" ]; then -%> +wrong user 2 +<% fi -%> +<% if [ "$YADM_DISTRO" = "wrongdistro1" ]; then -%> +wrong distro 1 +<% fi -%> +<% if [ "$YADM_DISTRO" = "{LOCAL_DISTRO}" ]; then -%> +Included section for distro = <%=$YADM_DISTRO%> (<%=$YADM_DISTRO%> again) +<% fi -%> +<% if [ "$YADM_DISTRO" = "wrongdistro2" ]; then -%> +wrong distro 2 +<% fi -%> +end of template +''' +EXPECTED = f''' +start of template +esh class = >{LOCAL_CLASS}< +esh os = >{LOCAL_SYSTEM}< +esh host = >{LOCAL_HOST}< +esh user = >{LOCAL_USER}< +esh distro = >{LOCAL_DISTRO}< +Included section for class = {LOCAL_CLASS} ({LOCAL_CLASS} repeated) +Included section for os = {LOCAL_SYSTEM} ({LOCAL_SYSTEM} repeated) +Included section for host = {LOCAL_HOST} ({LOCAL_HOST} again) +Included section for user = {LOCAL_USER} ({LOCAL_USER} repeated) +Included section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again) +end of template +''' + + +def test_template_esh(runner, yadm, tmpdir): + """Test processing by esh""" + + input_file = tmpdir.join('input') + input_file.write(TEMPLATE, ensure=True) + input_file.chmod(FILE_MODE) + output_file = tmpdir.join('output') + + script = f""" + YADM_TEST=1 source {yadm} + local_class="{LOCAL_CLASS}" + local_system="{LOCAL_SYSTEM}" + local_host="{LOCAL_HOST}" + local_user="{LOCAL_USER}" + local_distro="{LOCAL_DISTRO}" + template_esh "{input_file}" "{output_file}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert output_file.read().strip() == str(EXPECTED).strip() + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode + + +def test_source(runner, yadm, tmpdir): + """Test YADM_SOURCE""" + + input_file = tmpdir.join('input') + input_file.write('<%= $YADM_SOURCE %>', ensure=True) + input_file.chmod(FILE_MODE) + output_file = tmpdir.join('output') + + script = f""" + YADM_TEST=1 source {yadm} + template_esh "{input_file}" "{output_file}" + """ + run = runner(command=['bash'], inp=script) + assert run.success + assert run.err == '' + assert output_file.read().strip() == str(input_file) + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode diff --git a/test/test_unit_template_j2.py b/test/test_unit_template_j2.py index 85c6822..f81f4c6 100644 --- a/test/test_unit_template_j2.py +++ b/test/test_unit_template_j2.py @@ -1,6 +1,9 @@ """Unit tests: template_j2cli & template_envtpl""" +import os import pytest +FILE_MODE = 0o754 + LOCAL_CLASS = "j2_Test+@-!^Class" LOCAL_SYSTEM = "j2_Test+@-!^System" LOCAL_HOST = "j2_Test+@-!^Host" @@ -82,6 +85,7 @@ def test_template_j2(runner, yadm, tmpdir, processor): input_file = tmpdir.join('input') input_file.write(TEMPLATE, ensure=True) + input_file.chmod(FILE_MODE) output_file = tmpdir.join('output') script = f""" @@ -97,6 +101,7 @@ def test_template_j2(runner, yadm, tmpdir, processor): assert run.success assert run.err == '' assert output_file.read() == EXPECTED + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode @pytest.mark.parametrize('processor', ('j2cli', 'envtpl')) @@ -105,6 +110,7 @@ def test_source(runner, yadm, tmpdir, processor): input_file = tmpdir.join('input') input_file.write('{{YADM_SOURCE}}', ensure=True) + input_file.chmod(FILE_MODE) output_file = tmpdir.join('output') script = f""" @@ -115,3 +121,4 @@ def test_source(runner, yadm, tmpdir, processor): assert run.success assert run.err == '' assert output_file.read().strip() == str(input_file) + assert os.stat(output_file).st_mode == os.stat(input_file).st_mode diff --git a/test/test_unit_upgrade.py b/test/test_unit_upgrade.py index 73f4cac..3463740 100644 --- a/test/test_unit_upgrade.py +++ b/test/test_unit_upgrade.py @@ -1,53 +1,39 @@ """Unit tests: upgrade""" import pytest -LEGACY_PATHS = [ - 'config', - 'encrypt', - 'files.gpg', - 'bootstrap', - 'hooks/pre_command', - 'hooks/post_command', -] -# used: -# YADM_COMPATIBILITY -# YADM_DIR -# YADM_LEGACY_DIR -# GIT_PROGRAM -@pytest.mark.parametrize('condition', ['compat', 'equal', 'existing_repo']) +@pytest.mark.parametrize('condition', ['override', 'equal', 'existing_repo']) def test_upgrade_errors(tmpdir, runner, yadm, condition): """Test upgrade() error conditions""" - compatibility = 'YADM_COMPATIBILITY=1' if condition == 'compat' else '' - home = tmpdir.mkdir('home') yadm_dir = home.join('.config/yadm') - legacy_dir = home.join('.yadm') + yadm_data = home.join('.local/share/yadm') + override = '' + if condition == 'override': + override = 'override' if condition == 'equal': - legacy_dir = yadm_dir + yadm_data = yadm_dir if condition == 'existing_repo': yadm_dir.ensure_dir('repo.git') - legacy_dir.ensure_dir('repo.git') + yadm_data.ensure_dir('repo.git') script = f""" YADM_TEST=1 source {yadm} - {compatibility} YADM_DIR="{yadm_dir}" - YADM_REPO="{yadm_dir}/repo.git" - YADM_LEGACY_DIR="{legacy_dir}" + YADM_DATA="{yadm_data}" + YADM_REPO="{yadm_data}/repo.git" + YADM_LEGACY_ARCHIVE="files.gpg" + YADM_OVERRIDE_REPO="{override}" upgrade """ run = runner(command=['bash'], inp=script) assert run.failure - assert run.err == '' - assert 'Unable to upgrade' in run.out - if condition == 'compat': - assert 'YADM_COMPATIBILITY' in run.out - if condition == 'equal': - assert 'has been resolved as' in run.out - if condition == 'existing_repo': - assert 'already exists' in run.out + assert 'Unable to upgrade' in run.err + if condition in ['override', 'equal']: + assert 'Paths have been overridden' in run.err + elif condition == 'existing_repo': + assert 'already exists' in run.err @pytest.mark.parametrize( @@ -59,22 +45,29 @@ def test_upgrade(tmpdir, runner, yadm, condition): mock for git. echo will return true, simulating a positive result from "git ls-files". Also echo will report the parameters for "git mv". """ + legacy_paths = ('config', 'encrypt', 'bootstrap', 'hooks/pre_cmd') home = tmpdir.mkdir('home') yadm_dir = home.join('.config/yadm') - legacy_dir = home.join('.yadm') + yadm_data = home.join('.local/share/yadm') + yadm_legacy = home.join('.yadm') if condition != 'no-paths': - legacy_dir.join('repo.git/config').write('test-repo', ensure=True) - for lpath in LEGACY_PATHS: - legacy_dir.join(lpath).write(lpath, ensure=True) + yadm_dir.join('repo.git/config').write('test-repo', ensure=True) + yadm_dir.join('files.gpg').write('files.gpg', ensure=True) + for path in legacy_paths: + yadm_legacy.join(path).write(path, ensure=True) mock_git = "" - if condition in ['tracked', 'submodules']: + if condition != 'no-paths': mock_git = f''' function git() {{ echo "$@" - if [[ "$*" == *.gitmodules* ]]; then - return { '0' if condition == 'submodules' else '1' } + if [[ "$*" = *"submodule status" ]]; then + { 'echo " 1234567 mymodule (1.0)"' + if condition == 'submodules' else ':' } + fi + if [[ "$*" = *ls-files* ]]; then + return { 1 if condition == 'untracked' else 0 } fi return 0 }} @@ -82,9 +75,11 @@ def test_upgrade(tmpdir, runner, yadm, condition): script = f""" YADM_TEST=1 source {yadm} + YADM_LEGACY_DIR="{yadm_legacy}" YADM_DIR="{yadm_dir}" - YADM_REPO="{yadm_dir}/repo.git" - YADM_LEGACY_DIR="{legacy_dir}" + YADM_DATA="{yadm_data}" + YADM_REPO="{yadm_data}/repo.git" + YADM_ARCHIVE="{yadm_data}/archive" GIT_PROGRAM="git" {mock_git} function cd {{ echo "$@";}} @@ -96,25 +91,32 @@ def test_upgrade(tmpdir, runner, yadm, condition): if condition == 'no-paths': assert 'Upgrade is not necessary' in run.out else: - for lpath in LEGACY_PATHS + ['repo.git']: + for (lpath, npath) in [ + ('repo.git', 'repo.git'), ('files.gpg', 'archive')]: + expected = ( + f'Moving {yadm_dir.join(lpath)} ' + f'to {yadm_data.join(npath)}') + assert expected in run.out + for path in legacy_paths: expected = ( - f'Moving {legacy_dir.join(lpath)} ' - f'to {yadm_dir.join(lpath)}') + f'Moving {yadm_legacy.join(path)} ' + f'to {yadm_dir.join(path)}') assert expected in run.out if condition == 'untracked': - assert 'test-repo' in yadm_dir.join('repo.git/config').read() - for lpath in LEGACY_PATHS: - assert lpath in yadm_dir.join(lpath).read() + assert 'test-repo' in yadm_data.join('repo.git/config').read() + assert 'files.gpg' in yadm_data.join('archive').read() + for path in legacy_paths: + assert path in yadm_dir.join(path).read() elif condition in ['tracked', 'submodules']: - for lpath in LEGACY_PATHS: - expected = ( - f'mv {legacy_dir.join(lpath)} ' - f'{yadm_dir.join(lpath)}') - assert expected in run.out + expected = ( + f'mv {yadm_dir.join("files.gpg")} ' + f'{yadm_data.join("archive")}') + assert expected in run.out assert 'files tracked by yadm have been renamed' in run.out if condition == 'submodules': - assert 'submodule deinit -f .' in run.out - assert 'submodule update --init --recursive' in run.out + assert 'submodule deinit -- mymodule' in run.out + assert 'submodule update --init --recursive -- mymodule' \ + in run.out else: - assert 'submodule deinit -f .' not in run.out + assert 'submodule deinit' not in run.out assert 'submodule update --init --recursive' not in run.out diff --git a/test/test_unit_x_program.py b/test/test_unit_x_program.py index 3233a3d..8302f3c 100644 --- a/test/test_unit_x_program.py +++ b/test/test_unit_x_program.py @@ -16,24 +16,24 @@ import pytest ]) @pytest.mark.parametrize('program', ['git', 'gpg']) def test_x_program( - runner, yadm_y, paths, program, executable, success, value, match): + runner, yadm_cmd, paths, program, executable, success, value, match): """Set yadm.X-program, and test result of require_X""" # set configuration if executable: - os.system(' '.join(yadm_y( + os.system(' '.join(yadm_cmd( 'config', f'yadm.{program}-program', executable))) # test require_[git,gpg] script = f""" YADM_TEST=1 source {paths.pgm} - YADM_CONFIG="{paths.config}" + YADM_OVERRIDE_CONFIG="{paths.config}" + configure_paths require_{program} echo ${program.upper()}_PROGRAM """ run = runner(command=['bash'], inp=script) assert run.success == success - assert run.err == '' # [GIT,GPG]_PROGRAM set correctly if value == 'program': @@ -43,4 +43,6 @@ def test_x_program( # error reported about bad config if match: - assert match in run.out + assert match in run.err + else: + assert run.err == '' diff --git a/test/test_upgrade.py b/test/test_upgrade.py new file mode 100644 index 0000000..1ccf075 --- /dev/null +++ b/test/test_upgrade.py @@ -0,0 +1,129 @@ +"""Test upgrade""" + +import os +import pytest + + +@pytest.mark.parametrize( + 'versions', [ + ('1.12.0', '2.5.0'), + ('1.12.0',), + ('2.5.0',), + ], ids=[ + '1.12.0 -> 2.5.0 -> latest', + '1.12.0 -> latest', + '2.5.0 -> latest', + ]) +@pytest.mark.parametrize( + 'submodule', [False, True], + ids=['no submodule', 'with submodules']) +def test_upgrade(tmpdir, runner, versions, submodule): + """Upgrade tests""" + # pylint: disable=too-many-statements + home = tmpdir.mkdir('HOME') + env = {'HOME': str(home)} + + if submodule: + ext_repo = tmpdir.mkdir('ext_repo') + ext_repo.join('afile').write('some data') + + for cmd in (('init',), ('add', 'afile'), ('commit', '-m', 'test')): + run = runner(['git', '-C', str(ext_repo), *cmd]) + assert run.success + + os.environ.pop('XDG_CONFIG_HOME', None) + os.environ.pop('XDG_DATA_HOME', None) + + def run_version(version, *args, check_stderr=True): + yadm = 'yadm-%s' % version if version else '/yadm/yadm' + run = runner([yadm, *args], shell=True, cwd=str(home), env=env) + assert run.success + if check_stderr: + assert run.err == '' + return run + + # Initialize the repo with the first version + first = versions[0] + run_version(first, 'init') + + home.join('file').write('some data') + run_version(first, 'add', 'file') + run_version(first, 'commit', '-m', '"First commit"') + + if submodule: + # When upgrading via 2.5.0 we can't have a submodule that's been added + # after being cloned as 2.5.0 fails the upgrade in that case. + can_upgraded_cloned_submodule = '2.5.0' not in versions[1:] + if can_upgraded_cloned_submodule: + # Check out a repo and then add it as a submodule + run = runner(['git', '-C', str(home), 'clone', str(ext_repo), 'b']) + assert run.success + run_version(first, 'submodule', 'add', str(ext_repo), 'b') + + # Add submodule without first checking it out + run_version(first, 'submodule', 'add', str(ext_repo), 'a', + check_stderr=False) + run_version(first, 'submodule', 'add', str(ext_repo), 'c', + check_stderr=False) + + run_version(first, 'commit', '-m', '"Add submodules"') + + for path in ('.yadm', '.config/yadm'): + yadm_dir = home.join(path) + if yadm_dir.exists(): + break + + yadm_dir.join('bootstrap').write('init stuff') + run_version(first, 'add', yadm_dir.join('bootstrap')) + run_version(first, 'commit', '-m', 'bootstrap') + + yadm_dir.join('encrypt').write('secret') + + hooks_dir = yadm_dir.mkdir('hooks') + hooks_dir.join('pre_status').write('status') + hooks_dir.join('post_commit').write('commit') + + run_version(first, 'config', 'local.class', 'test') + run_version(first, 'config', 'foo.bar', 'true') + + # Run upgrade with intermediate versions and latest + latest = None + for version in versions[1:] + (latest,): + run = run_version(version, 'upgrade', check_stderr=not submodule) + if submodule: + lines = run.err.splitlines() + if can_upgraded_cloned_submodule: + assert 'Migrating git directory of' in lines[0] + assert str(home.join('b/.git')) in lines[1] + assert str(yadm_dir.join('repo.git/modules/b')) in lines[2] + del lines[:3] + for line in lines: + assert line.startswith('Submodule') + assert 'registered for path' in line + + # Verify result for the final upgrade + run_version(latest, 'status') + + run = run_version(latest, 'show', 'HEAD:file') + assert run.out == 'some data' + + if submodule: + if can_upgraded_cloned_submodule: + assert home.join('b/afile').read() == 'some data' + assert home.join('a/afile').read() == 'some data' + assert home.join('c/afile').read() == 'some data' + + yadm_dir = home.join('.config/yadm') + + assert yadm_dir.join('bootstrap').read() == 'init stuff' + assert yadm_dir.join('encrypt').read() == 'secret' + + hooks_dir = yadm_dir.join('hooks') + assert hooks_dir.join('pre_status').read() == 'status' + assert hooks_dir.join('post_commit').read() == 'commit' + + run = run_version(latest, 'config', 'local.class') + assert run.out.rstrip() == 'test' + + run = run_version(latest, 'config', 'foo.bar') + assert run.out.rstrip() == 'true' diff --git a/test/test_version.py b/test/test_version.py index 023eb82..08bebe1 100644 --- a/test/test_version.py +++ b/test/test_version.py @@ -26,10 +26,11 @@ def test_semantic_version(expected_version): 'does not conform to MAJOR.MINOR.PATCH') +@pytest.mark.parametrize('cmd', ['--version', 'version']) def test_reported_version( - runner, yadm_y, expected_version): + runner, yadm_cmd, cmd, expected_version): """Report correct version""" - run = runner(command=yadm_y('version')) + run = runner(command=yadm_cmd(cmd)) assert run.success assert run.err == '' assert run.out == f'yadm {expected_version}\n' @@ -1,6 +1,6 @@ #!/bin/sh # yadm - Yet Another Dotfiles Manager -# Copyright (C) 2015-2020 Tim Byrne +# Copyright (C) 2015-2021 Tim Byrne # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -20,30 +20,38 @@ if [ -z "$BASH_VERSION" ]; then [ "$YADM_TEST" != 1 ] && exec bash "$0" "$@" fi -VERSION=2.4.0 +VERSION=3.0.2 YADM_WORK="$HOME" YADM_DIR= +YADM_DATA= + YADM_LEGACY_DIR="${HOME}/.yadm" +YADM_LEGACY_ARCHIVE="files.gpg" # these are the default paths relative to YADM_DIR -YADM_REPO="repo.git" YADM_CONFIG="config" YADM_ENCRYPT="encrypt" -YADM_ARCHIVE="files.gpg" YADM_BOOTSTRAP="bootstrap" YADM_HOOKS="hooks" YADM_ALT="alt" +# these are the default paths relative to YADM_DATA +YADM_REPO="repo.git" +YADM_ARCHIVE="archive" + HOOK_COMMAND="" FULL_COMMAND="" GPG_PROGRAM="gpg" +OPENSSL_PROGRAM="openssl" GIT_PROGRAM="git" AWK_PROGRAM=("gawk" "awk") GIT_CRYPT_PROGRAM="git-crypt" +TRANSCRYPT_PROGRAM="transcrypt" J2CLI_PROGRAM="j2" ENVTPL_PROGRAM="envtpl" +ESH_PROGRAM="esh" LSB_RELEASE_PROGRAM="lsb_release" OS_RELEASE="/etc/os-release" @@ -55,6 +63,9 @@ ENCRYPT_INCLUDE_FILES="unparsed" LEGACY_WARNING_ISSUED=0 INVALID_ALT=() +GPG_OPTS=() +OPENSSL_OPTS=() + # flag causing path translations with cygpath USE_CYGPATH=0 @@ -81,18 +92,20 @@ function main() { done FULL_COMMAND="${_fc[*]}" - # create the YADM_DIR if it doesn't exist yet - [ -d "$YADM_DIR" ] || mkdir -p "$YADM_DIR" + # create the YADM_DIR & YADM_DATA if they doesn't exist yet + [ -d "$YADM_DIR" ] || mkdir -p "$YADM_DIR" + [ -d "$YADM_DATA" ] || mkdir -p "$YADM_DATA" # parse command line arguments local retval=0 - internal_commands="^(alt|bootstrap|clean|clone|config|decrypt|encrypt|enter|git-crypt|help|init|introspect|list|perms|upgrade|version)$" + internal_commands="^(alt|bootstrap|clean|clone|config|decrypt|encrypt|enter|git-crypt|help|--help|init|introspect|list|perms|transcrypt|upgrade|version|--version)$" if [ -z "$*" ] ; then # no argumnts will result in help() help elif [[ "$1" =~ $internal_commands ]] ; then # for internal commands, process all of the arguments - YADM_COMMAND="${1/-/_}" + YADM_COMMAND="${1//-/_}" + YADM_COMMAND="${YADM_COMMAND/__/}" YADM_ARGS=() shift @@ -109,7 +122,7 @@ function main() { -d) # used by all commands DEBUG="YES" ;; - -f) # used by init() and clone() + -f) # used by init(), clone() and upgrade() FORCE="YES" ;; -l) # used by decrypt() @@ -160,7 +173,7 @@ function score_file() { conditions="${src#*##}" if [ "${tgt#$YADM_ALT/}" != "${tgt}" ]; then - tgt="${YADM_WORK}/${tgt#$YADM_ALT/}" + tgt="${YADM_BASE}/${tgt#$YADM_ALT/}" fi score=0 @@ -169,6 +182,10 @@ function score_file() { label=${field%%.*} value=${field#*.} [ "$field" = "$label" ] && value="" # when .value is omitted + # extension isn't a condition and doesn't affect the score + if [[ "$label" =~ ^(e|extension)$ ]]; then + continue + fi score=$((score + 1000)) # default condition if [[ "$label" =~ ^(default)$ ]]; then @@ -222,7 +239,9 @@ function score_file() { return 0 # unsupported values else - INVALID_ALT+=("$src") + if [[ "${src##*/}" =~ .\#\#. ]]; then + INVALID_ALT+=("$src") + fi score=0 return fi @@ -249,11 +268,28 @@ function record_score() { done # if we don't find an existing index, create one by appending to the array if [ "$index" -eq -1 ]; then - alt_targets+=("$tgt") - # set index to the last index (newly created one) - for index in "${!alt_targets[@]}"; do :; done - # and set its initial score to zero - alt_scores[$index]=0 + # $YADM_CONFIG must be processed first, in case other templates lookup yadm configurations + if [ "$tgt" = "$YADM_CONFIG" ]; then + alt_targets=("$tgt" "${alt_targets[@]}") + alt_sources=("$src" "${alt_sources[@]}") + alt_scores=(0 "${alt_scores[@]}") + index=0 + # increase the index of any existing alt_template_cmds + new_cmds=() + for cmd_index in "${!alt_template_cmds[@]}"; do + new_cmds[$((cmd_index+1))]="${alt_template_cmds[$cmd_index]}" + done + alt_template_cmds=() + for cmd_index in "${!new_cmds[@]}"; do + alt_template_cmds[$cmd_index]="${new_cmds[$cmd_index]}" + done + else + alt_targets+=("$tgt") + # set index to the last index (newly created one) + for index in "${!alt_targets[@]}"; do :; done + # and set its initial score to zero + alt_scores[$index]=0 + fi fi # record nothing if a template command is registered for this file @@ -298,6 +334,8 @@ function choose_template_cmd() { if [ "$kind" = "default" ] || [ "$kind" = "" ] && awk_available; then echo "template_default" + elif [ "$kind" = "esh" ] && esh_available; then + echo "template_esh" elif [ "$kind" = "j2cli" ] || [ "$kind" = "j2" ] && j2cli_available; then echo "template_j2cli" elif [ "$kind" = "envtpl" ] || [ "$kind" = "j2" ] && envtpl_available; then @@ -327,13 +365,18 @@ BEGIN { c["user"] = user c["distro"] = distro c["source"] = source - vld = conditions() ifs = "^{%" blank "*if" els = "^{%" blank "*else" blank "*%}$" end = "^{%" blank "*endif" blank "*%}$" skp = "^{%" blank "*(if|else|endif)" + vld = conditions() + inc_start = "^{%" blank "*include" blank "+\"?" + inc_end = "\"?" blank "*%}$" + inc = inc_start ".+" inc_end prt = 1 + err = 0 } +END { exit err } { replace_vars() } # variable replacements $0 ~ vld, $0 ~ end { if ($0 ~ vld || $0 ~ end) prt=1; @@ -345,14 +388,32 @@ $0 ~ vld, $0 ~ end { if ($0 ~ els || $0 ~ end) prt=1; if ($0 ~ skp) next; } -{ if (prt) print } +{ if (!prt) next } +$0 ~ inc { + file = $0 + sub(inc_start, "", file) + sub(inc_end, "", file) + sub(/^[^\/].*$/, source_dir "/&", file) + + while ((res = getline <file) > 0) { + replace_vars() + print + } + if (res < 0) { + printf "%s:%d: error: could not read '%s'\n", FILENAME, NR, file | "cat 1>&2" + err = 1 + } + close(file) + next +} +{ print } function replace_vars() { for (label in c) { gsub(("{{" blank "*yadm\\." label blank "*}}"), c[label]) } } function conditions() { - pattern = "^{%" blank "*if" blank "*(" + pattern = ifs blank "*(" for (label in c) { value = c[label] gsub(/[\\.^$(){}\[\]|*+?]/, "\\\\&", value) @@ -371,9 +432,14 @@ EOF -v user="$local_user" \ -v distro="$local_distro" \ -v source="$input" \ + -v source_dir="$(dirname "$input")" \ "$awk_pgm" \ - "$input" > "$temp_file" - [ -f "$temp_file" ] && mv -f "$temp_file" "$output" + "$input" > "$temp_file" || rm -f "$temp_file" + + if [ -f "$temp_file" ] ; then + copy_perms "$input" "$temp_file" + mv -f "$temp_file" "$output" + fi } function template_j2cli() { @@ -388,7 +454,11 @@ function template_j2cli() { YADM_DISTRO="$local_distro" \ YADM_SOURCE="$input" \ "$J2CLI_PROGRAM" "$input" -o "$temp_file" - [ -f "$temp_file" ] && mv -f "$temp_file" "$output" + + if [ -f "$temp_file" ] ; then + copy_perms "$input" "$temp_file" + mv -f "$temp_file" "$output" + fi } function template_envtpl() { @@ -403,7 +473,30 @@ function template_envtpl() { YADM_DISTRO="$local_distro" \ YADM_SOURCE="$input" \ "$ENVTPL_PROGRAM" --keep-template "$input" -o "$temp_file" - [ -f "$temp_file" ] && mv -f "$temp_file" "$output" + + if [ -f "$temp_file" ] ; then + copy_perms "$input" "$temp_file" + mv -f "$temp_file" "$output" + fi +} + +function template_esh() { + input="$1" + output="$2" + temp_file="${output}.$$.$RANDOM" + + "$ESH_PROGRAM" -o "$temp_file" "$input" \ + YADM_CLASS="$local_class" \ + YADM_OS="$local_system" \ + YADM_HOSTNAME="$local_host" \ + YADM_USER="$local_user" \ + YADM_DISTRO="$local_distro" \ + YADM_SOURCE="$input" + + if [ -f "$temp_file" ] ; then + copy_perms "$input" "$temp_file" + mv -f "$temp_file" "$output" + fi } # ****** yadm Commands ****** @@ -429,57 +522,44 @@ function alt() { local do_copy=0 [ "$(config --bool yadm.alt-copy)" == "true" ] && do_copy=1 - # deprecated yadm.cygwin-copy option (to be removed) - [ "$(config --bool yadm.cygwin-copy)" == "true" ] && do_copy=1 - cd_work "Alternates" || return # determine all tracked files - local tracked_files - tracked_files=() + local tracked_files=() local IFS=$'\n' for tracked_file in $("$GIT_PROGRAM" ls-files | LC_ALL=C sort); do tracked_files+=("$tracked_file") done # generate data for removing stale links - local possible_alts - possible_alts=() + local possible_alts=() local IFS=$'\n' for possible_alt in "${tracked_files[@]}" "${ENCRYPT_INCLUDE_FILES[@]}"; do if [[ $possible_alt =~ .\#\#. ]]; then base_alt="${possible_alt%%##*}" - yadm_alt="${YADM_WORK}/${base_alt}" + yadm_alt="${YADM_BASE}/${base_alt}" if [ "${yadm_alt#$YADM_ALT/}" != "${yadm_alt}" ]; then base_alt="${yadm_alt#$YADM_ALT/}" fi - possible_alts+=("$YADM_WORK/${base_alt}") + possible_alts+=("$YADM_BASE/${base_alt}") fi done - local alt_linked - alt_linked=() - - if [ "$YADM_COMPATIBILITY" = "1" ]; then - alt_past_linking - else - alt_future_linking - fi + local alt_linked=() + alt_linking remove_stale_links - report_invalid_alts } function report_invalid_alts() { - [ "$YADM_COMPATIBILITY" = "1" ] && return [ "$LEGACY_WARNING_ISSUED" = "1" ] && return [ "${#INVALID_ALT[@]}" = "0" ] && return local path_list for invalid in "${INVALID_ALT[@]}"; do path_list="$path_list * $invalid"$'\n' done - cat <<EOF + cat <<EOF >&2 **WARNING** Invalid alternates have been detected. @@ -546,19 +626,15 @@ function set_local_alt_values() { } -function alt_future_linking() { +function alt_linking() { - local alt_scores - local alt_targets - local alt_sources - local alt_template_cmds - alt_scores=() - alt_targets=() - alt_sources=() - alt_template_cmds=() + local alt_scores=() + local alt_targets=() + local alt_sources=() + local alt_template_cmds=() for alt_path in $(for tracked in "${tracked_files[@]}"; do printf "%s\n" "$tracked" "${tracked%/*}"; done | LC_ALL=C sort -u) "${ENCRYPT_INCLUDE_FILES[@]}"; do - alt_path="$YADM_WORK/$alt_path" + alt_path="$YADM_BASE/$alt_path" if [[ "$alt_path" =~ .\#\#. ]]; then if [ -e "$alt_path" ] ; then score_file "$alt_path" @@ -597,85 +673,15 @@ function alt_future_linking() { } -function alt_past_linking() { - - if [ -z "$local_class" ] ; then - match_class="%" - else - match_class="$local_class" - fi - match_class="(%|$match_class)" - match_system="(%|$local_system)" - match_host="(%|$local_host)" - match_user="(%|$local_user)" - - # regex for matching "<file>##CLASS.SYSTEM.HOSTNAME.USER" - match1="^(.+)##(()|$match_system|$match_system\.$match_host|$match_system\.$match_host\.$match_user)$" - match2="^(.+)##($match_class|$match_class\.$match_system|$match_class\.$match_system\.$match_host|$match_class\.$match_system\.$match_host\.$match_user)$" - - # loop over all "tracked" files - # for every file which matches the above regex, create a symlink - for match in $match1 $match2; do - last_linked='' - local IFS=$'\n' - # the alt_paths looped over here are a unique sorted list of both files and their immediate parent directory - for alt_path in $(for tracked in "${tracked_files[@]}"; do printf "%s\n" "$tracked" "${tracked%/*}"; done | LC_ALL=C sort -u) "${ENCRYPT_INCLUDE_FILES[@]}"; do - alt_path="$YADM_WORK/$alt_path" - if [ -e "$alt_path" ] ; then - if [[ $alt_path =~ $match ]] ; then - if [ "$alt_path" != "$last_linked" ] ; then - new_link="${BASH_REMATCH[1]}" - debug "Linking $alt_path to $new_link" - [ -n "$loud" ] && echo "Linking $alt_path to $new_link" - if [ "$do_copy" -eq 1 ]; then - if [ -L "$new_link" ]; then - rm -f "$new_link" - fi - cp -f "$alt_path" "$new_link" - else - ln_relative "$alt_path" "$new_link" - fi - last_linked="$alt_path" - fi - fi - fi - done - done - - # loop over all "tracked" files - # for every file which is a *##yadm.j2 create a real file - local match="^(.+)##yadm\\.j2$" - for tracked_file in "${tracked_files[@]}" "${ENCRYPT_INCLUDE_FILES[@]}"; do - tracked_file="$YADM_WORK/$tracked_file" - if [ -e "$tracked_file" ] ; then - if [[ $tracked_file =~ $match ]] ; then - real_file="${BASH_REMATCH[1]}" - if envtpl_available; then - debug "Creating $real_file from template $tracked_file" - [ -n "$loud" ] && echo "Creating $real_file from template $tracked_file" - temp_file="${real_file}.$$.$RANDOM" - YADM_CLASS="$local_class" \ - YADM_OS="$local_system" \ - YADM_HOSTNAME="$local_host" \ - YADM_USER="$local_user" \ - YADM_DISTRO="$local_distro" \ - "$ENVTPL_PROGRAM" --keep-template "$tracked_file" -o "$temp_file" - [ -f "$temp_file" ] && mv -f "$temp_file" "$real_file" - else - debug "envtpl not available, not creating $real_file from template $tracked_file" - [ -n "$loud" ] && echo "envtpl not available, not creating $real_file from template $tracked_file" - fi - fi - fi - done - -} - function ln_relative() { local full_source full_target target_dir - full_source="$1" - full_target="$2" - target_dir="${full_target%/*}" + local full_source="$1" + local full_target="$2" + local target_dir="${full_target%/*}" + if [ "$target_dir" == "" ]; then + target_dir="/" + fi + local rel_source rel_source=$(relative_path "$target_dir" "$full_source") ln -nfs "$rel_source" "$full_target" alt_linked+=("$rel_source") @@ -699,13 +705,23 @@ function clean() { } +function _default_remote_branch() { + local ls_remote + ls_remote=$("$GIT_PROGRAM" ls-remote -q --symref "$1" 2>/dev/null) + match="^ref:[[:blank:]]+refs/heads/([^[:blank:]]+)" + if [[ "$ls_remote" =~ $match ]] ; then + echo "${BASH_REMATCH[1]}" + else + echo master + fi +} + function clone() { DO_BOOTSTRAP=1 - local branch - branch="master" + local branch= - clone_args=() + local repo_url= while [[ $# -gt 0 ]] ; do key="$1" case $key in @@ -722,22 +738,29 @@ function clone() { --no-bootstrap) # prevent bootstrap, without prompt DO_BOOTSTRAP=3 ;; - *) # main arguments are kept intact - clone_args+=("$1") + *) # use first found argument as the URL + [ -z "$repo_url" ] && repo_url="$1" ;; esac shift done + [ -z "$repo_url" ] && error_out "No repository provided" + + [ -z "$branch" ] && branch=$(_default_remote_branch "$repo_url") + [ -n "$DEBUG" ] && display_private_perms "initial" + # shellcheck disable=SC2119 # clone will begin with a bare repo - local empty= - init $empty + init + + # configure local HEAD with the correct branch + printf 'ref: refs/heads/%s\n' "$branch" > "${YADM_REPO}/HEAD" # add the specified remote, and configure the repo to track origin/$branch debug "Adding remote to new repo" - "$GIT_PROGRAM" remote add origin "${clone_args[@]}" + "$GIT_PROGRAM" remote add origin "$repo_url" debug "Configuring new repo to track origin/${branch}" "$GIT_PROGRAM" config "branch.${branch}.remote" origin "$GIT_PROGRAM" config "branch.${branch}.merge" "refs/heads/${branch}" @@ -747,13 +770,13 @@ function clone() { "$GIT_PROGRAM" fetch origin || { debug "Removing repo after failed clone" rm -rf "$YADM_REPO" - error_out "Unable to fetch origin ${clone_args[0]}" + error_out "Unable to fetch origin $repo_url" } debug "Verifying '${branch}' is a valid branch to merge" [ -f "${YADM_REPO}/refs/remotes/origin/${branch}" ] || { debug "Removing repo after failed clone" rm -rf "$YADM_REPO" - error_out "Clone failed, 'origin/${branch}' does not exist in ${clone_args[0]}" + error_out "Clone failed, 'origin/${branch}' does not exist in $repo_url" } if [ "$YADM_WORK" = "$HOME" ]; then @@ -848,6 +871,8 @@ EOF CHANGES_POSSIBLE=1 else + # make sure parent folder of config file exists + assert_parent "$YADM_CONFIG" # operate on the yadm configuration file "$GIT_PROGRAM" config --file="$(mixed_path "$YADM_CONFIG")" "$@" @@ -855,9 +880,98 @@ EOF } +function _set_gpg_options() { + gpg_key="$(config yadm.gpg-recipient)" + if [ "$gpg_key" = "ASK" ]; then + GPG_OPTS=("--no-default-recipient" "-e") + elif [ "$gpg_key" != "" ]; then + GPG_OPTS=("-e" "-r $gpg_key") + else + GPG_OPTS=("-c") + fi +} + +function _get_openssl_ciphername() { + OPENSSL_CIPHERNAME="$(config yadm.openssl-ciphername)" + if [ -z "$OPENSSL_CIPHERNAME" ]; then + OPENSSL_CIPHERNAME="aes-256-cbc" + fi + echo "$OPENSSL_CIPHERNAME" +} + +function _set_openssl_options() { + cipher_name="$(_get_openssl_ciphername)" + OPENSSL_OPTS=("-${cipher_name}" -salt) + if [ "$(config --bool yadm.openssl-old)" == "true" ]; then + OPENSSL_OPTS+=(-md md5) + else + OPENSSL_OPTS+=(-pbkdf2 -iter 100000 -md sha512) + fi +} + +function _get_cipher() { + output_archive="$1" + yadm_cipher="$(config yadm.cipher)" + if [ -z "$yadm_cipher" ]; then + yadm_cipher="gpg" + fi +} + +function _decrypt_from() { + + local output_archive + local yadm_cipher + _get_cipher "$1" + + case "$yadm_cipher" in + gpg) + require_gpg + $GPG_PROGRAM -d "$output_archive" + ;; + + openssl) + require_openssl + _set_openssl_options + $OPENSSL_PROGRAM enc -d "${OPENSSL_OPTS[@]}" -in "$output_archive" + ;; + + *) + error_out "Unknown cipher '$yadm_cipher'" + ;; + + esac + +} + +function _encrypt_to() { + + local output_archive + local yadm_cipher + _get_cipher "$1" + + case "$yadm_cipher" in + gpg) + require_gpg + _set_gpg_options + $GPG_PROGRAM --yes "${GPG_OPTS[@]}" --output "$output_archive" + ;; + + openssl) + require_openssl + _set_openssl_options + $OPENSSL_PROGRAM enc -e "${OPENSSL_OPTS[@]}" -out "$output_archive" + ;; + + *) + error_out "Unknown cipher '$yadm_cipher'" + ;; + + esac + +} + function decrypt() { - require_gpg require_archive [ -f "$YADM_ENCRYPT" ] && exclude_encrypted @@ -869,7 +983,7 @@ function decrypt() { fi # decrypt the archive - if ($GPG_PROGRAM -d "$YADM_ARCHIVE" || echo 1) | tar v${tar_option}f - -C "$YADM_WORK"; then + if (_decrypt_from "$YADM_ARCHIVE" || echo 1) | tar v${tar_option}f - -C "$YADM_WORK"; then [ ! "$DO_LIST" = "YES" ] && echo "All files decrypted." else error_out "Unable to extract encrypted files." @@ -881,33 +995,19 @@ function decrypt() { function encrypt() { - require_gpg require_encrypt exclude_encrypted parse_encrypt cd_work "Encryption" || return - # Build gpg options for gpg - GPG_KEY="$(config yadm.gpg-recipient)" - if [ "$GPG_KEY" = "ASK" ]; then - GPG_OPTS=("--no-default-recipient" "-e") - elif [ "$GPG_KEY" != "" ]; then - GPG_OPTS=("-e") - for key in $GPG_KEY; do - GPG_OPTS+=("-r $key") - done - else - GPG_OPTS=("-c") - fi - # report which files will be encrypted echo "Encrypting the following files:" printf '%s\n' "${ENCRYPT_INCLUDE_FILES[@]}" echo # encrypt all files which match the globs - if tar -f - -c "${ENCRYPT_INCLUDE_FILES[@]}" | $GPG_PROGRAM --yes "${GPG_OPTS[@]}" --output "$YADM_ARCHIVE"; then + if tar -f - -c "${ENCRYPT_INCLUDE_FILES[@]}" | _encrypt_to "$YADM_ARCHIVE"; then echo "Wrote new file: $YADM_ARCHIVE" else error_out "Unable to write $YADM_ARCHIVE" @@ -934,18 +1034,27 @@ function git_crypt() { enter "${GIT_CRYPT_PROGRAM} $*" } +function transcrypt() { + require_transcrypt + enter "${TRANSCRYPT_PROGRAM} $*" +} + function enter() { command="$*" require_shell require_repo - shell_opts="" - shell_path="" + local -a shell_opts + local shell_path="" if [[ "$SHELL" =~ bash$ ]]; then - shell_opts="--norc" + shell_opts=("--norc") shell_path="\w" elif [[ "$SHELL" =~ [cz]sh$ ]]; then - shell_opts="-f" + shell_opts=("-f") + if [[ "$SHELL" =~ zsh$ && "$TERM" = "dumb" ]]; then + # Disable ZLE for tramp + shell_opts+=("--no-zle") + fi shell_path="%~" fi @@ -960,7 +1069,7 @@ function enter() { [ "${#shell_cmd[@]}" -eq 0 ] && echo "Entering yadm repo" yadm_prompt="yadm shell ($YADM_REPO) $shell_path > " - PROMPT="$yadm_prompt" PS1="$yadm_prompt" "$SHELL" $shell_opts "${shell_cmd[@]}" + PROMPT="$yadm_prompt" PS1="$yadm_prompt" "$SHELL" "${shell_opts[@]}" "${shell_cmd[@]}" return_code="$?" if [ "${#shell_cmd[@]}" -eq 0 ]; then @@ -1023,12 +1132,14 @@ Commands: yadm perms - Fix perms for private files yadm enter [COMMAND] - Run sub-shell with GIT variables set yadm git-crypt [OPTIONS] - Run git-crypt commands for the yadm repo + yadm transcrypt [OPTIONS] - Run transcrypt commands for the yadm repo Files: - \$HOME/.config/yadm/config - yadm's configuration file - \$HOME/.config/yadm/repo.git - yadm's Git repository - \$HOME/.config/yadm/encrypt - List of globs used for encrypt/decrypt - \$HOME/.config/yadm/files.gpg - Encrypted data stored here + \$HOME/.config/yadm/config - yadm's configuration file + \$HOME/.config/yadm/encrypt - List of globs to encrypt/decrypt + \$HOME/.config/yadm/bootstrap - Script run via: yadm bootstrap + \$HOME/.local/share/yadm/repo.git - yadm's Git repository + \$HOME/.local/share/yadm/archive - Encrypted data stored here Use "man yadm" for complete documentation. EOF @@ -1037,6 +1148,7 @@ EOF } +# shellcheck disable=SC2120 function init() { # safety check, don't attempt to init when the repo is already present @@ -1076,13 +1188,14 @@ config decrypt encrypt enter -gitconfig git-crypt +gitconfig help init introspect list perms +transcrypt upgrade version EOF @@ -1099,10 +1212,14 @@ yadm.auto-alt yadm.auto-exclude yadm.auto-perms yadm.auto-private-dirs +yadm.cipher yadm.git-program yadm.gpg-perms yadm.gpg-program yadm.gpg-recipient +yadm.openssl-ciphername +yadm.openssl-old +yadm.openssl-program yadm.ssh-perms EOF } @@ -1116,6 +1233,7 @@ function introspect_switches() { --yadm-archive --yadm-bootstrap --yadm-config +--yadm-data --yadm-dir --yadm-encrypt --yadm-repo @@ -1177,40 +1295,82 @@ function perms() { function upgrade() { - local actions_performed - actions_performed=0 - local repo_updates - repo_updates=0 + local actions_performed=0 + local -a submodules + local repo_updates=0 - [ "$YADM_COMPATIBILITY" = "1" ] && \ - error_out "Unable to upgrade. YADM_COMPATIBILITY is set to '1'." + [[ -n "${YADM_OVERRIDE_REPO}${YADM_OVERRIDE_ARCHIVE}" || "$YADM_DATA" = "$YADM_DIR" ]] && \ + error_out "Unable to upgrade. Paths have been overridden with command line options" - [ "$YADM_DIR" = "$YADM_LEGACY_DIR" ] && \ - error_out "Unable to upgrade. yadm dir has been resolved as '$YADM_LEGACY_DIR'." + # choose a legacy repo, the version 2 location will be favored + local LEGACY_REPO= + [ -d "$YADM_LEGACY_DIR/repo.git" ] && LEGACY_REPO="$YADM_LEGACY_DIR/repo.git" + [ -d "$YADM_DIR/repo.git" ] && LEGACY_REPO="$YADM_DIR/repo.git" # handle legacy repo - if [ -d "$YADM_LEGACY_DIR/repo.git" ]; then + if [ -d "$LEGACY_REPO" ]; then + # choose # legacy repo detected, it must be moved to YADM_REPO if [ -e "$YADM_REPO" ]; then error_out "Unable to upgrade. '$YADM_REPO' already exists. Refusing to overwrite it." else actions_performed=1 - echo "Moving $YADM_LEGACY_DIR/repo.git to $YADM_REPO" + echo "Moving $LEGACY_REPO to $YADM_REPO" + + export GIT_DIR="$LEGACY_REPO" + + # Must absorb git dirs, otherwise deinit below will fail for modules that have + # been cloned first and then added as a submodule. + "$GIT_PROGRAM" submodule absorbgitdirs + + local submodule_status + submodule_status=$("$GIT_PROGRAM" -C "$YADM_WORK" submodule status) + while read -r sha submodule rest; do + [ "$submodule" == "" ] && continue + if [[ "$sha" = -* ]]; then + continue + fi + "$GIT_PROGRAM" -C "$YADM_WORK" submodule deinit ${FORCE:+-f} -- "$submodule" || { + for other in "${submodules[@]}"; do + "$GIT_PROGRAM" -C "$YADM_WORK" submodule update --init --recursive -- "$other" + done + error_out "Unable to upgrade. Could not deinit submodule $submodule" + } + submodules+=("$submodule") + done <<< "$submodule_status" + assert_parent "$YADM_REPO" - mv "$YADM_LEGACY_DIR/repo.git" "$YADM_REPO" + mv "$LEGACY_REPO" "$YADM_REPO" fi fi - - # handle other legacy paths GIT_DIR="$YADM_REPO" export GIT_DIR + + # choose a legacy archive, the version 2 location will be favored + local LEGACY_ARCHIVE= + [ -e "$YADM_LEGACY_DIR/$YADM_LEGACY_ARCHIVE" ] && LEGACY_ARCHIVE="$YADM_LEGACY_DIR/$YADM_LEGACY_ARCHIVE" + [ -e "$YADM_DIR/$YADM_LEGACY_ARCHIVE" ] && LEGACY_ARCHIVE="$YADM_DIR/$YADM_LEGACY_ARCHIVE" + + # handle legacy archive + if [ -e "$LEGACY_ARCHIVE" ]; then + actions_performed=1 + echo "Moving $LEGACY_ARCHIVE to $YADM_ARCHIVE" + assert_parent "$YADM_ARCHIVE" + # test to see if path is "tracked" in repo, if so 'git mv' must be used + if "$GIT_PROGRAM" ls-files --error-unmatch "$LEGACY_ARCHIVE" &> /dev/null; then + "$GIT_PROGRAM" mv "$LEGACY_ARCHIVE" "$YADM_ARCHIVE" && repo_updates=1 + else + mv -i "$LEGACY_ARCHIVE" "$YADM_ARCHIVE" + fi + fi + + # handle any remaining version 1 paths for legacy_path in \ "$YADM_LEGACY_DIR/config" \ "$YADM_LEGACY_DIR/encrypt" \ - "$YADM_LEGACY_DIR/files.gpg" \ "$YADM_LEGACY_DIR/bootstrap" \ "$YADM_LEGACY_DIR"/hooks/{pre,post}_* \ - ; \ + ; do if [ -e "$legacy_path" ]; then new_filename=${legacy_path#$YADM_LEGACY_DIR/} @@ -1228,19 +1388,15 @@ function upgrade() { done # handle submodules, which need to be reinitialized - if [ "$actions_performed" -ne 0 ]; then - cd_work "Upgrade submodules" - if "$GIT_PROGRAM" ls-files --error-unmatch .gitmodules &> /dev/null; then - "$GIT_PROGRAM" submodule deinit -f . - "$GIT_PROGRAM" submodule update --init --recursive - fi - fi + for submodule in "${submodules[@]}"; do + "$GIT_PROGRAM" -C "$YADM_WORK" submodule update --init --recursive -- "$submodule" + done [ "$actions_performed" -eq 0 ] && \ echo "No legacy paths found. Upgrade is not necessary" [ "$repo_updates" -eq 1 ] && \ - echo "Some files tracked by yadm have been renamed. This changes should probably be commited now." + echo "Some files tracked by yadm have been renamed. These changes should probably be commited now." exit 0 @@ -1310,7 +1466,8 @@ function is_valid_branch_name() { # * "~", "^", ":", "\", space # * end with a "/" # * end with ".lock" - [[ "$1" =~ (\/\.|\.\.|[~^:\\ ]|\/$|\.lock$) ]] && return 1 + pattern='(\/\.|\.\.|[~^:\\ ]|\/$|\.lock$)' + [[ "$1" =~ $pattern ]] && return 1 return 0 } @@ -1344,6 +1501,13 @@ function process_global_args() { YADM_DIR="$2" shift ;; + --yadm-data) # override the standard YADM_DATA + if [[ ! "$2" =~ ^/ ]] ; then + error_out "You must specify a fully qualified yadm data directory" + fi + YADM_DATA="$2" + shift + ;; --yadm-repo) # override the standard YADM_REPO if [[ ! "$2" =~ ^/ ]] ; then error_out "You must specify a fully qualified repo path" @@ -1388,23 +1552,25 @@ function process_global_args() { } -function set_yadm_dir() { - - # only resolve YADM_DIR if it hasn't been provided already - [ -n "$YADM_DIR" ] && return +function set_yadm_dirs() { - # compatibility with major version 1 ignores XDG_CONFIG_HOME - if [ "$YADM_COMPATIBILITY" = "1" ]; then - YADM_DIR="$YADM_LEGACY_DIR" - return + # only resolve YADM_DATA if it hasn't been provided already + if [ -z "$YADM_DATA" ]; then + local base_yadm_data="$XDG_DATA_HOME" + if [[ ! "$base_yadm_data" =~ ^/ ]] ; then + base_yadm_data="${HOME}/.local/share" + fi + YADM_DATA="${base_yadm_data}/yadm" fi - local base_yadm_dir - base_yadm_dir="$XDG_CONFIG_HOME" - if [[ ! "$base_yadm_dir" =~ ^/ ]] ; then - base_yadm_dir="${HOME}/.config" + # only resolve YADM_DIR if it hasn't been provided already + if [ -z "$YADM_DIR" ]; then + local base_yadm_dir="$XDG_CONFIG_HOME" + if [[ ! "$base_yadm_dir" =~ ^/ ]] ; then + base_yadm_dir="${HOME}/.config" + fi + YADM_DIR="${base_yadm_dir}/yadm" fi - YADM_DIR="${base_yadm_dir}/yadm" issue_legacy_path_warning @@ -1418,21 +1584,22 @@ function issue_legacy_path_warning() { # no warnings if YADM_DIR is resolved as the leacy path [ "$YADM_DIR" = "$YADM_LEGACY_DIR" ] && return - # no warnings if the legacy directory doesn't exist - [ ! -d "$YADM_LEGACY_DIR" ] && return + # no warnings if overrides have been provided + [[ -n "${YADM_OVERRIDE_REPO}${YADM_OVERRIDE_ARCHIVE}" || "$YADM_DATA" = "$YADM_DIR" ]] && return # test for legacy paths - local legacy_found - legacy_found=() + local legacy_found=() # this is ordered by importance for legacy_path in \ + "$YADM_DIR/$YADM_REPO" \ + "$YADM_DIR/$YADM_LEGACY_ARCHIVE" \ "$YADM_LEGACY_DIR/$YADM_REPO" \ + "$YADM_LEGACY_DIR/$YADM_BOOTSTRAP" \ "$YADM_LEGACY_DIR/$YADM_CONFIG" \ "$YADM_LEGACY_DIR/$YADM_ENCRYPT" \ - "$YADM_LEGACY_DIR/$YADM_ARCHIVE" \ - "$YADM_LEGACY_DIR/$YADM_BOOTSTRAP" \ "$YADM_LEGACY_DIR/$YADM_HOOKS"/{pre,post}_* \ - ; \ + "$YADM_LEGACY_DIR/$YADM_LEGACY_ARCHIVE" \ + ; do [ -e "$legacy_path" ] && legacy_found+=("$legacy_path") done @@ -1444,26 +1611,25 @@ function issue_legacy_path_warning() { path_list="$path_list * $legacy_path"$'\n' done - cat <<EOF + cat <<EOF >&2 **WARNING** - Legacy configuration paths have been detected. + Legacy paths have been detected. - Beginning with version 2.0.0, yadm uses the XDG Base Directory Specification - to find its configurations. Read more about this change here: + With version 3.0.0, yadm uses the XDG Base Directory Specification + to find its configurations and data. Read more about these changes here: + https://yadm.io/docs/upgrade_from_2 https://yadm.io/docs/upgrade_from_1 - In your environment, the configuration directory has been resolved to: + In your environment, the data directory has been resolved to: - $YADM_DIR + $YADM_DATA To remove this warning do one of the following: - * Run "yadm upgrade" to move the yadm data to the new directory. (RECOMMENDED) - * Manually move yadm configurations to the directory listed above. - * Specify your preferred yadm directory with -Y each execution. - * Define an environment variable "YADM_COMPATIBILITY=1" to run in version 1 - compatibility mode. (DEPRECATED) + * Run "yadm upgrade" to move the yadm data to the new paths. (RECOMMENDED) + * Manually move yadm data to new default paths and reinit any submodules. + * Specify your preferred paths with --yadm-data and --yadm-archive each execution. Legacy paths detected: ${path_list} @@ -1476,15 +1642,17 @@ LEGACY_WARNING_ISSUED=1 function configure_paths() { - # change all paths to be relative to YADM_DIR - YADM_REPO="$YADM_DIR/$YADM_REPO" + # change paths to be relative to YADM_DIR YADM_CONFIG="$YADM_DIR/$YADM_CONFIG" YADM_ENCRYPT="$YADM_DIR/$YADM_ENCRYPT" - YADM_ARCHIVE="$YADM_DIR/$YADM_ARCHIVE" YADM_BOOTSTRAP="$YADM_DIR/$YADM_BOOTSTRAP" YADM_HOOKS="$YADM_DIR/$YADM_HOOKS" YADM_ALT="$YADM_DIR/$YADM_ALT" + # change paths to be relative to YADM_DATA + YADM_REPO="$YADM_DATA/$YADM_REPO" + YADM_ARCHIVE="$YADM_DATA/$YADM_ARCHIVE" + # independent overrides for paths if [ -n "$YADM_OVERRIDE_REPO" ]; then YADM_REPO="$YADM_OVERRIDE_REPO" @@ -1513,6 +1681,14 @@ function configure_paths() { [ -n "$work" ] && YADM_WORK="$work" fi + # YADM_BASE is used for manipulating the base worktree path for much of the + # alternate file processing + if [ "$YADM_WORK" == "/" ]; then + YADM_BASE="" + else + YADM_BASE="$YADM_WORK" + fi + } function configure_repo() { @@ -1572,7 +1748,7 @@ function debug() { function error_out() { - echo_e "ERROR: $*" + echo_e "ERROR: $*" >&2 exit_with_hook 1 } @@ -1590,12 +1766,14 @@ function invoke_hook() { exit_status="$2" hook_command="${YADM_HOOKS}/${mode}_$HOOK_COMMAND" - if [ -x "$hook_command" ] ; then + if [ -x "$hook_command" ] || \ + { [[ $OPERATING_SYSTEM == MINGW* ]] && [ -f "$hook_command" ] ;} ; then debug "Invoking hook: $hook_command" # expose some internal data to all hooks YADM_HOOK_COMMAND=$HOOK_COMMAND YADM_HOOK_DIR=$YADM_DIR + YADM_HOOK_DATA=$YADM_DATA YADM_HOOK_EXIT=$exit_status YADM_HOOK_FULL_COMMAND=$FULL_COMMAND YADM_HOOK_REPO=$YADM_REPO @@ -1607,6 +1785,7 @@ function invoke_hook() { export YADM_HOOK_COMMAND export YADM_HOOK_DIR + export YADM_HOOK_DATA export YADM_HOOK_EXIT export YADM_HOOK_FULL_COMMAND export YADM_HOOK_REPO @@ -1660,7 +1839,9 @@ function assert_private_dirs() { function assert_parent() { basedir=${1%/*} - [ -e "$basedir" ] || mkdir -p "$basedir" + if [ -n "$basedir" ]; then + [ -e "$basedir" ] || mkdir -p "$basedir" + fi } function display_private_perms() { @@ -1705,7 +1886,7 @@ function parse_encrypt() { if [ -f "$YADM_ENCRYPT" ] ; then # parse both included/excluded while IFS='' read -r line || [ -n "$line" ]; do - if [[ ! $line =~ ^# && ! $line =~ ^[[:space:]]*$ ]] ; then + if [[ ! $line =~ ^# && ! $line =~ ^[[:blank:]]*$ ]] ; then local IFS=$'\n' for pattern in $line; do if [[ "$pattern" =~ $exclude_pattern ]]; then @@ -1862,6 +2043,34 @@ function join_string { printf "%s" "${*:2}" } +function get_mode { + local filename="$1" + local mode + + # most *nixes + mode=$(stat -c '%a' "$filename" 2>/dev/null) + if [ -z "$mode" ] ; then + # BSD-style + mode=$(stat -f '%p' "$filename" 2>/dev/null) + mode=${mode: -4} + fi + + # only accept results if they are octal + if [[ ! $mode =~ ^[0-7]+$ ]] ; then + mode="" + fi + + echo "$mode" +} + +function copy_perms { + local source="$1" + local dest="$2" + mode=$(get_mode "$source") + [ -n "$mode" ] && chmod "$mode" "$dest" + return 0 +} + # ****** Prerequisites Functions ****** function require_archive() { @@ -1874,8 +2083,7 @@ function require_git() { local alt_git alt_git="$(config yadm.git-program)" - local more_info - more_info="" + local more_info="" if [ "$alt_git" != "" ] ; then GIT_PROGRAM="$alt_git" @@ -1888,8 +2096,7 @@ function require_gpg() { local alt_gpg alt_gpg="$(config yadm.gpg-program)" - local more_info - more_info="" + local more_info="" if [ "$alt_gpg" != "" ] ; then GPG_PROGRAM="$alt_gpg" @@ -1898,6 +2105,19 @@ function require_gpg() { command -v "$GPG_PROGRAM" &> /dev/null || error_out "This functionality requires GPG to be installed, but the command '$GPG_PROGRAM' cannot be located.$more_info" } +function require_openssl() { + local alt_openssl + alt_openssl="$(config yadm.openssl-program)" + + local more_info="" + + if [ "$alt_openssl" != "" ] ; then + OPENSSL_PROGRAM="$alt_openssl" + more_info="\nThis command has been set via the yadm.openssl-program configuration." + fi + command -v "$OPENSSL_PROGRAM" &> /dev/null || + error_out "This functionality requires OpenSSL to be installed, but the command '$OPENSSL_PROGRAM' cannot be located.$more_info" +} function require_repo() { [ -d "$YADM_REPO" ] || error_out "Git repo does not exist. did you forget to run 'init' or 'clone'?" } @@ -1908,6 +2128,10 @@ function require_git_crypt() { command -v "$GIT_CRYPT_PROGRAM" &> /dev/null || error_out "This functionality requires git-crypt to be installed, but the command '$GIT_CRYPT_PROGRAM' cannot be located." } +function require_transcrypt() { + command -v "$TRANSCRYPT_PROGRAM" &> /dev/null || + error_out "This functionality requires transcrypt to be installed, but the command '$TRANSCRYPT_PROGRAM' cannot be located." +} function bootstrap_available() { [ -f "$YADM_BOOTSTRAP" ] && [ -x "$YADM_BOOTSTRAP" ] && return return 1 @@ -1924,6 +2148,10 @@ function envtpl_available() { command -v "$ENVTPL_PROGRAM" &> /dev/null && return return 1 } +function esh_available() { + command -v "$ESH_PROGRAM" &> /dev/null && return + return 1 +} function readlink_available() { command -v "readlink" &> /dev/null && return return 1 @@ -1969,7 +2197,7 @@ if [ "$YADM_TEST" != 1 ] ; then process_global_args "$@" set_operating_system set_awk - set_yadm_dir + set_yadm_dirs configure_paths main "${MAIN_ARGS[@]}" fi @@ -1,5 +1,5 @@ .\" vim: set spell so=8: -.TH yadm 1 "6 February 2020" "2.4.0" +.TH yadm 1 "7 January 2021" "3.0.2" .SH NAME @@ -58,7 +58,10 @@ list .BR yadm " git-crypt [ options ] +.BR yadm " transcrypt [ options ] + .BR yadm " upgrade +.RB [ -f ] .BR yadm " introspect .I category @@ -114,12 +117,10 @@ if it exists. .BI clone " url Clone a remote repository for tracking dotfiles. After the contents of the remote repository have been fetched, a "merge" of -.I origin/master -is attempted. +the remote HEAD branch is attempted. If there are conflicting files already present in the .IR work-tree , -this merge will fail and instead a "reset" of -.I origin/master +this merge will fail and instead a "reset" of the remote HEAD branch will be done, followed by a "stash". This "stash" operation will preserve the original data. @@ -143,7 +144,7 @@ yadm stash pop .RE The repository is stored in -.IR $HOME/.config/yadm/repo.git . +.IR $HOME/.local/share/yadm/repo.git . By default, .I $HOME will be used as the @@ -152,8 +153,7 @@ but this can be overridden with the .BR -w " option. yadm can be forced to overwrite an existing repository by providing the .BR -f " option. -If you want to use a branch other than -.IR origin/master , +If you want to use a branch other than the remote HEAD branch you can specify it using the .BR -b " option. By default yadm will ask the user if the bootstrap program should be run (if it @@ -165,14 +165,14 @@ without prompting the user. .TP .B config This command manages configurations for yadm. -This command works exactly they way +This command works exactly the way .BR git-config (1) does. See the CONFIGURATION section for more details. .TP .B decrypt Decrypt all files stored in -.IR $HOME/.config/yadm/files.gpg . +.IR $HOME/.local/share/yadm/archive . Files decrypted will be relative to the configured .IR work-tree " (usually .IR $HOME ). @@ -209,8 +209,7 @@ Emacs Tramp and Magit can manage files by using this configuration: .RE .RS -With this config, use (magit-status "/yadm::"). If you find issue with Emacs 27 and zsh, -trying running (setenv "SHELL" "/bin/bash"). +With this config, use (magit-status "/yadm::"). .RE .TP .BI git-crypt " options @@ -248,7 +247,7 @@ Print a summary of yadm commands. .B init Initialize a new, empty repository for tracking dotfiles. The repository is stored in -.IR $HOME/.config/yadm/repo.git . +.IR $HOME/.local/share/yadm/repo.git . By default, .I $HOME will be used as the @@ -281,50 +280,43 @@ configuration .I yadm.auto-perms to "false". .TP +.BI transcrypt " options +If transcrypt is installed, this command allows you to pass options directly to +transcrypt, with the environment configured to use the yadm repository. + +transcrypt enables transparent encryption and decryption of files in a git repository. +You can read +https://github.com/elasticdog/transcrypt +for details. +.TP .B upgrade -Version 2 of yadm uses a different directory for storing your configurations. -When you start to use version 2 for the first time, you may see warnings about +Version 3 of yadm uses a different directory for storing data. +When you start to use version 3 for the first time, you may see warnings about moving your data to this new directory. The easiest way to accomplish this is by running "yadm upgrade". This command will start by moving your yadm repo to the new path. -Next it will move any configuration data to the new path. -If the configurations are tracked within your yadm repo, this command will -"stage" the renaming of those files in the repo's index. -Upgrading will also re-initialize all submodules you have added (otherwise they -will be broken when the repo moves). +Next it will move any archive data. +If the archive is tracked within your yadm repo, this command will +"stage" the renaming of that file in the repo's index. + +Upgrading will attempt to de-initialize and re-initialize your submodules. If +your submodules cannot be de-initialized, the upgrade will fail. The most +common reason submodules will fail to de-initialize is because they have local +modifications. If you are willing to lose the local modifications to those +submodules, you can use the +.B -f +option with the "upgrade" command to force the de-initialization. + After running "yadm upgrade", you should run "yadm status" to review changes which have been staged, and commit them to your repository. You can read -https://yadm.io/docs/upgrade_from_1 +https://yadm.io/docs/upgrade_from_2 for more information. .TP .B version Print the version of yadm. -.SH COMPATIBILITY - -Beginning with version 2.0.0, yadm introduced a couple major changes which may -require you to adjust your configurations. -See the -.B upgrade -command for help making those adjustments. - -First, yadm now uses the "XDG Base Directory Specification" to find its -configurations. You can read -https://yadm.io/docs/upgrade_from_1 -for more information. - -Second, the naming conventions for alternate files have been changed. -You can read https://yadm.io/docs/alternates for more information. - -If you want to retain the old functionality, you can set an environment variable, -.IR YADM_COMPATIBILITY=1 . -Doing so will automatically use the old yadm directory, and process alternates -the same as the pre-2.0.0 version. This compatibility mode is deprecated, and -will be removed in future versions. This mode exists solely for transitioning -to the new paths and naming of alternates. - .SH OPTIONS yadm supports a set of universal options that alter the paths it uses. The @@ -343,6 +335,10 @@ Each option should be followed by a fully qualified path. .TP .B -Y,--yadm-dir Override the yadm directory. +yadm stores its configurations relative to this directory. +.TP +.B --yadm-data +Override the yadm data directory. yadm stores its data relative to this directory. .TP .B --yadm-repo @@ -383,12 +379,6 @@ The following is the full list of supported configurations: If set to "true", alternate files will be copies instead of symbolic links. This might be desirable, because some systems may not properly support symlinks. - -NOTE: The deprecated -.I yadm.cygwin-copy -option used by older versions of yadm has been replaced by -.IR yadm.alt-copy . -The old option will be removed in the next version of yadm. .TP .B yadm.auto-alt Disable the automatic linking described in the section ALTERNATES. If disabled, @@ -410,6 +400,11 @@ This feature is enabled by default. .B yadm.auto-private-dirs Disable the automatic creating of private directories described in the section PERMISSIONS. .TP +.B yadm.cipher +Configure which encryption system is used by the encrypt/decrypt commands. +Valid options are "gpg" and "openssl". The default is "gpg". +Detailed information can be found in the section ENCRYPTION. +.TP .B yadm.git-program Specify an alternate program to use instead of "git". By default, the first "git" found in $PATH is used. @@ -433,6 +428,20 @@ If set to "ASK", gpg will interactively ask for recipients. See the ENCRYPTION section for more details. This feature is disabled by default. .TP +.B yadm.openssl-ciphername +Specify which cipher should be used by openssl. +"aes-256-cbc" is used by default. +.TP +.B yadm.openssl-old +Newer versions of openssl support the pbkdf2 key derivation function. This is +used by default. If this configuration is set to "true", openssl operations +will use options compatible with older versions of openssl. If you change this +option, you will need to recreate your encrypted archive. +.TP +.B yadm.openssl-program +Specify an alternate program to use instead of "openssl". +By default, the first "openssl" found in $PATH is used. +.TP .B yadm.ssh-perms Disable the permission changes to .IR $HOME/.ssh/* . @@ -519,6 +528,11 @@ and trimming off any domain. .TP .B default Valid when no other alternate is valid. +.TP +.BR extension , " e +A special "condition" that doesn't affect the selection process. Its purpose is +instead to allow the alternate file to end with a certain extension to +e.g. make editors highlight the content properly. .LP .BR NOTE : @@ -626,6 +640,15 @@ upon which is available on most *nix systems. To use this processor, specify the value of "default" or just leave the value off (e.g. "##template"). .TP +.B ESH +ESH is a template processor written in POSIX compliant shell. It allows +executing shell commands within templates. This can be used to reference your +own configurations within templates, for example: + + <% yadm config mysection.myconfig %> + +To use the ESH template processor, specify the value of "esh" +.TP .B j2cli To use the j2cli Jinja template processor, specify the value of "j2" or "j2cli". @@ -643,7 +666,7 @@ to create or overwrite files. During processing, the following variables are available in the template: - Default Jinja Description + Default Jinja or ESH Description ------------- ------------- -------------------------- yadm.class YADM_CLASS Locally defined yadm class yadm.distro YADM_DISTRO lsb_release -si @@ -665,10 +688,11 @@ Examples: .I whatever##template with the following content - {% if yadm.user == 'harvey' %} + {% if yadm.user == "harvey" %} config={{yadm.class}}-{{yadm.os}} {% else %} config=dev-whatever + {% include "whatever.extra" %} {% endif %} would output a file named @@ -677,9 +701,12 @@ with the following content if the user is "harvey": config=work-Linux -and the following otherwise: +and the following otherwise (if +.I whatever.extra +contains admin=false): config=dev-whatever + admin=false An equivalent Jinja template named .I whatever##template.j2 @@ -689,8 +716,20 @@ would look like: config={{YADM_CLASS}}-{{YADM_OS}} {% else -%} config=dev-whatever + {% include 'whatever.extra' %} {% endif -%} +An equivalent ESH templated named +.I whatever##template.esh +would look like: + + <% if [ "$YADM_USER" = "harvey" ]; then -%> + config=<%= $YADM_CLASS %>-<%= $YADM_OS %> + <% else -%> + config=dev-whatever + <%+ whatever.extra %> + <% fi -%> + .SH ENCRYPTION It can be useful to manage confidential files, like SSH or GPG keys, across @@ -698,9 +737,15 @@ multiple systems. However, doing so would put plain text data into a Git repository, which often resides on a public system. yadm can make it easy to encrypt and decrypt a set of files so the encrypted version can be maintained in the Git repository. -This feature will only work if the +This feature will only work if a supported tool is available. +Both .BR gpg (1) -command is available. +and +.BR openssl (1) +are supported. +gpg is used by default, but openssl can be configured with the +.I yadm.cypher +configuration. To use this feature, a list of patterns must be created and saved as .IR $HOME/.config/yadm/encrypt . @@ -725,8 +770,8 @@ The .B yadm encrypt command will find all files matching the patterns, and prompt for a password. Once a password has confirmed, the matching files will be encrypted and saved as -.IR $HOME/.config/yadm/files.gpg . -The patterns and files.gpg should be added to the yadm repository so they are +.IR $HOME/.local/share/yadm/archive . +The "encrypt" and "archive" files should be added to the yadm repository so they are available across multiple systems. To decrypt these files later, or on another system run @@ -756,15 +801,20 @@ This can be disabled using the .I yadm.auto-exclude configuration. -.B Using git-crypt +.B Using transcrypt or git-crypt -A completely separate option for encrypting data is to install and use git-crypt. -Once installed, you can run git-crypt commands for the yadm repo by running +A completely separate option for encrypting data is to install and use +transcrypt or git-crypt. +Once installed, you can use these tools by running +.B "yadm transcrypt" +or .BR "yadm git-crypt" . -git-crypt enables transparent encryption and decryption of files in a git repository. -You can read -https://github.com/AGWA/git-crypt -for details. +These tools enables transparent encryption and decryption of files in a git +repository. See the following web sites for more information: + +- https://github.com/elasticdog/transcrypt + +- https://github.com/AGWA/git-crypt .LP .SH PERMISSIONS @@ -774,7 +824,7 @@ dependent upon the user's umask. Because of this, yadm will automatically update the permissions of some file paths. The "group" and "others" permissions will be removed from the following files: -.RI - " $HOME/.config/yadm/files.gpg +.RI - " $HOME/.local/share/yadm/archive - All files matching patterns in .I $HOME/.config/yadm/encrypt @@ -870,12 +920,25 @@ is defined as a fully qualified path, this directory will be Otherwise it will be .IR "$HOME/.config/yadm" . +Similarly, yadm's data files are relative to the "yadm data directory". +yadm uses the "XDG Base Directory Specification" to determine this directory. +If the environment variable +.B $XDG_DATA_HOME +is defined as a fully qualified path, this directory will be +.IR "$XDG_DATA_HOME/yadm" . +Otherwise it will be +.IR "$HOME/.local/share/yadm" . + The following are the default paths yadm uses for its own data. Most of these paths can be altered using universal options. See the OPTIONS section for details. .TP .I $HOME/.config/yadm -The yadm directory. By default, all data yadm stores is relative to this +The yadm directory. By default, all configs yadm stores is relative to this +directory. +.TP +.I $HOME/.local/share/yadm +The yadm data directory. By default, all data yadm stores is relative to this directory. .TP .I $YADM_DIR/config @@ -886,13 +949,13 @@ This is a directory to keep "alternate files" without having them side-by-side with the resulting symlink or processed template. Alternate files placed in this directory will be created relative to $HOME instead. .TP -.I $YADM_DIR/repo.git +.I $YADM_DATA/repo.git Git repository used by yadm. .TP .I $YADM_DIR/encrypt List of globs used for encrypt/decrypt .TP -.I $YADM_DIR/files.gpg +.I $YADM_DATA/archive All files encrypted with .B yadm encrypt are stored in this file. @@ -917,7 +980,7 @@ Initial push of master to origin .B echo ".ssh/*.key" >> $HOME/.config/yadm/encrypt Add a new pattern to the list of encrypted files .TP -.B yadm encrypt ; yadm add ~/.config/yadm/files.gpg ; yadm commit +.B yadm encrypt ; yadm add ~/.local/share/yadm/archive ; yadm commit Commit a new set of encrypted files .SH REPORTING BUGS @@ -934,5 +997,8 @@ Tim Byrne <sultan@locehilios.com> .BR git (1), .BR gpg (1) +.BR openssl (1) +.BR transcrypt (1) +.BR git-crypt (1) https://yadm.io/ @@ -34,7 +34,9 @@ yadm git-crypt [ options ] - yadm upgrade + yadm transcrypt [ options ] + + yadm upgrade [-f] yadm introspect category @@ -73,10 +75,11 @@ clone url Clone a remote repository for tracking dotfiles. After the con- tents of the remote repository have been fetched, a "merge" of - origin/master is attempted. If there are conflicting files - already present in the work-tree, this merge will fail and - instead a "reset" of origin/master will be done, followed by a - "stash". This "stash" operation will preserve the original data. + the remote HEAD branch is attempted. If there are conflicting + files already present in the work-tree, this merge will fail and + instead a "reset" of the remote HEAD branch will be done, fol- + lowed by a "stash". This "stash" operation will preserve the + original data. You can review the stashed conflicts by running the command @@ -89,26 +92,26 @@ or yadm stash pop - The repository is stored in $HOME/.config/yadm/repo.git. By - default, $HOME will be used as the work-tree, but this can be + The repository is stored in $HOME/.local/share/yadm/repo.git. + By default, $HOME will be used as the work-tree, but this can be overridden with the -w option. yadm can be forced to overwrite an existing repository by providing the -f option. If you want - to use a branch other than origin/master, you can specify it - using the -b option. By default yadm will ask the user if the - bootstrap program should be run (if it exists). The options - --bootstrap or --no-bootstrap will either force the bootstrap to - be run, or prevent it from being run, without prompting the - user. + to use a branch other than the remote HEAD branch you can spec- + ify it using the -b option. By default yadm will ask the user + if the bootstrap program should be run (if it exists). The + options --bootstrap or --no-bootstrap will either force the + bootstrap to be run, or prevent it from being run, without + prompting the user. config This command manages configurations for yadm. This command - works exactly they way git-config(1) does. See the CONFIGURA- - TION section for more details. + works exactly the way git-config(1) does. See the CONFIGURATION + section for more details. decrypt - Decrypt all files stored in $HOME/.config/yadm/files.gpg. Files - decrypted will be relative to the configured work-tree (usually - $HOME). Using the -l option will list the files stored without - extracting them. + Decrypt all files stored in $HOME/.local/share/yadm/archive. + Files decrypted will be relative to the configured work-tree + (usually $HOME). Using the -l option will list the files stored + without extracting them. encrypt Encrypt all files matching the patterns found in $HOME/.con- @@ -136,9 +139,7 @@ (tramp-remote-shell "/bin/sh") (tramp-remote-shell-args ("-c")))) - With this config, use (magit-status "/yadm::"). If you find - issue with Emacs 27 and zsh, trying running (setenv "SHELL" - "/bin/bash"). + With this config, use (magit-status "/yadm::"). git-crypt options If git-crypt is installed, this command allows you to pass @@ -165,7 +166,7 @@ help Print a summary of yadm commands. init Initialize a new, empty repository for tracking dotfiles. The - repository is stored in $HOME/.config/yadm/repo.git. By + repository is stored in $HOME/.local/share/yadm/repo.git. By default, $HOME will be used as the work-tree, but this can be overridden with the -w option. yadm can be forced to overwrite an existing repository by providing the -f option. @@ -185,64 +186,64 @@ can be disabled by setting the configuration yadm.auto-perms to "false". + transcrypt options + If transcrypt is installed, this command allows you to pass + options directly to transcrypt, with the environment configured + to use the yadm repository. + + transcrypt enables transparent encryption and decryption of + files in a git repository. You can read + https://github.com/elasticdog/transcrypt for details. + upgrade - Version 2 of yadm uses a different directory for storing your - configurations. When you start to use version 2 for the first - time, you may see warnings about moving your data to this new - directory. The easiest way to accomplish this is by running - "yadm upgrade". This command will start by moving your yadm - repo to the new path. Next it will move any configuration data - to the new path. If the configurations are tracked within your - yadm repo, this command will "stage" the renaming of those files - in the repo's index. Upgrading will also re-initialize all sub- - modules you have added (otherwise they will be broken when the - repo moves). After running "yadm upgrade", you should run "yadm - status" to review changes which have been staged, and commit - them to your repository. - - You can read https://yadm.io/docs/upgrade_from_1 for more infor- + Version 3 of yadm uses a different directory for storing data. + When you start to use version 3 for the first time, you may see + warnings about moving your data to this new directory. The eas- + iest way to accomplish this is by running "yadm upgrade". This + command will start by moving your yadm repo to the new path. + Next it will move any archive data. If the archive is tracked + within your yadm repo, this command will "stage" the renaming of + that file in the repo's index. + + Upgrading will attempt to de-initialize and re-initialize your + submodules. If your submodules cannot be de-initialized, the + upgrade will fail. The most common reason submodules will fail + to de-initialize is because they have local modifications. If + you are willing to lose the local modifications to those submod- + ules, you can use the -f option with the "upgrade" command to + force the de-initialization. + + After running "yadm upgrade", you should run "yadm status" to + review changes which have been staged, and commit them to your + repository. + + You can read https://yadm.io/docs/upgrade_from_2 for more infor- mation. version Print the version of yadm. -## COMPATIBILITY - Beginning with version 2.0.0, yadm introduced a couple major changes - which may require you to adjust your configurations. See the upgrade - command for help making those adjustments. - - First, yadm now uses the "XDG Base Directory Specification" to find its - configurations. You can read https://yadm.io/docs/upgrade_from_1 for - more information. - - Second, the naming conventions for alternate files have been changed. - You can read https://yadm.io/docs/alternates for more information. - - If you want to retain the old functionality, you can set an environment - variable, YADM_COMPATIBILITY=1. Doing so will automatically use the - old yadm directory, and process alternates the same as the pre-2.0.0 - version. This compatibility mode is deprecated, and will be removed in - future versions. This mode exists solely for transitioning to the new - paths and naming of alternates. - - ## OPTIONS - yadm supports a set of universal options that alter the paths it uses. - The default paths are documented in the FILES section. Any path speci- - fied by these options must be fully qualified. If you always want to - override one or more of these paths, it may be useful to create an - alias for the yadm command. For example, the following alias could be + yadm supports a set of universal options that alter the paths it uses. + The default paths are documented in the FILES section. Any path speci- + fied by these options must be fully qualified. If you always want to + override one or more of these paths, it may be useful to create an + alias for the yadm command. For example, the following alias could be used to override the repository directory. alias yadm='yadm --yadm-repo /alternate/path/to/repo' - The following is the full list of universal options. Each option + The following is the full list of universal options. Each option should be followed by a fully qualified path. -Y,--yadm-dir - Override the yadm directory. yadm stores its data relative to - this directory. + Override the yadm directory. yadm stores its configurations + relative to this directory. + + --yadm-data + Override the yadm data directory. yadm stores its data relative + to this directory. --yadm-repo Override the location of the yadm repository. @@ -276,10 +277,6 @@ bolic links. This might be desirable, because some systems may not properly support symlinks. - NOTE: The deprecated yadm.cygwin-copy option used by older ver- - sions of yadm has been replaced by yadm.alt-copy. The old - option will be removed in the next version of yadm. - yadm.auto-alt Disable the automatic linking described in the section ALTER- NATES. If disabled, you may still run "yadm alt" manually to @@ -299,42 +296,63 @@ Disable the automatic creating of private directories described in the section PERMISSIONS. + yadm.cipher + Configure which encryption system is used by the encrypt/decrypt + commands. Valid options are "gpg" and "openssl". The default is + "gpg". Detailed information can be found in the section ENCRYP- + TION. + yadm.git-program - Specify an alternate program to use instead of "git". By + Specify an alternate program to use instead of "git". By default, the first "git" found in $PATH is used. yadm.gpg-perms - Disable the permission changes to $HOME/.gnupg/*. This feature + Disable the permission changes to $HOME/.gnupg/*. This feature is enabled by default. yadm.gpg-program - Specify an alternate program to use instead of "gpg". By + Specify an alternate program to use instead of "gpg". By default, the first "gpg" found in $PATH is used. yadm.gpg-recipient Asymmetrically encrypt files with a gpg public/private key pair. - Provide a "key ID" to specify which public key to encrypt with. + Provide a "key ID" to specify which public key to encrypt with. The key must exist in your public keyrings. Multiple recipients - can be specified (separated by space). If left blank or not - provided, symmetric encryption is used instead. If set to - "ASK", gpg will interactively ask for recipients. See the - ENCRYPTION section for more details. This feature is disabled + can be specified (separated by space). If left blank or not + provided, symmetric encryption is used instead. If set to + "ASK", gpg will interactively ask for recipients. See the + ENCRYPTION section for more details. This feature is disabled by default. + yadm.openssl-ciphername + Specify which cipher should be used by openssl. "aes-256-cbc" + is used by default. + + yadm.openssl-old + Newer versions of openssl support the pbkdf2 key derivation + function. This is used by default. If this configuration is set + to "true", openssl operations will use options compatible with + older versions of openssl. If you change this option, you will + need to recreate your encrypted archive. + + yadm.openssl-program + Specify an alternate program to use instead of "openssl". By + default, the first "openssl" found in $PATH is used. + yadm.ssh-perms Disable the permission changes to $HOME/.ssh/*. This feature is enabled by default. - The following four "local" configurations are not stored in the + The following four "local" configurations are not stored in the $HOME/.config/yadm/config, they are stored in the local repository. local.class - Specify a class for the purpose of symlinking alternate files. + Specify a class for the purpose of symlinking alternate files. By default, no class will be matched. local.hostname - Override the hostname for the purpose of symlinking alternate + Override the hostname for the purpose of symlinking alternate files. local.os @@ -349,9 +367,9 @@ to have an automated way of choosing an alternate version of a file for a different operating system, host, user, etc. - yadm will automatically create a symbolic link to the appropriate ver- - sion of a file, when a valid suffix is appended to the filename. The - suffix contains the conditions that must be met for that file to be + yadm will automatically create a symbolic link to the appropriate ver- + sion of a file, when a valid suffix is appended to the filename. The + suffix contains the conditions that must be met for that file to be used. The suffix begins with "##", followed by any number of conditions sepa- @@ -359,9 +377,9 @@ ##<condition>[,<condition>,...] - Each condition is an attribute/value pair, separated by a period. Some - conditions do not require a "value", and in that case, the period and - value can be omitted. Most attributes can be abbreviated as a single + Each condition is an attribute/value pair, separated by a period. Some + conditions do not require a "value", and in that case, the period and + value can be omitted. Most attributes can be abbreviated as a single letter. <attribute>[.<value>] @@ -371,25 +389,25 @@ template, t - Valid when the value matches a supported template processor. + Valid when the value matches a supported template processor. See the TEMPLATES section for more details. user, u - Valid if the value matches the current user. Current user is + Valid if the value matches the current user. Current user is calculated by running id -u -n. distro, d - Valid if the value matches the distro. Distro is calculated by - running lsb_release -si or by inspecting the ID from /etc/os- + Valid if the value matches the distro. Distro is calculated by + running lsb_release -si or by inspecting the ID from /etc/os- release. - os, o Valid if the value matches the OS. OS is calculated by running + os, o Valid if the value matches the OS. OS is calculated by running uname -s. class, c Valid if the value matches the local.class configuration. Class must be manually set using yadm config local.class <class>. See - the CONFIGURATION section for more details about setting + the CONFIGURATION section for more details about setting local.class. hostname, h @@ -399,6 +417,12 @@ default Valid when no other alternate is valid. + extension, e + A special "condition" that doesn't affect the selection process. + Its purpose is instead to allow the alternate file to end with a + certain extension to e.g. make editors highlight the content + properly. + NOTE: The OS for "Windows Subsystem for Linux" is reported as "WSL", even though uname identifies as "Linux". @@ -495,23 +519,32 @@ most *nix systems. To use this processor, specify the value of "default" or just leave the value off (e.g. "##template"). - j2cli To use the j2cli Jinja template processor, specify the value of + ESH ESH is a template processor written in POSIX compliant shell. It + allows executing shell commands within templates. This can be + used to reference your own configurations within templates, for + example: + + <% yadm config mysection.myconfig %> + + To use the ESH template processor, specify the value of "esh" + + j2cli To use the j2cli Jinja template processor, specify the value of "j2" or "j2cli". envtpl To use the envtpl Jinja template processor, specify the value of "j2" or "envtpl". - NOTE: Specifying "j2" as the processor will attempt to use j2cli or + NOTE: Specifying "j2" as the processor will attempt to use j2cli or envtpl, whichever is available. - If the template processor specified is available, templates will be + If the template processor specified is available, templates will be processed to create or overwrite files. - During processing, the following variables are available in the tem- + During processing, the following variables are available in the tem- plate: - Default Jinja Description + Default Jinja or ESH Description ------------- ------------- -------------------------- yadm.class YADM_CLASS Locally defined yadm class yadm.distro YADM_DISTRO lsb_release -si @@ -520,40 +553,53 @@ yadm.user YADM_USER id -u -n yadm.source YADM_SOURCE Template filename - NOTE: The OS for "Windows Subsystem for Linux" is reported as "WSL", + NOTE: The OS for "Windows Subsystem for Linux" is reported as "WSL", even though uname identifies as "Linux". - NOTE: If lsb_release is not available, DISTRO will be the ID specified + NOTE: If lsb_release is not available, DISTRO will be the ID specified in /etc/os-release. Examples: whatever##template with the following content - {% if yadm.user == 'harvey' %} + {% if yadm.user == "harvey" %} config={{yadm.class}}-{{yadm.os}} {% else %} config=dev-whatever + {% include "whatever.extra" %} {% endif %} - would output a file named whatever with the following content if the + would output a file named whatever with the following content if the user is "harvey": config=work-Linux - and the following otherwise: + and the following otherwise (if whatever.extra contains admin=false): config=dev-whatever + admin=false - An equivalent Jinja template named whatever##template.j2 would look + An equivalent Jinja template named whatever##template.j2 would look like: {% if YADM_USER == 'harvey' -%} config={{YADM_CLASS}}-{{YADM_OS}} {% else -%} config=dev-whatever + {% include 'whatever.extra' %} {% endif -%} + An equivalent ESH templated named whatever##template.esh would look + like: + + <% if [ "$YADM_USER" = "harvey" ]; then -%> + config=<%= $YADM_CLASS %>-<%= $YADM_OS %> + <% else -%> + config=dev-whatever + <%+ whatever.extra %> + <% fi -%> + ## ENCRYPTION It can be useful to manage confidential files, like SSH or GPG keys, @@ -561,7 +607,9 @@ into a Git repository, which often resides on a public system. yadm can make it easy to encrypt and decrypt a set of files so the encrypted version can be maintained in the Git repository. This feature will - only work if the gpg(1) command is available. + only work if a supported tool is available. Both gpg(1) and openssl(1) + are supported. gpg is used by default, but openssl can be configured + with the yadm.cypher configuration. To use this feature, a list of patterns must be created and saved as $HOME/.config/yadm/encrypt. This list of patterns should be relative @@ -579,9 +627,9 @@ The yadm encrypt command will find all files matching the patterns, and prompt for a password. Once a password has confirmed, the matching - files will be encrypted and saved as $HOME/.config/yadm/files.gpg. The - patterns and files.gpg should be added to the yadm repository so they - are available across multiple systems. + files will be encrypted and saved as $HOME/.local/share/yadm/archive. + The "encrypt" and "archive" files should be added to the yadm reposi- + tory so they are available across multiple systems. To decrypt these files later, or on another system run yadm decrypt and provide the correct password. After files are decrypted, permissions @@ -598,13 +646,17 @@ This is to prevent accidentally committing sensitive data to the repos- itory. This can be disabled using the yadm.auto-exclude configuration. - Using git-crypt + Using transcrypt or git-crypt A completely separate option for encrypting data is to install and use - git-crypt. Once installed, you can run git-crypt commands for the yadm - repo by running yadm git-crypt. git-crypt enables transparent encryp- - tion and decryption of files in a git repository. You can read - https://github.com/AGWA/git-crypt for details. + transcrypt or git-crypt. Once installed, you can use these tools by + running yadm transcrypt or yadm git-crypt. These tools enables trans- + parent encryption and decryption of files in a git repository. See the + following web sites for more information: + + - https://github.com/elasticdog/transcrypt + + - https://github.com/AGWA/git-crypt @@ -614,7 +666,7 @@ automatically update the permissions of some file paths. The "group" and "others" permissions will be removed from the following files: - - $HOME/.config/yadm/files.gpg + - $HOME/.local/share/yadm/archive - All files matching patterns in $HOME/.config/yadm/encrypt @@ -683,30 +735,40 @@ a fully qualified path, this directory will be $XDG_CONFIG_HOME/yadm. Otherwise it will be $HOME/.config/yadm. + Similarly, yadm's data files are relative to the "yadm data directory". + yadm uses the "XDG Base Directory Specification" to determine this + directory. If the environment variable $XDG_DATA_HOME is defined as a + fully qualified path, this directory will be $XDG_DATA_HOME/yadm. Oth- + erwise it will be $HOME/.local/share/yadm. + The following are the default paths yadm uses for its own data. Most of these paths can be altered using universal options. See the OPTIONS section for details. $HOME/.config/yadm - The yadm directory. By default, all data yadm stores is relative - to this directory. + The yadm directory. By default, all configs yadm stores is rela- + tive to this directory. + + $HOME/.local/share/yadm + The yadm data directory. By default, all data yadm stores is + relative to this directory. $YADM_DIR/config Configuration file for yadm. $YADM_DIR/alt - This is a directory to keep "alternate files" without having - them side-by-side with the resulting symlink or processed tem- - plate. Alternate files placed in this directory will be created + This is a directory to keep "alternate files" without having + them side-by-side with the resulting symlink or processed tem- + plate. Alternate files placed in this directory will be created relative to $HOME instead. - $YADM_DIR/repo.git + $YADM_DATA/repo.git Git repository used by yadm. $YADM_DIR/encrypt List of globs used for encrypt/decrypt - $YADM_DIR/files.gpg + $YADM_DATA/archive All files encrypted with yadm encrypt are stored in this file. @@ -726,7 +788,7 @@ echo .ssh/*.key >> $HOME/.config/yadm/encrypt Add a new pattern to the list of encrypted files - yadm encrypt ; yadm add ~/.config/yadm/files.gpg ; yadm commit + yadm encrypt ; yadm add ~/.local/share/yadm/archive ; yadm commit Commit a new set of encrypted files @@ -741,7 +803,7 @@ ## SEE ALSO - git(1), gpg(1) + git(1), gpg(1) openssl(1) transcrypt(1) git-crypt(1) https://yadm.io/ @@ -1,7 +1,7 @@ %{!?_pkgdocdir: %global _pkgdocdir %{_docdir}/%{name}-%{version}} Name: yadm Summary: Yet Another Dotfiles Manager -Version: 2.4.0 +Version: 3.0.2 Group: Development/Tools Release: 1%{?dist} URL: https://yadm.io @@ -29,7 +29,7 @@ encrypted before they are included in the repository. # this is done to allow paths other than yadm-x.x.x (for example, when building # from branches instead of release tags) -cd *yadm-* +test -f yadm || cd *yadm-* %{__mkdir} -p %{buildroot}%{_bindir} %{__cp} yadm %{buildroot}%{_bindir} |