support annex.shared-sop-command for encryption=shared

This works well, and it interoperates with gpg in my testing (although some
SOP commands might choose to use a profile that does not so caveat emptor).

Note that for creating the Cipher, gpg --gen-random is still used. SOP
does not have an eqivilant, and as long as the user has gpg around,
which seems likely, it doesn't matter that it uses gpg here, it's not being
used for encryption. That seemed better than implementing a second way
to get high quality entropy, at least for now.

The need for the sop command to run in an empty directory has each call
to encrypt and decrypt creating a new temporary directory. That is some
unncessary overhead, though probably swamped by the overhead of running
the sop command. This could be improved in the future by passing an
already empty directory to them, or a sufficiently empty directory
(.git/annex/tmp would probably suffice).

Sponsored-by: Brett Eisenberg on Patreon

7 files changed, 116 insertions, 68 deletions

diff --git a/CHANGELOG b/CHANGELOG
index 7e2796be3f..b539ee6212 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,10 +2,14 @@ git-annex (10.20231228) UNRELEASED; urgency=medium
 
   * info: Added "annex sizes of repositories" table to the overall display.
   * import: Sped up import from special remotes.
-  * assistant: When generating a gpg secret key, avoid hardcoding the
-    key algorithm and size.
+  * Support using commands that implement the Stateless OpenPGP command line
+    interface, as an alternative to gpg.
+    Currently only supported for encryption=shared special remotes,
+    when annex.shared-sop-command is configured.
   * test: Test a specified Stateless OpenPGP command when
     run with eg --test-git-config annex.shared-sop-command=sqop
+  * assistant: When generating a gpg secret key, avoid hardcoding the
+    key algorithm and size.
 
  -- Joey Hess <id@joeyh.name>  Fri, 29 Dec 2023 11:52:06 -0400
 
diff --git a/Crypto.hs b/Crypto.hs
index 288042b1be..3fa6d781d1 100644
--- a/Crypto.hs
+++ b/Crypto.hs
@@ -1,7 +1,6 @@
 {- git-annex crypto
  -
- - Currently using gpg; could later be modified to support different
- - crypto backends if necessary.
+ - Currently using gpg by default, or optionally stateless OpenPGP.
  -
  - Copyright 2011-2024 Joey Hess <id@joeyh.name>
  -
@@ -32,7 +31,7 @@ module Crypto (
 	readBytesStrictly,
 	encrypt,
 	decrypt,
-	LensGpgEncParams(..),
+	LensEncParams(..),
 
 	prop_HmacSha1WithCipher_sane
 ) where
@@ -40,6 +39,7 @@ module Crypto (
 import qualified Data.ByteString as S
 import qualified Data.ByteString.Lazy as L
 import Control.Monad.IO.Class
+import qualified Data.ByteString.Short as S (toShort)
 
 import Annex.Common
 import qualified Utility.Gpg as Gpg
@@ -48,29 +48,33 @@ import Types.Crypto
 import Types.Remote
 import Types.Key
 import Annex.SpecialRemote.Config
-import qualified Data.ByteString.Short as S (toShort)
+import Utility.Tmp.Dir
+
+{- The number of bytes of entropy used to generate a Cipher.
+ -
+ - Since a Cipher is base-64 encoded, the actual size of a Cipher
+ - is larger than this. 512 bytes of date base-64 encodes to 684
+ - characters.
+ -}
+cipherSize :: Int
+cipherSize = 512
 
 {- The beginning of a Cipher is used for MAC'ing; the remainder is used
- - as the GPG symmetric encryption passphrase when using the hybrid
- - scheme. Note that the cipher itself is base-64 encoded, hence the
- - string is longer than 'cipherSize': 683 characters, padded to 684.
+ - as the symmetric encryption passphrase.
  -
- - The 256 first characters that feed the MAC represent at best 192
- - bytes of entropy.  However that's more than enough for both the
- - default MAC algorithm, namely HMAC-SHA1, and the "strongest"
+ - Due to the base-64 encoding of the Cipher, the beginning 265 characters
+ - represent at best 192 bytes of entropy. However that's more than enough
+ - for both the default MAC algorithm, namely HMAC-SHA1, and the "strongest"
  - currently supported, namely HMAC-SHA512, which respectively need
  - (ideally) 64 and 128 bytes of entropy.
  -
- - The remaining characters (320 bytes of entropy) is enough for GnuPG's
- - symmetric cipher; unlike weaker public key crypto, the key does not
- - need to be too large.
+ - The remaining characters (320 bytes of entropy) is enough for
+ - the symmetric encryption passphrase; unlike weaker public key crypto,
+ - that does not need to be too large.
  -}
 cipherBeginning :: Int
 cipherBeginning = 256
 
-cipherSize :: Int
-cipherSize = 512
-
 cipherPassphrase :: Cipher -> S.ByteString
 cipherPassphrase (Cipher c) = S.drop cipherBeginning c
 cipherPassphrase (MacOnlyCipher _) = giveup "MAC-only cipher"
@@ -80,7 +84,7 @@ cipherMac (Cipher c) = S.take cipherBeginning c
 cipherMac (MacOnlyCipher c) = c
 
 {- Creates a new Cipher, encrypted to the specified key id. -}
-genEncryptedCipher :: LensGpgEncParams c => Gpg.GpgCmd -> c -> Gpg.KeyId -> EncryptedCipherVariant -> Bool -> IO StorableCipher
+genEncryptedCipher :: LensEncParams c => Gpg.GpgCmd -> c -> Gpg.KeyId -> EncryptedCipherVariant -> Bool -> IO StorableCipher
 genEncryptedCipher cmd c keyid variant highQuality = do
 	ks <- Gpg.findPubKeys cmd keyid
 	random <- Gpg.genRandom cmd highQuality size
@@ -106,7 +110,7 @@ genSharedPubKeyCipher cmd keyid highQuality = do
 {- Updates an existing Cipher, making changes to its keyids.
  -
  - When the Cipher is encrypted, re-encrypts it. -}
-updateCipherKeyIds :: LensGpgEncParams encparams => Gpg.GpgCmd -> encparams -> [(Bool, Gpg.KeyId)] -> StorableCipher -> IO StorableCipher
+updateCipherKeyIds :: LensEncParams encparams => Gpg.GpgCmd -> encparams -> [(Bool, Gpg.KeyId)] -> StorableCipher -> IO StorableCipher
 updateCipherKeyIds _ _ _ SharedCipher{} = giveup "Cannot update shared cipher"
 updateCipherKeyIds _ _ [] c = return c
 updateCipherKeyIds cmd encparams changes encipher@(EncryptedCipher _ variant ks) = do
@@ -130,7 +134,7 @@ updateCipherKeyIds' cmd changes (KeyIds ks) = do
 	listKeyIds = concat <$$> mapM (keyIds <$$> Gpg.findPubKeys cmd)
 
 {- Encrypts a Cipher to the specified KeyIds. -}
-encryptCipher :: LensGpgEncParams c => Gpg.GpgCmd -> c -> Cipher -> EncryptedCipherVariant -> KeyIds -> IO StorableCipher
+encryptCipher :: LensEncParams c => Gpg.GpgCmd -> c -> Cipher -> EncryptedCipherVariant -> KeyIds -> IO StorableCipher
 encryptCipher cmd c cip variant (KeyIds ks) = do
 	-- gpg complains about duplicate recipient keyids
 	let ks' = nub $ sort ks
@@ -147,10 +151,10 @@ encryptCipher cmd c cip variant (KeyIds ks) = do
 		MacOnlyCipher x -> x
 
 {- Decrypting an EncryptedCipher is expensive; the Cipher should be cached. -}
-decryptCipher :: LensGpgEncParams c => Gpg.GpgCmd -> c -> StorableCipher -> IO Cipher
+decryptCipher :: LensEncParams c => Gpg.GpgCmd -> c -> StorableCipher -> IO Cipher
 decryptCipher cmd c cip = decryptCipher' cmd Nothing c cip
 
-decryptCipher' :: LensGpgEncParams c => Gpg.GpgCmd -> Maybe [(String, String)] -> c -> StorableCipher -> IO Cipher
+decryptCipher' :: LensEncParams c => Gpg.GpgCmd -> Maybe [(String, String)] -> c -> StorableCipher -> IO Cipher
 decryptCipher' _ _ _ (SharedCipher t) = return $ Cipher t
 decryptCipher' _ _ _ (SharedPubKeyCipher t _) = return $ MacOnlyCipher t
 decryptCipher' cmd environ c (EncryptedCipher t variant _) =
@@ -198,16 +202,23 @@ readBytesStrictly a h = liftIO (S.hGetContents h) >>= a
 
 {- Runs a Feeder action, that generates content that is symmetrically
  - encrypted with the Cipher (unless it is empty, in which case
- - public-key encryption is used) using the given gpg options, and then
- - read by the Reader action. 
+ - public-key encryption is used), and then read by the Reader action. 
  -
- - Note that the Reader must fully consume gpg's input before returning.
+ - Note that the Reader must fully consume all input before returning.
  -}
-encrypt :: (MonadIO m, MonadMask m, LensGpgEncParams c) => Gpg.GpgCmd -> c -> Cipher -> Feeder -> Reader m a -> m a
-encrypt cmd c cipher = case cipher of
-	Cipher{} -> Gpg.feedRead cmd (params ++ Gpg.stdEncryptionParams True) $
-			cipherPassphrase cipher
-	MacOnlyCipher{} -> Gpg.feedRead' cmd $ params ++ Gpg.stdEncryptionParams False
+encrypt :: (MonadIO m, MonadMask m, LensEncParams c) => Gpg.GpgCmd -> c -> Cipher -> Feeder -> Reader m a -> m a
+encrypt gpgcmd c cipher feeder reader = case cipher of
+	Cipher{} -> 
+		let passphrase = cipherPassphrase cipher
+		in case statelessOpenPGPCommand c of
+			Just sopcmd -> withTmpDir "sop" $ \d ->
+				SOP.encryptSymmetric sopcmd passphrase
+					(SOP.EmptyDirectory d)
+					(statelessOpenPGPProfile c)
+					(SOP.Armoring False)
+					feeder reader
+			Nothing -> Gpg.feedRead gpgcmd (params ++ Gpg.stdEncryptionParams True) passphrase feeder reader
+	MacOnlyCipher{} -> Gpg.feedRead' gpgcmd (params ++ Gpg.stdEncryptionParams False) feeder reader
   where
 	params = getGpgEncParams c
 
@@ -215,12 +226,19 @@ encrypt cmd c cipher = case cipher of
  - Cipher (or using a private key if the Cipher is empty), and read by the
  - Reader action.
  -
- - Note that the Reader must fully consume gpg's input before returning.
+ - Note that the Reader must fully consume all input before returning.
  - -}
-decrypt :: (MonadIO m, MonadMask m, LensGpgEncParams c) => Gpg.GpgCmd -> c -> Cipher -> Feeder -> Reader m a -> m a
-decrypt cmd c cipher = case cipher of
-	Cipher{} -> Gpg.feedRead cmd params $ cipherPassphrase cipher
-	MacOnlyCipher{} -> Gpg.feedRead' cmd params
+decrypt :: (MonadIO m, MonadMask m, LensEncParams c) => Gpg.GpgCmd -> c -> Cipher -> Feeder -> Reader m a -> m a
+decrypt cmd c cipher feeder reader = case cipher of
+	Cipher{} -> 
+		let passphrase = cipherPassphrase cipher
+		in case statelessOpenPGPCommand c of
+			Just sopcmd -> withTmpDir "sop" $ \d ->
+				SOP.decryptSymmetric sopcmd passphrase
+					(SOP.EmptyDirectory d)
+					feeder reader
+			Nothing -> Gpg.feedRead cmd params passphrase feeder reader
+	MacOnlyCipher{} -> Gpg.feedRead' cmd params feeder reader
   where
 	params = Param "--decrypt" : getGpgDecParams c
 
@@ -235,19 +253,26 @@ prop_HmacSha1WithCipher_sane = known_good == macWithCipher' HmacSha1 "foo" "bar"
   where
 	known_good = "46b4ec586117154dacd49d664e5d63fdc88efb51"
 
-class LensGpgEncParams a where
-	{- Base parameters for encrypting. Does not include specification
+class LensEncParams a where
+	{- Base gpg parameters for encrypting. Does not include specification
 	 - of recipient keys. -}
 	getGpgEncParamsBase :: a -> [CommandParam]
-	{- Parameters for encrypting. When the remote is configured to use
+	{- Gpg parameters for encrypting. When the remote is configured to use
 	 - public-key encryption, includes specification of recipient keys. -}
 	getGpgEncParams :: a -> [CommandParam]
-	{- Parameters for decrypting. -}
+	{- Gpg parameters for decrypting. -}
 	getGpgDecParams :: a -> [CommandParam]
+	{- Set when stateless OpenPGP should be used rather than gpg.
+	 - It is currently only used for SharedEncryption and not the other
+	 - schemes which use public keys. -}
+	statelessOpenPGPCommand :: a -> Maybe SOP.SOPCmd
+	{- When using stateless OpenPGP, this may be set to a profile
+	 - which should be used instead of the default. -}
+	statelessOpenPGPProfile :: a -> Maybe SOP.SOPProfile
 
 {- Extract the GnuPG options from a pair of a Remote Config and a Remote
  - Git Config. -}
-instance LensGpgEncParams (ParsedRemoteConfig, RemoteGitConfig) where
+instance LensEncParams (ParsedRemoteConfig, RemoteGitConfig) where
 	getGpgEncParamsBase (_c,gc) = map Param (remoteAnnexGnupgOptions gc)
 	getGpgEncParams (c,gc) = getGpgEncParamsBase (c,gc) ++
  		{- When the remote is configured to use public-key encryption,
@@ -261,9 +286,21 @@ instance LensGpgEncParams (ParsedRemoteConfig, RemoteGitConfig) where
 					getRemoteConfigValue pubkeysField c
 			_ -> []
 	getGpgDecParams (_c,gc) = map Param (remoteAnnexGnupgDecryptOptions gc)
+	statelessOpenPGPCommand (c,gc) = case remoteAnnexSharedSOPCommand gc of
+		Nothing -> Nothing
+		Just sopcmd ->
+			{- So far stateless OpenPGP is only supported
+			 - for SharedEncryption, not other encryption
+			 - methods that involve public keys. -}
+			case getRemoteConfigValue encryptionField c of
+				Just SharedEncryption -> Just sopcmd
+				_ -> Nothing
+	statelessOpenPGPProfile (_c,gc) = remoteAnnexSharedSOPProfile gc
 
 {- Extract the GnuPG options from a Remote. -}
-instance LensGpgEncParams (RemoteA a) where
+instance LensEncParams (RemoteA a) where
 	getGpgEncParamsBase r = getGpgEncParamsBase (config r, gitconfig r)
 	getGpgEncParams r = getGpgEncParams (config r, gitconfig r)
 	getGpgDecParams r = getGpgDecParams (config r, gitconfig r)
+	statelessOpenPGPCommand r = statelessOpenPGPCommand (config r, gitconfig r)
+	statelessOpenPGPProfile r  = statelessOpenPGPProfile (config r, gitconfig r)
diff --git a/Remote/Helper/Chunked.hs b/Remote/Helper/Chunked.hs
index 3e7f8a47e1..29b97b761a 100644
--- a/Remote/Helper/Chunked.hs
+++ b/Remote/Helper/Chunked.hs
@@ -115,7 +115,7 @@ numChunks = pred . fromJust . fromKey keyChunkNum . fst . nextChunkKeyStream
  - writes a whole L.ByteString at a time.
  -}
 storeChunks
-	:: LensGpgEncParams encc
+	:: LensEncParams encc
 	=> UUID
 	-> ChunkConfig
 	-> EncKey
@@ -250,7 +250,7 @@ removeChunks remover u chunkconfig encryptor k = do
  - Handles decrypting the content when encryption is used.
  -}
 retrieveChunks 
-	:: LensGpgEncParams encc
+	:: LensEncParams encc
 	=> Retriever
 	-> UUID
 	-> VerifyConfig
@@ -391,7 +391,7 @@ retrieveChunks retriever u vc chunkconfig encryptor basek dest basep enc encc
  - into place. (And it may even already be in the right place..)
  -}
 writeRetrievedContent
-	:: LensGpgEncParams encc
+	:: LensEncParams encc
 	=> FilePath
 	-> Maybe (Cipher, EncKey)
 	-> encc
diff --git a/Types/Crypto.hs b/Types/Crypto.hs
index 38f4daeb10..2b3b065d71 100644
--- a/Types/Crypto.hs
+++ b/Types/Crypto.hs
@@ -35,6 +35,7 @@ data EncryptionMethod
 	| HybridEncryption
 	deriving (Typeable, Eq)
 
+-- A base-64 encoded random value used for encryption.
 -- XXX ideally, this would be a locked memory region
 data Cipher = Cipher ByteString | MacOnlyCipher ByteString
 
diff --git a/Types/GitConfig.hs b/Types/GitConfig.hs
index b8158ea8e6..0c531fbf06 100644
--- a/Types/GitConfig.hs
+++ b/Types/GitConfig.hs
@@ -47,7 +47,7 @@ import Types.View
 import Config.DynamicConfig
 import Utility.HumanTime
 import Utility.Gpg (GpgCmd, mkGpgCmd)
-import Utility.StatelessOpenPGP (SOPCmd(..))
+import Utility.StatelessOpenPGP (SOPCmd(..), SOPProfile(..))
 import Utility.ThreadScheduler (Seconds(..))
 import Utility.Url (Scheme, mkScheme)
 
@@ -374,7 +374,7 @@ data RemoteGitConfig = RemoteGitConfig
 	, remoteAnnexGnupgOptions :: [String]
 	, remoteAnnexGnupgDecryptOptions :: [String]
 	, remoteAnnexSharedSOPCommand :: Maybe SOPCmd
-	, remoteAnnexSharedSOPProfile :: Maybe String
+	, remoteAnnexSharedSOPProfile :: Maybe SOPProfile
 	, remoteAnnexRsyncUrl :: Maybe String
 	, remoteAnnexBupRepo :: Maybe String
 	, remoteAnnexBorgRepo :: Maybe String
@@ -444,7 +444,8 @@ extractRemoteGitConfig r remotename = do
 		, remoteAnnexGnupgDecryptOptions = getoptions "gnupg-decrypt-options"
 		, remoteAnnexSharedSOPCommand = SOPCmd <$>
 			notempty (getmaybe "shared-sop-command")
-		, remoteAnnexSharedSOPProfile = notempty $ getmaybe "shared-sop-profile"
+		, remoteAnnexSharedSOPProfile = SOPProfile <$>
+			notempty (getmaybe "shared-sop-profile")
 		, remoteAnnexRsyncUrl = notempty $ getmaybe "rsyncurl"
 		, remoteAnnexBupRepo = getmaybe "buprepo"
 		, remoteAnnexBorgRepo = getmaybe "borgrepo"
diff --git a/Utility/StatelessOpenPGP.hs b/Utility/StatelessOpenPGP.hs
index 35b4b17ccc..7a4c2063ca 100644
--- a/Utility/StatelessOpenPGP.hs
+++ b/Utility/StatelessOpenPGP.hs
@@ -9,9 +9,10 @@
 
 module Utility.StatelessOpenPGP (
 	SOPCmd(..),
-	SopSubCmd,
+	SOPSubCmd,
+	SOPProfile(..),
 	Password,
-	Profile,
+	EmptyDirectory(..),
 	Armoring(..),
 	encryptSymmetric,
 	decryptSymmetric,
@@ -37,18 +38,18 @@ import qualified Data.ByteString as B
 newtype SOPCmd = SOPCmd { unSOPCmd :: String }
 
 {- The subcommand to run eg encrypt. -}
-type SopSubCmd = String
+type SOPSubCmd = String
+
+newtype SOPProfile = SOPProfile String
 
 {- Note that SOP requires passwords to be UTF-8 encoded, and that they
  - may try to trim trailing whitespace. They may also forbid leading
  - whitespace, or forbid some non-printing characters. -}
 type Password = B.ByteString
 
-type Profile = String
-
 newtype Armoring = Armoring Bool
 
-{- The path to an empty temporary directory.
+{- The path to a sufficiently empty directory.
  -
  - This is unfortunately needed because of an infelicity in the SOP
  - standard, as documented in section 9.9 "Be Careful with Special
@@ -61,6 +62,9 @@ newtype Armoring = Armoring Bool
  - special designators, an empty directory has to be provided, and the
  - command is run in that directory. Of course, this necessarily means
  - that any relative paths passed to the command have to be made absolute.
+ -
+ - The directory does not really have to be empty, it just needs to be one
+ - that should not contain any files with names starting with "@".
  -}
 newtype EmptyDirectory = EmptyDirectory FilePath
 
@@ -70,7 +74,7 @@ encryptSymmetric
 	=> SOPCmd
 	-> Password
 	-> EmptyDirectory
-	-> Maybe Profile
+	-> Maybe SOPProfile
 	-> Armoring
 	-> (Handle -> IO ())
 	-> (Handle -> m a)
@@ -84,7 +88,8 @@ encryptSymmetric sopcmd password emptydirectory mprofile armoring feeder reader
 			Armoring True -> Nothing
 		, Just "--as=binary"
 		, case mprofile of
-			Just profile -> Just $ "--profile=" ++ profile
+			Just (SOPProfile profile) -> 
+				Just $ "--profile=" ++ profile
 			Nothing -> Nothing
 		]
 
@@ -121,7 +126,7 @@ test_encrypt_decrypt_Symmetric a b password armoring v = catchBoolIO $
 feedRead
 	:: (MonadIO m, MonadMask m)
 	=> SOPCmd
-	-> SopSubCmd
+	-> SOPSubCmd
 	-> [CommandParam]
 	-> Password
 	-> EmptyDirectory
@@ -166,7 +171,7 @@ feedRead cmd subcmd params password emptydirectory feeder reader = do
 feedRead'
 	:: (MonadIO m, MonadMask m)
 	=> SOPCmd
-	-> SopSubCmd
+	-> SOPSubCmd
 	-> [CommandParam]
 	-> Maybe EmptyDirectory
 	-> (Handle -> IO ())
diff --git a/doc/todo/support_using_Stateless_OpenPGP_instead_of_gpg.mdwn b/doc/todo/support_using_Stateless_OpenPGP_instead_of_gpg.mdwn
index 75bebba247..e6600e88a5 100644
--- a/doc/todo/support_using_Stateless_OpenPGP_instead_of_gpg.mdwn
+++ b/doc/todo/support_using_Stateless_OpenPGP_instead_of_gpg.mdwn
@@ -19,6 +19,14 @@ That example uses symmetric encryption, which is what git-annex uses
 for encryption=shared. So git-annex could use this or gpg to access the
 same encrypted special remote.
 
+Update: That's implemented now, when annex.shared-sop-command is configured
+it will be used for encryption=shared special remotes. It interoperates
+fine with using gpg, as long as the sop command uses a compatable profile
+(setting annex.shared-sop-profile = rfc4880 is probably a good idea).
+
+Leaving this todo open because there are other encryption schemes than 
+encryption=shared, for which using sop is not yet supported.
+
 For the public key encryption used by the other encryption= schemes,
 sop would be harder to use, because it does not define any location
 to store private keys. Though it is possible to export gpg private keys
@@ -39,14 +47,6 @@ It can detect if a password is needed by trying the sop operation without a
 password and checking for an exit code of 67.
 See [this issue on sop password prompting](https://gitlab.com/dkg/openpgp-stateless-cli/-/issues/64)
 
-git-annex also uses gpg to generate random data for an encryption cipher
-when setting up an encrypted special remote. Of course there are other ways
-to generate random data, but it does make sense to use gpg or a similar
-tool for this particular random data generation, since it's effectively a
-secret key. And genRandom already using --armor. So when using sop, it
-seems like it might make sense to use the `generate-key` command, and
-extract the armored data from it. Although note that not all of that output
-is random, the first several bytes are OpenPGP header that doesn't change
-much.
-
 See also: [[todo/whishlist__58___GPG_alternatives_like_AGE]]
+
+[[!meta title="support using Stateless OpenPGP instead of gpg for encryption methods other than encryption=shared"]]