Last modified: 2023-03-29

The manual setup

This section describes the manual setup of everything. Feel free to skip to the semi-automatic setup.

Stage	Sync mail	Sync trigger	Mail storage	Send mail	Passwords
0 aerc	aerc	manual	none	aerc	aerc
1 keepassxc	aerc	manual	none	aerc	keepassxc
2 mbsync	mbsync	manual	maildir	aerc	keepassxc
3 systemd	mbsync	systemd service	maildir	aerc	keepassxc
4 msmtp	mbsync	systemd service	maildir	msmtp	keepassxc

Stage 0: Test `aerc` and email access

First of all, make sure the you can connect with plain aerc to you account (at least one of them). I have use arec's build in wizard. This will get you default aerc experience.

Tip

Consider using email account with very few emails (maybe eve create a new one) since you might end up downloading it's content multiple times for debugging purposes.

Configuration file ~/.config/aerc/accounts.conf could look like this:

[tester@white-hat-hacker.icu]
source   = imaps://tester%40white-hat-hacker.icu:PASSWORD@mail.white-hat-hacker.icu:993
outgoing = smtps+plain://tester%40white-hat-hacker.icu:PASSWORD@mail.white-hat-hacker.icu:465
default  = INBOX
from     = tester <tester@white-hat-hacker.icu>
copy-to  = Sent

Warning

The user configuration file must be explicitly readable/writeable by its owner or aerc will fail (chmod 600).

PASSWORD is plain text password with URL encoding.

Stage 1: Configure `keepassxc` with secret service

Next step is to make sure that integration with keepassxc works. This is a bit clusterfuck of setting, but I will do my best to describe it.

Tip

Consider having the title and username identical in the keepassxc database, to avoid naming conflicts.

In the keepassxc in settings enable Enable Freedesktop.org Secret Service integration. Right below this should open up more settings (in keepassxc v 2.7.1 you have to save the new setting first to get access to configuration of the service itself).

So, below the Enable Freedesktop.org Secret Service integration should be now more settings available. In tab General, in the box Exposed database groups edit your database and expose the group with email accounts by going into Secret Service Integration, selecting Expose entries unser this group and there pick your group with email accounts.

If successful, you should have following:

At this moment I have created a test entry called test_secret_service in the exposed group with password 123abc. Then I have tried to retrieve it from terminal:

\$ secret-tool lookup "Title" "test_secret_service"
123abc

This means it works!

Tip

If you do not want to keep getting keepassxc database unlock prompts on every email refresh when the database is locked, uncheck the Promp to unlock database before searching.

Next, let's create a script ~/.local/email-stuff/get_password_from_secret_service.sh to retrieve the passwords (modified script from aerc-wiki):

#!/bin/sh

secret-tool lookup "\$1" "\$2"
# wait until the password is available
while [[ \$? != 0 ]]; do
	sleep 1
	secret-tool lookup "\$1" "\$2"
done

From now on you can replace plain-text passwords in configuration files with command to retrieve them from keepassxc.

Warning

If you are beeing bombarder with keepassxc that application requests access even though you pressed Allow All & Future, go to database settings and uncheck Confirm when passwords are retrieved by clients

Stage 2: Download mail with `mbsync/isync`

Next, let's setup offline storage.

Info

The name of the accouns and channels can be anything, but I decided to use username to keep things simple.

Emails will be synchronized over IMAP with mbsync/isync, config file at ~/.config/mbsyn/mbsyncrc.conf:

IMAPAccount tester@white-hat-hacker.icu
Host mail.white-hat-hacker.icu
User tester@white-hat-hacker.icu
#Pass
PassCmd "~/.local/email-stuff/get_password_from_secret_service.sh 'Title' 'tester@white-hat-hacker.icu'"
#
# Use SSL
SSLType IMAPS
# The following line should work. If you get certificate errors, uncomment the two following lines and read the "Troubleshooting" section.
CertificateFile /etc/ssl/certs/ca-certificates.crt
#CertificateFile ~/.cert/imap.gmail.com.pem
#CertificateFile ~/.cert/Equifax_Secure_CA.pem

IMAPStore tester@white-hat-hacker.icu-remote
Account tester@white-hat-hacker.icu

MaildirStore tester@white-hat-hacker.icu-local
SubFolders Verbatim
# The trailing "/" is important
Path ~/.mail/tester@white-hat-hacker.icu/
Inbox ~/.mail/tester@white-hat-hacker.icu/INBOX

Channel tester
Far :tester@white-hat-hacker.icu-remote:
Near :tester@white-hat-hacker.icu-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
#Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
# Or include everything
Patterns *
# Automatically create missing mailboxes, both locally and on the server
Create Both
# Sync the movement of messages between folders and deletions, add after making sure the sync works
Expunge Both
# Save the synchronization state files in the relevant directory
SyncState *

Now update config file for aerc:

diff --git a/.config/aerc/accounts.conf b/.config/aerc/accounts.conf
index efb5adf..7557c28 100644
--- a/.config/aerc/accounts.conf
+++ b/.config/aerc/accounts.conf
@@ -1,5 +1,5 @@
 [tester@white-hat-hacker.icu]
-source   = imaps://tester%40white-hat-hacker.icu:PASSWORD@mail.white-hat-hacker.icu:993
+source   = maildir://~/.mail/tester@white-hat-hacker.icu
 outgoing = smtps+plain://tester%40white-hat-hacker.icu:PASSWORD@mail.white-hat-hacker.icu:465
 default  = INBOX
 from     = tester <tester@white-hat-hacker.icu>

Warning

Create the appropriate directory before attempting to sync:

\$ mkdir -p ~/.mail/tester@white-hat-hacker.icu

Otherwise you get error:

Maildir error: cannot open store '~/.mail/tester@white-hat-hacker.icu/'

To synchronize the email (or download in this case), run command:

\$ mbsync --config ~/.config/mbsync/mbsyncrc.conf -a

Now when you run aerc, all emails should load instantly since they are store locally.

Stage 3: Automatic email retrieval

I will use mainly push notifications, however push notification is only triggered on arrival of new email. Therefore I need two systemd service types:

trigger sync on boot / user login
start push notifications

First is easy, create a systemd service ~/.config/systemd/user/mbsync.service:

[Unit]
Description=Mailbox synchronization service

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync --config ${HOME}/.config/mbsync/mbsyncrc.conf -Va

[Install]
WantedBy=default.target

Then enable the service:

systemctl enable --now --user mbsync.service

This service will synchronize the emails on boot or user login (I am not sure). But it will have to wait for keepassxc database to be unlocked anyway.

Now the push notification system. Let's create a override for the existing goimapnotify service:

systemctl edit --user goimapnotify@.service

 [Unit]
 Description=Execute scripts on IMAP mailbox changes (new/deleted/updated messages) using IDLE, golang version.
+After=mbsync

 [Service]
 Type=simple
 ExecStart=/usr/bin/goimapnotify -conf %h/.config/imapnotify/%i.conf
 Restart=always
 RestartSec=30
 
 [Install]
 WantedBy=default.target

This is to avoid conflict between push imapnotify and mbsync. (although they might have some lock-files ... well better safe than sorry)

Next is a configuration file for the imapnotify at ~/.config/imapnotify/testerwhite-hat-hackericu.conf:

{
  "host": "mail.white-hat-hacker.icu",
  "port": 993,
  "tls": true,
  "tlsOptions": {
    "rejectUnauthorized": false
  },
  "username": "tester@white-hat-hacker.icu",
  "passwordCmd": "~/.local/email-stuff/get_password_from_secret_service.sh 'Title' 'tester@white-hat-hacker.icu' | head -n1",
  "onNewMail": "mbsync --config ~/.config/mbsync/mbsyncrc.conf tester@white-hat-hacker.icu",
  "wait": 5,
  "boxes": [
    "INBOX"
  ]
}

Warning

The | head -n1 is important! Without it the imapnotify will not wait for stdout and fail.

Warning

The wait: 5 is there so that if you get multiple emails at once, the sync is not triggered on each email.

Now start and enable the service:

systemctl enable --now --user goimapnotify@testerwhite-hat-hackericu.service

And that is it for automatic synchronization with the server.

Stage 4: Send mail with `msmtp`

Next is offline sending of emails with msmtp. Or to be more specific, ability to write emails while offline and have them queued for later when online. Let's start with configuration file at $XDG_CONFIG_HOME/msmtp/config which is more less msmtp basic setup:

# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        ~/.cache/msmtp.log

# Gmail
account        tester@white-hat-hacker.icu
host           mail.white-hat-hacker.icu
port           465
from           tester@white-hat-hacker.icu
user           tester  
#password       plain-text-password
passwordeval   "~/.local/email-stuff/get_password_from_secret_service.sh 'Title' 'tester@white-hat-hacker.icu'"

# A freemail service
#account        freemail
#host           smtp.freemail.example
#from           joe_smith@freemail.example

# Set a default account
account default : tester@white-hat-hacker.icu

Warning

The user configuration file must be explicitly readable/writeable by its owner or msmtp will fail (chmod 600).

Change the permissions:

chmod 600 \$XDG_CONFIG_HOME/msmtp/config

I have tried script from /usr/share/doc/msmtp/msmtpqueue/ (and I have changed the QUEUEDIR), but I had some issues. Instead I ended up using scripts from /usr/share/doc/msmtp/msmtpq/. Copy your selected scripts into ~/.local/email-stuff/ (I select this directory since the configuration is done by editing the script itself, so to keep it in your home).

The configuration is done by editing the msmtpq itself:

diff --git a/.local/email-stuff/msmtpq b/.local/email-stuff/msmtpq
index 1b39fc6..fcd3232 100755
--- a/.local/email-stuff/msmtpq
+++ b/.local/email-stuff/msmtpq
@@ -70,7 +70,7 @@ MSMTP=msmtp
 ##            ( chmod 0700 msmtp.queue )
 ##
 ## the queue dir - modify this to reflect where you'd like it to be  (no quotes !!)
-Q=~/.msmtp.queue
+Q=~/.mail/.msmtpqueue
 [ -d "$Q" ] || mkdir -m 0700 "$Q" || \
   err '' "msmtpq : can't find or create msmtp queue directory [ $Q ]" ''     # if not present - complain ; quit
 ##
@@ -84,7 +84,7 @@ Q=~/.msmtp.queue
 ##     (doing so would be inadvisable under most conditions, however)
 ##
 ## the queue log file - modify (or comment out) to taste  (but no quotes !!)
-LOG=~/log/msmtp.queue.log
+LOG=~/.cache/msmtp.queue.log
 ## ======================================================================================
 
 ## msmtpq can use the following environment variables :

Create the queue directory:

mkdir -p ~/.mail/.msmtpqueue
chmod 0700 ~/.mail/.msmtpqueue

As described in the readme, you can use msmtp-queue to work with the queue.

msmtp-queue usage :
-----------------

msmtp-queue offers the following options :

  msmtp-queue -r -- runs (flushes) all the contents of the queue

  msmtp-queue -R -- sends selected individual mail(s) in the queue

  msmtp-queue
  msmtp-queue -d -- displays the contents of the queue   (<-- default)

  msmtp-queue -p -- allows the specific purging of one or more mails

  msmtp-queue -a -- purges all mail in the queue

  msmtp-queue -h -- offers a helpful blurt

I don't know what would be the best automatic trigger to send the queued emails. Right now you can either run the command msmtp-queue -r, or just send another email which will trigger queue flush.

Probably the best option is systemd service with timer that will run msmtp-queue -r command.

Stage 5: email indexing with `notmuch` (Work-in-Progress)

Bug

Setup described in this section is not properly working yet.

First of all, I will move the configuration file into .config/notmuch where it belongs. So I am adding export NOTMUCH_CONFIG="${XDG_CONFIG_HOME}/notmuch/notmuch.conf" to ~/.bash_profile.

Now run notmuch setup and enter information accordingly. As for Top-level directory of your email archive enter your /home/USER/.mail. Then run notmuch new.

diff --git a/.config/systemd/user/mbsync.service b/.config/systemd/user/mbsync.service
index 6428b7c..b020fbf 100644
--- a/.config/systemd/user/mbsync.service
+++ b/.config/systemd/user/mbsync.service
@@ -4,6 +4,7 @@ Description=Mailbox synchronization service
 [Service]
 Type=oneshot
 ExecStart=/usr/bin/mbsync --config ${HOME}/.config/mbsync/mbsyncrc.conf -Va
+ExecStartPost=/usr/bin/notmuch new
 
 [Install]
 WantedBy=default.target

Then update the ~/.config/aerc/accounts.conf:

diff --git a/.config/aerc/accounts.conf b/.config/aerc/accounts.conf
index d2f133e..5c99162 100644
--- a/.config/aerc/accounts.conf
+++ b/.config/aerc/accounts.conf
@@ -1,5 +1,5 @@
 [tester@white-hat-hacker.icu]
-source   = maildir://~/.mail/tester@white-hat-hacker.icu
+source   = notmuch://~/.mail/tester@white-hat-hacker.icu
 outgoing = /home/atom/.local/email-stuff/msmtpq
 from     = <tester@white-hat-hacker.icu>
 default  = INBOX

With notmuch, deletion of files gets a bit complicated. Basically, via aerc you add a custom label and then can notmuch to cleanup. I will be mostly following this aerc: notmuch guide.

Create a script to syncrhonise and cleanup the maildir at ~/.local/email-stuff/mail-sync.sh:

#!/bin/sh

MBSYNC=\$(pgrep mbsync)
NOTMUCH=\$(pgrep notmuch)

if [ -n "\$MBSYNC" -o -n "\$NOTMUCH" ]; then
    echo "Already running one instance of mbsync or notmuch. Exiting..."
    exit 0
fi

echo "Deleting messages tagged as *deleted*"
notmuch search --format=text0 --output=files tag:deleted | xargs -0 --no-run-if-empty rm -v

mbsync --config \${HOME}/.config/mbsync/mbsyncrc.conf -Va
notmuch new

diff --git a/.config/imapnotify/testerwhite-hat-hackericu.conf b/.config/imapnotify/testerwhite-hat-hackericu.conf
index 711189e..3144fb8 100644
--- a/.config/imapnotify/testerwhite-hat-hackericu.conf
+++ b/.config/imapnotify/testerwhite-hat-hackericu.conf
@@ -7,7 +7,7 @@
   },
   "username": "tester@white-hat-hacker.icu",
   "passwordCmd": "~/.local/email-stuff/get_password_from_secret_service.sh 'Title' 'tester@white-hat-hacker.icu' | head -n1",
-  "onNewMail": "mbsync --config ~/.config/mbsync/mbsyncrc.conf tester@white-hat-hacker.icu",
+  "onNewMail": "~/.local/email-stuff/mail-sync.sh",
   "wait": 5,
   "boxes": [
     "INBOX"

Tip

This might not be optimal since now all accounts will be synchronised on arrival of one email, but you could theoretically pass along the account name and therefore optimise this.

white-hat-hacker

The manual setup

Stage 0: Test aerc and email access

Stage 1: Configure keepassxc with secret service

Stage 2: Download mail with mbsync/isync

Stage 3: Automatic email retrieval

Stage 4: Send mail with msmtp

Stage 5: email indexing with notmuch (Work-in-Progress)

Stage 0: Test `aerc` and email access

Stage 1: Configure `keepassxc` with secret service

Stage 2: Download mail with `mbsync/isync`

Stage 4: Send mail with `msmtp`

Stage 5: email indexing with `notmuch` (Work-in-Progress)