mirror of
https://github.com/mjl-/mox.git
synced 2024-12-26 08:23:48 +03:00
3fb41ff073
we match messages to their parents based on the "references" and "in-reply-to" headers (requiring the same base subject), and in absense of those headers we also by only base subject (against messages received max 4 weeks ago). we store a threadid with messages. all messages in a thread have the same threadid. messages also have a "thread parent ids", which holds all id's of parent messages up to the thread root. then there is "thread missing link", which is set when a referenced immediate parent wasn't found (but possibly earlier ancestors can still be found and will be in thread parent ids". threads can be muted: newly delivered messages are automatically marked as read/seen. threads can be marked as collapsed: if set, the webmail collapses the thread to a single item in the basic threading view (default is to expand threads). the muted and collapsed fields are copied from their parent on message delivery. the threading is implemented in the webmail. the non-threading mode still works as before. the new default threading mode "unread" automatically expands only the threads with at least one unread (not seen) meessage. the basic threading mode "on" expands all threads except when explicitly collapsed (as saved in the thread collapsed field). new shortcuts for navigation/interaction threads have been added, e.g. go to previous/next thread root, toggle collapse/expand of thread (or double click), toggle mute of thread. some previous shortcuts have changed, see the help for details. the message threading are added with an explicit account upgrade step, automatically started when an account is opened. the upgrade is done in the background because it will take too long for large mailboxes to block account operations. the upgrade takes two steps: 1. updating all message records in the database to add a normalized message-id and thread base subject (with "re:", "fwd:" and several other schemes stripped). 2. going through all messages in the database again, reading the "references" and "in-reply-to" headers from disk, and matching against their parents. this second step is also done at the end of each import of mbox/maildir mailboxes. new deliveries are matched immediately against other existing messages, currently no attempt is made to rematch previously delivered messages (which could be useful for related messages being delivered out of order). the threading is not yet exposed over imap.
89 lines
3.2 KiB
Bash
Executable file
89 lines
3.2 KiB
Bash
Executable file
#!/bin/sh
|
|
|
|
# note: If testdata/upgradetest.mbox.gz exists it will be imported it as part of
|
|
# testing the upgrades. If this is a large mailbox, it will highlight performance
|
|
# or resource consumption issues during upgrades.
|
|
|
|
# todo: should we also test with mox.conf and domains.conf files? should "mox backup" and "mox gentestdata" add them, and "mox verifydata" use them?
|
|
|
|
set -e
|
|
# set -x
|
|
|
|
# We'll set a max memory limit during upgrades. We modify the softlimit when
|
|
# importing the potentially large mbox file.
|
|
# Currently at 768MB, needed for upgrading with 500k messages from v0.0.5 to
|
|
# v0.0.6 (two new indexes on store.Message).
|
|
ulimit -S -d 768000
|
|
|
|
(rm -r testdata/upgrade 2>/dev/null || exit 0)
|
|
mkdir testdata/upgrade
|
|
cd testdata/upgrade
|
|
|
|
# Check that we can upgrade what we currently generate.
|
|
../../mox gentestdata data
|
|
../../mox verifydata data
|
|
../../mox openaccounts data test0 test1 test2
|
|
rm -r data
|
|
echo
|
|
|
|
# For each historic release (i.e. all tagged versions) except the first few that
|
|
# didn't have the gentestdata command, we generate a data directory for testing
|
|
# and simulate upgrading to the currently checked out version.
|
|
# The awk command reverses the tags, so we try the previous release first since
|
|
# it is the most likely to fail.
|
|
tagsrev=$(git tag --sort creatordate | grep -v '^v0\.0\.[123]$' | awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }')
|
|
if test "$tagsrev" = ""; then exit 0; fi
|
|
for tag in $tagsrev; do
|
|
echo "Testing upgrade from $tag to current."
|
|
mkdir $tag
|
|
(CGO_ENABLED=0 GOBIN=$PWD/$tag go install github.com/mjl-/mox@$tag)
|
|
# Generate with historic release.
|
|
./$tag/mox gentestdata $tag/data
|
|
# Verify with current code. v0.0.[45] had a message with wrong Size. We don't
|
|
# want to abort the upgrade check because of it.
|
|
if test $tag = v0.0.4 -o $tag = v0.0.5; then
|
|
../../mox verifydata -skip-size-check $tag/data
|
|
else
|
|
../../mox verifydata $tag/data
|
|
fi
|
|
echo
|
|
rm -r $tag/data
|
|
done
|
|
|
|
# Also go step-wise through each released version. Having upgraded step by step
|
|
# can have added more schema upgrades to the database files.
|
|
tags=$(git tag --sort creatordate | grep -v '^v0\.0\.[123]$' | cat)
|
|
first=yes
|
|
for tag in $tags; do
|
|
if test "$first" = yes; then
|
|
echo "Starting with test data for $tag."
|
|
./$tag/mox gentestdata stepdata
|
|
echo
|
|
first=
|
|
else
|
|
# v0.0.5 got the ximport command
|
|
if test $tag = v0.0.5 -a -f ../upgradetest.mbox.gz; then
|
|
ulimit -S -d unlimited
|
|
echo 'Importing bulk data for upgrading.'
|
|
gunzip < ../upgradetest.mbox.gz | time ./$tag/mox ximport mbox ./stepdata/accounts/test0 upgradetest /dev/stdin
|
|
echo
|
|
ulimit -S -d 768000
|
|
fi
|
|
|
|
echo "Upgrade data to $tag."
|
|
if test $tag = v0.0.4 -o $tag = v0.0.5; then
|
|
time ./$tag/mox verifydata stepdata
|
|
else
|
|
time ./$tag/mox verifydata -skip-size-check stepdata
|
|
time ./$tag/mox openaccounts stepdata test0 test1 test2
|
|
fi
|
|
echo
|
|
fi
|
|
done
|
|
echo "Testing final upgrade to current."
|
|
time ../../mox -cpuprof ../../upgrade-verifydata.cpu.pprof -memprof ../../upgrade-verifydata.mem.pprof verifydata -skip-size-check stepdata
|
|
time ../../mox -loglevel info -cpuprof ../../upgrade-openaccounts.cpu.pprof -memprof ../../upgrade-openaccounts.mem.pprof openaccounts stepdata test0 test1 test2
|
|
rm -r stepdata
|
|
rm */mox
|
|
cd ../..
|
|
rmdir testdata/upgrade/* testdata/upgrade
|