[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [msmtp-users] Bug in msmtpq might cause mail loss
Philipp Hartwig & all,
Thanks for finding the two bugs. They are now fixed according to
suggestions.
The first one is mea culpa ; I thought I had tested it - but
apparently not.
The second one, the race condition, was simply beyond my local norms -
I seldom send out more than a few mails at a time, it didn't occur to
me that someone might send many, rapidly. So thanks, in effect, for
expanding the 'horizon' of the script. It is more robust because of
your fix.
If anyone is interested I have attached the latest versions. Please
feel free to test and play with them. All feedback is welcome.
Please also note than in response to a recent complaint about
behaviour on case-insensitive file systems (ugh), I have slightly
changed the names of the scripts. Note that you will have to adjust
things for the name changes, particularly (.)muttrc. (msmtpQ/msmtpq
--> msmtpq/msmtp-queue). The changes necessary are very small and I am
hoping that this will not prove to be an issue in the distribution of
the new version.
The argv(0) behaviour has also been changed because of minor wierdness
in the construction of the distribution archives - there are no longer
any symbolic links necessary. msmtp-queue is a tiny wrapper script
that invokes msmtpq in queue maintenance/management mode.
Thank you Philipp & all - particularly Martin. Please let me know ;
feedback is solicited.
Best,
Chris
#!/usr/bin/env bash
#--------------------------------------------------------------
#
# msmtpq : queue funtions to both use & manage the msmtp queue,
# as it was defined by Martin Lambers
# Copyright (C) 2008, 2009, 2010, 2011 Chris Gianniotis
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or, at
# your option, any later version.
#
#--------------------------------------------------------------
# msmtpq is meant to be used both by an email client -
# in 'sendmail' mode
# for this purpose, it is evoked directly as 'msmtpq'
# it is also meant to be used to maintain the msmtp queue
# evoked by the wrapper script 'msmtp-queue'
# (which calls this script as msmtpq --q-mgnt)
# there is a queue log file, distinct from the msmtp log,
# for all events & operations on the msmtp queue
# that is defined below
# (mutt users, using msmtpq in 'sendmail' mode,
# should make at least the following two settings in their .muttrc
# set sendmail = /path/to/msmtpq
# set sendmail_wait = -1
#
# please see the msmtp man page and docs for further mutt settings
# )
## two essential patches by Philipp Hartwig
## 19 Oct 2011 & 27 Oct 2011
#--------------------------------------------------------------
# the msmtp queue contains unique filenames of the following form :
# two files for each mail in the queue
#
# creates new unique filenames of the form :
# MLF: ccyy-mm-dd-hh.mm.ss[-x].mail -- mail file
# MSF: ccyy-mm-dd-hh.mm.ss[-x].msmtp -- msmtp command line file
# where x is a consecutive number only appended for uniqueness
# if more than one mail per second is sent
#--------------------------------------------------------------
dsp() { local L ; for L ; do [ -n "$L" ] && echo " $L" || echo ; done ; }
err() { dsp '' "$@" '' ; exit 1 ; }
## ======================================================================================
## !!! please define or confirm the following three vars !!!
## !!! before using the msmtpq or msmtp-queue scripts !!!
## ======================================================================================
##
## only if necessary (in unusual circumstances - e.g. embedded systems),
## enter the location of the msmtp executable (no quotes !!)
## e.g. ( MSMTP=/path/to/msmtp )
MSMTP=msmtp
#[ -x "$MSMTP" ] || \
# log -e 1 "msmtpq : can't find the msmtp executable [ $MSMTP ]" # if not found - complain ; quit
##
## set the queue var to the location of the msmtp queue directory
## if the queue dir doesn't yet exist, better to create it (0700)
## before using this routine
## e.g. ( mkdir msmtp.queue )
## ( chmod 0700 msmtp.queue )
##
## the queue dir - modify this to reflect where you'd like it to be (no quotes !!)
Q=~/.msmtp.queue
[ -d "$Q" ] || \
err '' "msmtpq : can't find msmtp queue directory [ $Q ]" '' # if not present - complain ; quit
##
## set the queue log file var to the location of the msmtp queue log file
## where it is or where you'd like it to be
## ( note that the LOG setting could be the same as the )
## ( 'logfile' setting in .msmtprc - but there may be )
## ( some advantage in keeping the two logs separate )
## if you don't want the log at all unset (comment out) this var
## LOG=~/log/msmtp.queue.log --> #LOG=~/log/msmtp.queue.log
## (doing so would be inadvisable under most conditions, however)
##
## the queue log file - modify (or comment out) to taste (but no quotes !!)
LOG=~/log/msmtp.queue.log
## ======================================================================================
umask 077 # set secure permissions on created directories and files
declare -i CNT # a count of mail(s) currently in the queue
declare -a Q_LST # queue list array ; used selecting a mail (to send or remove)
trap on_exit EXIT # run 'on_exit' on exit
on_exit() {
[ -n "$LKD" ] && lock_queue -u 2>/dev/null # unlock the queue on exit
} # if the lock was set here
#
## ----------------------------------- common functions
#
## make an entry to the queue log file, possibly an error
## (log queue changes only ; not interactive chatter)
## usage : log [ -e errcode ] msg [ msg ... ]
## opts : -e <exit code> an error ; log msg & terminate w/prejudice
## display msg to user, as well
log() {
local ARG RC PFX="$('date' +'%Y %d %b %H:%M:%S')"
# time stamp prefix - "2008 13 Mar 03:59:45 "
if [ "$1" = '-e' ] ; then # there's an error exit code
RC="$2" # take it
shift 2 # shift opt & its arg off
fi
dsp "$@" # display msg to user, as well as logging it
if [ -n "$LOG" ] ; then # log is defined and in use
for ARG ; do # each msg line out
[ -n "$ARG" ] && \
echo "$PFX : $ARG" >> "$LOG" # line has content ; send it to log
done
fi
if [ -n "$RC" ] ; then # an error ; leave w/error return
[ -n "$LOG" ] && \
echo " exit code = $RC" >> "$LOG" # logging ok ; send exit code to log
exit $RC # exit w/return code
fi
}
## write/remove queue lockfile for a queue op
lock_queue() { # <-- '-u' to remove lockfile
local LOK="${Q}/.lock" # lock file name
local -i MAX=240 SEC=0 # max seconds to gain a lock ; seconds waiting
if [ -z "$1" ] ; then # lock queue
## Philipp Hartwig patch #2
'mkdir' "$LOK" 2>/dev/null && LKD='t'
while [ -z "$LKD" ] && (( SEC < MAX )) ; do # lock file present
sleep 1 # wait a second
(( ++SEC )) # accumulate seconds
'mkdir' "$LOK" 2>/dev/null && LKD='t' # make lockdir ; lock queue
done # try again while locked for MAX secs
[ -z "$LKD" ] && \
err '' "cannot use queue $Q : waited $MAX seconds for"\
" lockdir [ $LOK ] to vanish ; giving up"\
'if you are certain that no other instance of this script'\
" is running, then 'rmdir' the lock dir manually" '' # lock file still there, give up
elif [ "$1" = '-u' ] ; then # unlock queue
'rmdir' "$LOK" # remove the lock
fi
}
#
## ----------------------------------- functions for queue management
## ----------------------------------- queue maintenance mode - (msmtp-queue)
#
## show queue maintenance functions
usage() { # <-- error msg
dsp ''\
'usage : msmtp-queue functions' ''\
' msmtp-queue < op >'\
' ops : -r run (flush) mail queue - all mail in queue'\
' -R send selected individual mail(s) in queue'\
' -d display (list) queue contents (<-- default)'\
' -p purge individual mail(s) from queue'\
' -a purge all mail in queue'\
' -h this helpful blurt' ''\
' ( one op only ; any others ignored )' ''
[ -z "$1" ] && exit 0 || { dsp "$@" '' ; exit 1 ; }
}
## get user [y/n] acknowledgement
ok() {
local R YN='Yn' # default to yes
[ "$1" = '-n' ] && \
{ YN='yN' ; shift ; } # default to no ; change prompt ; shift off spec
dsp "$@"
while true ; do
echo -n " ok [${YN}] ..: "
read R
case $R in
y|Y) return 0 ;;
n|N) return 1 ;;
'') [ "$YN" = 'Yn' ] && return 0 || return 1 ;;
*) echo 'yYnN<cr> only please' ;;
esac
done
}
## send a queued mail out via msmtp
send_queued_mail() { # <-- mail id
local FQP="${Q}/${1}" # fully qualified path name
local -i RC=0 # for msmtp exit code
if [ -f "${FQP}.msmtp" ] ; then # corresponding .msmtp file found
# verify net connection - ping ip address of debian.org
# would ping -qnc2 -w4 be better ?
if ping -qnc1 -w4 debian.org > /dev/null 2>&1 ; then # connected
if $MSMTP $(< "${FQP}.msmtp") < "${FQP}.mail" ; then # this mail goes out the door
log "mail [ $2 ] [ $1 ] from queue ; send was successful ; purged from queue" # good news to user
'rm' -f ${FQP}.* # nuke both queue mail files
ALT='t' # set queue changed flag
else # send was unsuccessful
RC=$? # take msmtp exit code
log "mail [ $2 ] [ $1 ] from queue ; send failed ; msmtp rc = $RC" # bad news ...
fi
return $RC # func returns exit code
else # not connected
dsp "mail [ $2 ] [ $1 ] from queue ; couldn't be sent - host not connected"
return 1
fi
else # corresponding MSF file not found
log "preparing to send .mail file [ $1 ] [ ${FQP}.mail ] but"\
" corresponding .msmtp file [ ${FQP}.msmtp ] was not found in queue"\
' skipping this mail ; this is worth looking into' # give user the bad news
fi # (but allow continuation)
}
## run (flush) queue
run_queue() { # run queue
local M LST="$('ls' $Q/*.mail 2>/dev/null)" # list of mails in queue
local -i NDX=0
if [ -n "$LST" ] ; then # something in queue
for M in $LST ; do # process all mails
((NDX++))
send_queued_mail "$(basename $M .mail)" "$NDX" # send mail - pass {id} only
done
else # queue is empty
dsp '' 'mail queue is empty (nothing to send)' '' # inform user
fi
}
## display queue contents
display_queue() { # <-- { 'purge' | 'send' } (op label) ; { 'rec' } (record array of mail ids)
local M ID LST="$('ls' ${Q}/*.mail 2>/dev/null)" # list of mail ids in queue
CNT=0
if [ -n "$LST" ] ; then # list has contents (any mails in queue)
for M in $LST ; do # cycle through each
ID="$(basename $M .mail)" # take mail id from filename
((CNT++))
dsp '' "mail num=[ $CNT ] id=[ $ID ]" # show mail id ## patch in
'egrep' -s --colour -h '(^From:|^To:|^Subject:)' "$M" # show mail info
[ -n "$2" ] && Q_LST[$CNT]="$ID" # bang mail id into array (note 1-based array indexing)
done
echo
else # no mails ; no contents
[ -z "$1" ] && \
dsp '' 'no mail in queue' '' || \
dsp '' "mail queue is empty (nothing to $1)" '' # inform user
exit 0
fi
}
## delete all mail in queue, after confirmation
purge_queue() {
local YN # confirmation response
display_queue 'purge' # show queue contents
if ok -n 'remove (purge) all mail from the queue' ; then
'rm' -f "$Q"/*.*
log 'msmtp queue purged (all mail)'
else
dsp '' 'nothing done ; queue is untouched' ''
fi
}
## select a single mail from queue ; delete it or send it
## select by mail index (position in queue) or mail id
select_mail() { # <-- < 'purge' | 'send' >
local OK ID # mail id
local -i I NDX # mail index (position in queue)
while true ; do # purge an individual mail from queue
display_queue "$1" 'rec' # show queue contents ; make mail ids array
## allow selection also by mail index
if [ $CNT -eq 1 ] ; then # only one mail in queue ; take its id
NDX=1
ID="${Q_LST[1]}"
else # more than one mail ; select its id
while true ; do # get mail id
OK='t' # optimistic to a fault
dsp "enter mail number or id to $1" # <-- num or file name (only, no suff)
echo -n ' ( <cr> alone to exit ) ..: '
read ID
[ -n "$ID" ] || return # entry made - or say good bye
if [ "${ID:4:1}" != '-' ] ; then # mail id *not* entered ; test index num
for (( I=0 ; I<${#ID} ; I++ )) ; do # test index number
if [[ "${ID:${I}:1}" < '0' || \
"${ID:${I}:1}" > '9' ]] ; then
dsp '' "[ $ID ] is neither a valid mail id"\
'nor a valid mail number' ''
unset OK
fi
done
[ -z "$OK" ] && continue # format not ok (not all nums)
NDX=$ID
if [ $NDX -lt 1 ] || [ $NDX -gt $CNT ] ; then # test number range (1 - $CNT)
dsp '' "[ $NDX ] is out of range as a mail number"\
"validity is from 1 to $CNT"
continue # try again
fi
ID="${Q_LST[$NDX]}" # format & range were ok ; use it
break # valid mail selection
else # mail id entered
for (( NDX=1 ; NDX<=${#Q_LST[*]} ; NDX++ )) ; do # find entered id in queue list
[ "$ID" = "${Q_LST[$NDX]}" ] && break
done
[ $NDX -le ${#Q_LST[*]} ] && \
break || \
dsp '' "mail [ $ID ] not found ; invalid id" # mail selection valid (found) or
fi # fell through (not found) complain
done # and ask again
fi
if ok '' "$1 :"\
" mail num=[ $NDX ]"\
" id=[ $ID ]" '' ; then # confirm mail op
if [ "$1" = 'purge' ] ; then # purging
'rm' -f "$Q"/"$ID".* # msmtp - nukes single mail (both files) in queue
log "mail [ $ID ] purged from queue" # log op
ALT='t' # mark that a queue alteration has taken place
else # sending
send_queued_mail "$ID" "$NDX" # send out the mail
fi
else # user opts out
dsp '' 'nothing done to this queued email' # soothe user
[ $CNT -eq 1 ] && break # single mail ; user opted out
fi
dsp '' "--------------------------------------------------"
done
if [ -n "$ALT" ] ; then # queue was changed
dsp '' 'done' ''
else # queue is untouched
dsp '' 'nothing done ; queue is untouched' ''
fi
}
#
## ----------------------------------- functions for directly sending mail
## ----------------------------------- 'sendmail' mode - (msmtpq)
#
## ('sendmail' mode only)
## make base filename id for queue
make_id() {
local -i INC # increment counter for (possible) base fqp name collision
ID="$(date +%Y-%m-%d-%H.%M.%S)" # make filename id for queue
FQP="${Q}/$ID" # make fully qualified pathname
## Philipp Hartwig patch #1
if [ -f "${FQP}.mail" -o -f "${FQP}.msmtp" ] ; then # ensure fqp name is unique
INC=1 # initial increment
while [ -f "${FQP}-${INC}.mail" -o -f "${FQP}-${INC}.msmtp" ] ; do # fqp name w/incr exists
(( ++INC )) # bump increment
done
ID="${ID}-${INC}" # unique ; set id
FQP="${FQP}-${INC}" # unique ; set fqp name
fi
}
## ('sendmail' mode only)
## enqueue a mail
enqueue_mail() { # <-- all mail args ; mail text via TMP
if echo "$@" > "${FQP}.msmtp" ; then # write msmtp command line to queue .msmtp file
log "enqueued mail as : [ $ID ] ( $* ) : successful" # (queue .mail file is already there)
else # write failed ; bomb
log -e "$?" "queueing - writing msmtp cmd line { $* }"\
" to [ ${ID}.msmtp ] : failed"
fi
}
## ('sendmail' mode only)
## send a mail (if possible, otherwise enqueue it)
## if send is successful, msmtp will also log it (if enabled in ~/.msmtprc)
send_mail() { # <-- all mail args ; mail text via TMP
# verify net connection - ping ip address of debian.org
if ping -qnc1 -w4 debian.org > /dev/null 2>&1 ; then # we're online, connected
if $MSMTP "$@" < "${FQP}.mail" > /dev/null ; then # send mail using queue .mail fil
log "mail for [ $* ] : send was successful" # log it
'rm' -f "${FQP}.mail" # remove queue .mail file
run_queue # run/flush any other mails in queue
else # send failed
log "mail for [ $* ] : send was unsuccessful ; msmtp exit code was $?"
enqueue_mail "$@" # enqueue the mail
fi
else # not connected to net ; log msg
log "mail for [ $* ] : couldn't be sent - host not connected"
enqueue_mail "$@" # enqueue the mail
fi
}
#
## -- entry point
#
if [ ! "$1" = '--q-mgmt' ] ; then # msmtpq - sendmail mode
lock_queue # lock here
make_id # make base queue filename id for this mail
cat > "${FQP}.mail" || \
log -e "$?" "creating mail body file [ ${FQP}.mail ] : failed" # test for error
send_mail "$@" # send the mail if possible, queue it if not
else # msmtp-queue - queue management mode
shift # trim off first (--q-mgmt) arg
OP=${1:1} # trim off first char of OP arg
case "$OP" in # sort ops ; run according to spec
r) lock_queue ; run_queue ;; # run (flush) the queue
R) lock_queue ; select_mail send ;; # send individual mail(s) in queue
d|'') display_queue ;; # display (list) all mail in queue (default)
p) lock_queue ; select_mail purge ;; # purge individual mail(s) from queue
a) lock_queue ; purge_queue ;; # purge all mail in queue
h) usage ;; # show help
*) usage "[ -$OP ] is an unknown msmtp-queue option" ;;
esac
fi
exit 0
#!/usr/bin/env bash
##--------------------------------------------------------------
##
## msmtp-queue : wrapper script for msmtpq
## to expose the management functions
## for the msmtp queue
## Copyright (C) 2011 Chris Gianniotis
##
## This program is free software: you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or, at
## your option, any later version.
##
##--------------------------------------------------------------
## msmtp-queue
## is only a wrapper
## it invokes the queue management functions of msmtpq
## by calling it as msmtpq --q-mgmt
##
## all configuration and documentation is in the msmtpq script
## note that for 'special circumstances'
## e.g. embedded systems, etc.
## where the msmtpq script itself is not on the path
## change the below line to be
## exec /path/to/msmtpq --q-mgmt
exec msmtpq --q-mgmt
msmtpq is a modification of the scripts comprising Martin Lambers'
msmtpqueue package ; it attempts be useful to msmtp by holding mail sent
by an MUA when a machine, e.g. a laptop, is not connected to the net ;
queued mails are held as files in a user defined and created queue
directory ; when the machine is connected all mail can be sent directly
out ; the msmtpq package does queue handling for msmtp according to the
way the queue is used
msmtpq accepts mail input from an MUA (e.g. mutt) and sends it
directly out via msmtp if possible - or queues it if not
msmtp-queue is a wrapper script which simply calls msmtpq with the
--q-mgmt argument ; it exposes routines for queue
management and maintenance, that is, for queue display,
queue run (flush), mail purge from the queue, etc.
msmtp-queue is interactive at the command line while msmtpq, called
directly, has no interactive user input whatever - it simply takes its
input from the MUA and outputs directly to msmtp (or to the queue)
credit goes to Martin Lambers for the idea of the queue ; any bugs,
design errors, oversights, suggestions for improvement, etc. belong to
me ; all feedback is welcome ; I can be reached via the msmtp mailing
list
Installation :
------------
copy msmtpq and msmtp-queue to wherever location is best for you
(I use /usr/local/bin) ; the directory chosen should be on the path
replace the msmtp invocation in your mail client with msmtpq ; e.g. for
mutt : 'set sendmail = /path/to/msmtpq'
msmtpq will then take care of the management and routing of outgoing
mail ; normally sending a mail is nearly instantaneous, but a very long
mail (or one with large attachments) can take some time to make its way
through ; therefore it's wise to have the MUA background the process if
possible
In the case of mutt, in addition to the above setting,
'set sendmail = /path/to/msmtpq'
set, also
'set sendmail_wait = -1'
mutt then backgrounds the send and watches for a return code & message
from msmtpq
Configuration :
-------------
[ note that the msmtp-queue script simply invokes msmtpq ; for most
installations this works 'out of the box' (it is on the path) but for
special cases (embedded, etc.) it may be necessary to modify the msmtpq
invocation in msmtp-queue to specify the location of the msmtpq script ]
all config is done within the msmtpq script
set the MSMTP var to point to the location of the msmtp executable
(only necessary if it's not on the path)
set the Q var to point to the location of the queue directory
set the LOG var to point to the location of the queue log
the MSMTP variable can have the location of the msmtp executable entered
into it if it is not on the path ; this might be useful in unusual
circumstances, such as embedded systems, etc. ; otherwise, if you are
running a normal Linux distribution you can leave it as is ; msmtp will
by default be on the execution path
the Q variable should have the location of the queue directory
the LOG variable should have the desired name & location of the queue log
the locations are clearly marked near the beginnings of the script ;
modify all to the locations you prefer (the defaults work for me ; you
may or may not be happy with them) ... please note that it's preferable
to create the queue directory (with 0700 permissions) before using these
routines
note that the default msmtpq set up creates a separate log for queue
operations ; all operations which modify the queue in any way are logged
to the queue log ; this is distinct from the msmtp log set by the
'logfile' setting in .msmtprc ; I've been persuaded that it's better to
have separate logs (for the distinct functions) ; if this doesn't sit
well with you it's possible to define the queue log to be the same log
file as the one defined in .msmtprc ; it's also possible to turn off
queue logging entirely (by commenting out the 'LOG=' var - to be
'#LOG=') but this seems hardly advisable, particularly before you are
confident that all is working
mutt users please take note of the additional setup instructions in the
msmtp docs & man page.
in summary :
copy the msmtpq and msmtp-queue scripts to the directory where they will live
be certain that they are executable
( chmod +x msmtpq msmtp-queue )
create the queue dir (0700) if necessary
( mkdir -p /path/to/queue/dir )
( chmod 0700 /path/to/queue/dir )
enter or confirm the values of the three config vars in msmtpq
for mutt users -
set sendmail="/path/to/msmtpq"
set sendmail_wait = -1
set use_from=yes
set realname="Your Name"
set from=you@...45...
set envelope_from=yes
(and perhaps more on the msmtp man page & docs ...)
for other MUAs -
set the mta to '/path/to/msmtpq'
background its execution, if possible
log(s) rotation :
On a Debian or Debian-based system (e.g. Ubuntu or Mint) a file can be
created in /etc/logrotate.d (named, e.g. 'msmtp') that configures
logrotate to compress and rotate both the msmtp logs and msmtpq queue
logs.
Suggested contents :
#--------------------------------------------------
/home/xyz/log/msmtp.log /home/xyz/log/msmtp.queue.log {
rotate 9999
weekly
compress
notifempty
missingok
}
#--------------------------------------------------
adjust it to your log location(s) and see the logrotate man page for
further config information ; it should be trivial to adapt to other
systems
That's it.
Use :
---
msmtpq is entirely self contained ; beyond keeping an eye on the queue
and queue logs it should require no admin or user attention.
msmtp-queue offers the following options :
msmtp-queue -r -- runs (flushes) all the contents of the queue
msmtp-queue -R -- sends selected individual mail(s) in the queue
msmtp-queue
msmtp-queue -d -- displays the contents of the queue (<-- default)
msmtp-queue -p -- allows the specific purging of one or more mails
msmtp-queue -a -- purges all mail in the queue
msmtp-queue -h -- offers a helpful blurt
( msmtp-queue does not log the 'chatter' of user interaction but logs,
to the queue log, any event that changes the queue in any way )
I find it useful to have a display of the number of mails currently
in the queue in conky ; it uses one shell line (in the .conkyrc script) :
mail queue : ${exec /bin/ls -1 ${HOME}/.msmtp.queue/*.mail 2>/dev/null | /usr/bin/wc -l}
With hopes that it's useful,
Chris Gianniotis