Receive An Email When Your Server Disk Is Almost Full

A few weeks ago I had a very bad surprise in the morning when I saw that all my websites where down. The issue was because I had a bug in one of my project that created tons and tons of log files.

After a few days I had 1TO of log files on my server (!!!!). Basically when your disk is completely full, your server can't do anything and is not even able to log errors anymore.

I would have liked to know before everything crashed of course...

So I made a very simple bash script to send me an email if this kind of issue ever happen again.

In my home directory I created a file storage-warning.sh

#!/bin/bash
CURRENT=$(df | grep '/var' | awk '{print $5}' | sed 's/%//g')
THRESHOLD=90

if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
    mail -s 'Server Storage almost full!' [email protected] << EOF
Your /var partition available space is critically low. Used: $CURRENT%
EOF
fi

You can now run this script daily via a cron job very easily!

Using Telegram instead of email

Because SMTP servers and server email can really be a pain, it is sometimes easier to simply trigger a CURL request and trigger another type of custom action.

One that I use a lot myself is a personnal telegram bot that I use for several application.

It's basically a way to get mobile notification for free. Your own personnal bot.

When you have your telegram account setup you can talk to BotFather (https://telegram.me/BotFather). It's a bot that will setup your bot for you. Super easy. With that your own bot will be setup in a few seconds.

Telegram Script

I made a folder /etc/telegram/ in which I created two files:

telegram-notify.conf:

api-key=YOUR_API_KEY
user-id=YOUR_CHAT_ID

notify.sh (from https://github.com/NicolasBernaerts/debian-scripts/blob/master/telegram/telegram-notify)

#!/bin/bash
# ---------------------------------------------------
#  Send notification to a telegram account thru a bot account
#  Configuration is stored in /etc/telegram-notify.conf
#  Depends on curl 
#
#  Revision history :
#    10/01/2016, V1.0 - Creation by N. Bernaerts
#    22/01/2016, V1.1 - Handle emoticons
#    06/08/2016, V1.2 - Add API key and User ID parameters
#                       Remove dependency to PERL
#    08/08/2016, V1.3 - Add --document, --html and --silent options
#    10/11/2016, V1.4 - Add --icon option
#    11/01/2018, V1.5 - Add possibility of piped text, idea from Yasir Atabani
#    19/05/2018, V1.6 - Add socks5 proxy option, idea from RangerRU
#    28/06/2018, V1.7 - Add --warning and --config options, idea from Markus Hof
# ---------------------------------------------------

# initialise variables
NOTIFY_TEXT=""
DISPLAY_TEXT=""
DISPLAY_PICT=""
DISPLAY_ICON=""
DISPLAY_MODE="markdown"
DISPLAY_SILENT="false"

# Configuration file
FILE_CONF="/etc/telegram-notify.conf"

# -------------------------------------------------------
#   Check tools availability
# -------------------------------------------------------

command -v curl >/dev/null 2>&1 || { echo "[Error] Please install curl"; exit 1; }

# -------------------------------------------------------
#   Loop to load arguments
# -------------------------------------------------------

# if no argument, display help
if [ $# -eq 0 ] 
then
  echo "Tool to send a message to a Telegram User or Channel."
  echo "Message is sent from a Telegram Bot and can contain icon, text, image and/or document."
  echo "Main parameters are :"
  echo "  --text <text>       Text of the message or file with the text (use - for piped text)"
  echo "  --photo <file>      Image to display"
  echo "  --document <file>   Document to transfer"
  echo "Options are :"
  echo "  --title <title>     Title of the message (if text message)"
  echo "  --html              Use HTML mode for text content (markdown by default)"
  echo "  --silent            Send message in silent mode (no user notification on the client)"
  echo "  --config <file>     use alternate config file, instead of default ${FILE_CONF}"
  echo "  --user <user-id>    Recipient User or Channel ID (replaces user-id= in ${FILE_CONF})"
  echo "  --key <api-key>     API Key of your Telegram bot (replaces api-key= in ${FILE_CONF})"
  echo "Optional icons are :"
  echo "  --success           Add a success icon"
  echo "  --warning           Add a warning icon"
  echo "  --error             Add an error icon"
  echo "  --question          Add a question mark icon"
  echo "  --icon <code>       Add an icon by UTF code (ex 1F355)"
  echo "Here is an example of piped text :"
  echo "  echo 'text to be displayed' | telegram-notify --success --text -"
  exit
fi

# loop to retrieve arguments
while test $# -gt 0
do
  case "$1" in
    "--text") shift; DISPLAY_TEXT="$1"; shift; ;;
    "--photo") shift; PICTURE="$1"; shift; ;;
    "--document") shift; DOCUMENT="$1"; shift; ;;
    "--title") shift; DISPLAY_TITLE="$1"; shift; ;;
    "--html") DISPLAY_MODE="html"; shift; ;;
    "--silent") DISPLAY_SILENT="true"; shift; ;;
    "--config") shift; FILE_CONF="$1"; shift; ;;
    "--user") shift; USER_ID="$1"; shift; ;;
    "--key") shift; API_KEY="$1"; shift; ;;
    "--success") DISPLAY_ICON=$(echo -e "\U2705"); shift; ;;
    "--warning") DISPLAY_ICON=$(echo -e "\U26A0"); shift; ;;
    "--error") DISPLAY_ICON=$(echo -e "\U1F6A8"); shift; ;;
    "--question") DISPLAY_ICON=$(echo -e "\U2753"); shift; ;;
    "--icon") shift; DISPLAY_ICON=$(echo -e "\U$1"); shift; ;;
    *) shift; ;;
  esac
done

# -------------------------------------------------------
#   Read configuration
# -------------------------------------------------------

# if configuration file is present
if [ -f "${FILE_CONF}" ]
then
    # display used config file unless --silent parameter is used
    [ "${DISPLAY_SILENT}" = "false" ] && echo "[Info] Using configuration file ${FILE_CONF}"

    # if needed, load from configuration file
    [ "${API_KEY}" = "" ] && API_KEY=$(cat "${FILE_CONF}" | grep "api-key=" | cut -d'=' -f2)
    [ "${USER_ID}" = "" ] && USER_ID=$(cat "${FILE_CONF}" | grep "user-id=" | cut -d'=' -f2)

    # load socks proxy from configuration file
    SOCKS_PROXY=$(cat "${FILE_CONF}" | grep "socks-proxy=" | cut -d'=' -f2)
else
    # display warning unless --silent parameter is used
    [ "${DISPLAY_SILENT}" = "false" ] && echo "[Warning] Configuration file missing ${FILE_CONF}"
fi

# check API key and User ID
[ "${API_KEY}" = "" ] && { echo "[Error] Please provide API key or set it in ${FILE_CONF}"; exit 1; }
[ "${USER_ID}" = "" ] && { echo "[Error] Please provide User ID or set it in ${FILE_CONF}"; exit 1; }

# -------------------------------------------------------
#   Check for file existence
# -------------------------------------------------------

# if picture, check for image file
[ "${PICTURE}" != "" -a ! -f "${PICTURE}" ] && { echo "[error] Image file ${PICTURE} doesn't exist"; exit 1; }

# if document, check for document file
[ "${DOCUMENT}" != "" -a ! -f "${DOCUMENT}" ] && { echo "[error] Document file ${DOCUMENT} doesn't exist"; exit 1; }

# -------------------------------------------------------
#   String preparation : space and line break
# -------------------------------------------------------

# if text is to be read from pipe, get it
[ ! -t 0 -a "${DISPLAY_TEXT}" = "-" ] && DISPLAY_TEXT=$(cat)

# if text is a file, get its content
[ -f "${DISPLAY_TEXT}" ] && DISPLAY_TEXT=$(cat "${DISPLAY_TEXT}")

# convert \n to LF
DISPLAY_TEXT=$(echo "${DISPLAY_TEXT}" | sed 's:\\n:\n:g')

# if icon defined, include ahead of notification
[ "${DISPLAY_ICON}" != "" ] && NOTIFY_TEXT="${DISPLAY_ICON} "

# if title defined, add it with line break
if [ "${DISPLAY_TITLE}" != "" ]
then
    # convert title according to Markdown or HTML
    [ "${DISPLAY_MODE}" = "html" ] && NOTIFY_TEXT="${NOTIFY_TEXT}<b>${DISPLAY_TITLE}</b>%0A%0A" \
                       || NOTIFY_TEXT="${NOTIFY_TEXT}*${DISPLAY_TITLE}*%0A%0A"
fi

# if text defined, replace \n by HTML line break
[ "${DISPLAY_TEXT}" != "" ] && NOTIFY_TEXT="${NOTIFY_TEXT}${DISPLAY_TEXT}"

# -------------------------------------------------------
#   Notification
# -------------------------------------------------------

# default option
ARR_OPTIONS=( "--silent" "--insecure" )

# if needed, socks5 option
[ "${SOCKS_PROXY}" != "" ] && ARR_OPTIONS=( "${ARR_OPTIONS[@]}" "--socks5-hostname" "${SOCKS_PROXY}" )

# if photo defined, display it with icon and caption
if [ "${PICTURE}" != "" ]
then
  # display image
  CURL_RESULT=$(curl "${ARR_OPTIONS[@]}" --form chat_id=${USER_ID} --form disable_notification=${DISPLAY_SILENT} --form photo="@${PICTURE}" --form caption="${NOTIFY_TEXT}" "https://api.telegram.org/bot${API_KEY}/sendPhoto")

# if document defined, send it with icon and caption
elif [ "${DOCUMENT}" != "" ]
then
  # transfer document
  CURL_RESULT=$(curl "${ARR_OPTIONS[@]}" --form chat_id=${USER_ID} --form disable_notification=${DISPLAY_SILENT} --form document="@${DOCUMENT}" --form caption="${NOTIFY_TEXT}" "https://api.telegram.org/bot${API_KEY}/sendDocument")

# else, if text is defined, display it with icon and title
elif [ "${NOTIFY_TEXT}" != "" ]
then
  # display text message
  CURL_RESULT=$(curl "${ARR_OPTIONS[@]}" --data chat_id="${USER_ID}" --data "disable_notification=${DISPLAY_SILENT}" --data "parse_mode=${DISPLAY_MODE}" --data "text=${NOTIFY_TEXT}" "https://api.telegram.org/bot${API_KEY}/sendMessage")

#  else, nothing, error
else
  # error message
  echo "[Error] Nothing to notify"
  exit 1
fi

# check curl request result
echo ${CURL_RESULT} | grep '"ok":true' > /dev/null || { echo ${CURL_RESULT}; exit 1; }

# end
exit 0

Usage

As explained on the script author's blog you can:

telegram-notify --success --text "Action *sucessful* with markdown *bold* example"
telegram-notify --error --title "Error" --text "Error message with a title"
telegram-notify --question --title "File content display" --text "/tmp/log.txt"
telegram-notify --icon 1F355 --text "Message with custom icon 1F355 and embedded image" --photo "/tmp/icon.png"
telegram-notify --text "Result is available in the embedded document" --document "/tmp/result.log"

See: http://www.bernaerts-nicolas.fr/linux/75-debian/351-debian-send-telegram-notification

Warning Script

Create a storage-warning-telegram.sh

#!/bin/bash
CURRENT=$(df | grep '/var' | awk '{print $5}' | sed 's/%//g')
THRESHOLD=90

if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
    . /etc/telegram/notify.sh --error --title "Storage Warning" --text "Your /var partition available space is critically low. Used: $CURRENT%"
fi

Then run your daily script with a cronjob:

0 1 * * * . /home/mydnic/storage-warning/storage-warning-telegram.sh >> /dev/null 2>&1