Shicks! Manual

This manual describes how to install and use the Shicks! POP3/SMTP server, version 0.8

Background

The first section gives you some background on what Shicks! is, and what it can do for you.

What is POP3?

POP3 is a standardized internet protocol for retrieving mails from a mailbox. POP3 knows about users and mailboxes and how to read mail, but it doesn't know anything about how to send mail. This is why in most mail programs, you have to specify both a POP3 and a SMTP server.

Technical Note: The standard is defined in RFC 1725, which can be found online an http://www.ietf.org/rfc/rfc1725.txt. TCP/IP reserves Port 110 for POP3 servers.

What is SMTP?

SMTP is a standardized internet protocol for sending mails to a local or remote mailbox. SMTP knows how to send mail, but it doesn't know anything about how to read mail, or even what a mailbox is.

Technical Note: The standard is defined in RFC 821, which can be found online an http://www.ietf.org/rfc/rfc821.txt. TCP/IP reserves Port 25 for POP3 servers.

What is Shicks!?

Shicks! is a combined POP3/SMTP server. Actually, it is two distinct servers for each protocol, plus a MySQL database that is the central storage for both servers. So, the POP3 server can read the messages stored locally by the SMTP server in the MySQL database.

The MySQL database holds the mailboxes defined on your Server. For each mailbox, you have a username and a password. When you logon to the POP3 server, the username and password are checked against the database. Here are the most common operations:

When you send a message to another local mailbox, the SMTP server does the following: 1) A file containing a copy of the message is created on the server. 2.) An entry in the database is made that says that this file belongs to the receiving mailbox.
When you look for messages, the POP3 server does the following: 1) Look in the database if a message has been received, 2) Read the data from the harddisk.
When you send a message to a remote user, the "MX record" for that domain is looked up via DNS. The "MX record" assigns a mail server to a domain name. (For example, the domain "foo.bar" could have the mail server "mail.foo.bar" associated with it). Shicks! then directly talks SMTP to that server.
As a last resort, if the mail cannot be sent directly to the "MX record" mailserver, the mail is deferred using a "relay SMTP server" (a remote SMTP server), which usually is your ISP's SMTP server.
When a remote user sends you a message, the first and most important step is for the remote user to target the local SMTP server (See below). Then, everything proceeds as if you send a local message.

To be able to receive mails from remote users (users on the internet) their SMTP server has to find your SMTP server. This is done by resolving the name of the e-mail address. This is done by looking at the part of your e-mail address after the '@'. For example, an e-mail address "somebody@foo.com" will be resolved by looking up DNS entry for "foo.com" and connecting the SMTP server there. Actually, most of the time your ISP's SMTP server will do the name lookup for you.

So, in order to receive mails from remote users, your server must have a known DNS entry on the internet

Note that in Shicks!, in contrast to other mail servers, local addresses can be anything. If the SMTP server recognizes a local address, it sends the mail to the local recipient - no matter what. For example, even if your DNS entry is "foo.com", you can receive messages for "somebody@bar.com" as long as the messages are sent from internal users to internal users. Remote users will not be able to access "somebody@bar.com", because most likely "bar.com" will point to a different SMTP server.

But my machine is behind a firewall!

Well, this probably means you cannot receive remote mail. But Shicks! might still be usefull for you, if only to be able to better control mail sent inside your organisation without going to your ISP for every message. For example, if you share a DSL connection with two or three household members, you can easily use Shicks! to setup a local mail system. Using "full blown" mail servers would be way out of scope for your needs, probably.

What is an SMTP relay server

As explained above, when somebody sends mail to "somebody@foo.com", it will be resolved by looking up DNS entry for "foo.com" and connecting the SMTP server there. This is not 100% correct, you can associate a different mail server with a given domain name. The technical term is a MX record. so, for example, mails to "foo.com" can have a mailserver "mail.foo.com" associated.

A relay server is a SMTP server that serves multiple domains. Typically, your ISP has one for your account, so that you can just "talk SMTP" with your ISPs mail server, and the messages will be redirected to the target recipients. Starting with version 0.8, shicks! does no longer need a relay server and can directly send messages to target domains. To do the MX record lookup, you need to have the nslookup tool installed (in /usr/bin/nslookup on linux, or \winnt\system32\nslookup.exe for win32).

Installation Instructions

Before you start, you need to check that your system meets the following requirements:

Requirements

Python 2.0 or higher.
A MySQL database running.
The MySQLdb module for python must be installed.
A webserver. I use Apache, but other common webservers should work fine, too, provided they can run Python cgi-scripts.
The user must have the rights to access TCP/IP ports 25 (SMTP) and 110 (POP3). Also, no other application must be using these ports! On Linux, you'll get a warning if the port is already in use - on Windows, it will "succeed", but you'll not get much mail.
The operating system must be Linux, or Windows. For Linux you need root access. Other OS might work too, provided that the above mentioned conditions are met. Feedback welcome!

File Overview

In the archive, you'll find the following files (alphabetically sorted)

DATEINAME.PY	Besachreibung

Creating the database

The first step is to create the database. You can do this at the shell prompt using

mysqladmin create maildb

Here, "maildb" is the name of the database. You can choose a different name if you want, but then you'll have to edit the file "maildb.py". If the database admin account has a password, you can use the "-p" command line option to specify it. You would say:

mysqladmin -p create maildb

and then be asked for a password.

The next step is to create the structure of the database. For this, you need to connect to mysql.

From a command line, start "mysql".
Connect to the newly created database.
Execute the SQL script "maildb.sql", which should be part of your Shicks! distribution.

NOTE: Before overwriting the database, please do check if a database named "maildb" already exists and is in possible use by another application.

Here is a sample session on a Win32 machine:

D:\scripts\shicks> c:\mysql\bin\mysql.exe -p
Enter password: **************

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 3.23.43-nt

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> connect maildb;
Connection id:    2
Current database: maildb

mysql> source maildb.sql;
Query OK, 0 rows affected (0.05 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.02 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.03 sec)
Query OK, 0 rows affected (0.03 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.02 sec)
Query OK, 0 rows affected (0.05 sec)
Query OK, 0 rows affected (0.02 sec)

mysql>

You can omit the -p option if your database doesn't need name and password. If everything went OK, you now have the data structure set up completely.

Accessing the database from python

IF you have protected your MySQL database with name and password (and you really should), then you need to edit the file maildb.py. Have no fear, its a very small and simple file that holds your name and password for the database at the top. You need to edit this, otherwise python won't be able to connect to the database. More instructions can be found in the sourcecode.

If something goes wrong

If you had problems in any of the steps in this section, you should verify:

That the MySQL process is up and running. If possible, try to connect to the database with some tool like phpMyAdmin and see if it works.
Check if mysql and mysqladmin are on your executable path. (%PATH% for Windows, $PATH for Linux)
Check if your database is protected by name and password.

Installing the Shicks! Administrator Tool

First of all, open the file "shicks-admin.py" in your favourite editor. You need to change the line that reads

#! c:\\python21\\pythonw.exe -u

to point to the executable of python. (The linux distribution already uses /usr/bin/python as default, but just in case you have installed python to a different directory you should still check if that line is ok).

You need to copy the files

shicks-admin.py
maildb.py

to your servers' cgi-bin directory. The sourcecode currently assumes that the webserver path will always be

http://localhost/cgi-bin/shicks-admin.py

If your webserver has different settings (hint: IIS probably does), you need to either change the source or provide a name mapping between /cgi-bin/ and the directory you install shicks-admin.py to.

Once installed, open a webbrowser and go to http://localhost/cgi-bin/shicks-admin.py . You should see a welcome screen saying "Shicks! Administration".

First-time administration

Check the settings. The path has some defaults, you might want to change these to fit your installation. Note that the path will be created if it doesn't yet exist.

Add at least one user, having one email to the system, so you can later check if it works.

TODO: Add documentation for the admin tool, even though its pretty self-explanatory. Volunteers welcome :)

If something goes wrong

If you had problems in any of the steps in this section, you could take one of the following actions:

Verify that the webserver is up and running
Consult the logfiles of the webserver!
Check if you've changed the path to python in step #1

Starting the servers

Well, this is the easy bit: Just run POP3Server.py and SMTPServer.py. On Windows, if you have the ActivePython distribution installed, you can simply doubleclick the files. (You'll be presented with two console windows saying "POP3Server/SMTPServer is up & running"). On Linux, you should start the apps from a xterm.

Linux only: Determining the installation path for python

The python scripts come with the header line "#!/usr/bin/python". If your copy of python 2.0 resides in a different directory (or is named differently, e.g. "python2.1"), you must modify the script headers. An alternative would be to manually start the files by using a syntax like this:

root@zx81.oldskool.de $ /usr/local/bin/python2.1 SMTPServer.py
etc.

Installing the Shicks! cgi-Administrator

Copy the file "shicks-admin.py" to your httpd/cgi-bin directory and mark it as executable (the last step is only required on linux). Then, point your browser to http://localhost/cgi-bin/shicks-admin.py and off you go.

Editing the first-time configuration.

When you first started shicks!, it created some default configuration entries. You should now take the time to edit them to your personal needs. Go to the Shicks! Administrator and click on Configuration. The following settings are available:

Logfile directory: This is the directory where the shicks! logfiles are kept. It defaults to /usr/share/shicks/log on Linux, and c:\shicks\log on Windows.
Relay SMTP server: This is the address of the relay server you want to use. Normally, this is the SMTP server of your provider.
Use relay STMP server always: Normally, starting with version 0.8 mails are sent directly to the MX records for the address domain. (See chapter on What is an SMTP relay server above). You can however force shicks! to always use the relay server, if you set this variable to 1.
Directory for mails: This is the directory where the mails are kept. It defaults to /usr/share/shicks/mail on Linux, and c:\shicks\mail on Windows.
Admin account: This is an email address, where administrative messages will be sent to. Example: postmaster@acme.com
Local domain: This is the name of this domain. Example: acme.com
Unknown account: This is an email address for messages sent to the local domain but without a valid recipient. Example: unknown@acme.com
Local IP-Range: This is an (optional) list of local IP addresses. Local IP addresses are not scanned for spam when sending, and are allowed to send to any reciever. Example: 123.14.14.50-123.14.18.60
Strict relay test: If set to 1, either the sender or ALL receivers must be local addresses. If set to 0, either the sender or AT LEAST ONE OF the receivers must be local addresses. Defaults to 1.
Catch-all Normally, if the target address is on your local domain, but refers to an unknown user, the Unknown account will get the message. However, you can force Shicks! to only accept messages for configured users, if you disable Catch-All by setting this to 0.
Force SMTP auth Enables you to force SMTP auth for all connection (if you really want to)
Deny mails from these hosts You can configure a list of hosts which you don't want mails from. Example: *hotmail;*.kr
Physically delete mails after they have been fetched - self explaining ;)

SPAM detection

shicks! tries to automate spam detection by analyzing incoming subject lines. The keywords recognized are in the file "spam_data.py". You can edit this file if you have new subject lines.

Using DNSBL

Starting with version 0.6, shicks! uses DNSBL (DNS-based blacklists for spammers) filtering. This is enabled by default, so you don't need to change any configuration settings to get this. The following lists are checked:

You can add more DNSBLs in the settings, look for BlacklistDomains in the configuration

Using filter expressions

Both LocalIpRange and DenyHostsList now support regular expressions. The syntax is explained in the following. Each filter expression is a semicolon separated list of subexpressions:

<spec #1>;<spec #2>;<spec #3> ...

For example,

127.0.0.1;123.123.123.123;some.domain.com

consists of three distinct subexpressions, "127.0.0.1","123.123.123.123" and "some.domain.com". Each subexpression can be

A fully qualified domain name (e.g. "some.domain.com")
A fully qualified IP address (e.g. "127.0.0.1")
A partially qualified IP address using wildcards * (any number of characters) and ? (any single character) (e.g. "64.0.0.*")
If the expression starts with $, it is interpreted as a python regular expression. Refer to the Python Regular Expression Howto for an explanation of Python RE syntax.
An IP range. Both start and end of the IP range must be fully qualified IP addresses (e.g. "(212.0.0.0-212.0.0.255)")

Enabling SMTP auth

If you really want to, you can force-enable SMTP AUTH. To do so, set RequireSmtpAuth in the configuration to true. Be warned though, every mail client I've seen handles SMTP AUTH differently. I have tested with Pythons smtplib, Microsoft Outlook, and Mozilla Mail Client. Others may work, but they also may not work (SMTP AUTH stinks).

Checking if the mail works

First of, configure your email client. Email client configuration varies widely, so I cannot give a general manual on that; but all Email clients should have configuration entries for POP3 server and SMTP server, respectively. As server name, choose the name of the machine you let both servers run (for testing, this can be localhost [127.0.0.1]).

Then, you should check, if:

You can send yourself a plaintext message
You can send yourself a mail with attachment(s)
You can send yourself a HTML e-mail
You can send mail to a mailing list (and have it received, e.g. if you send it to a mailing list you yourself are a member of)
At this point, also look in to the Shicks! Administrator again, to see if the mail statistics are correctly.

Provided that you've set up a relay SMTP server, you can try the following:

You can send e-mail to someone outside your organisation.
You can receive e-mail from someone outside your organisation.

If everything works fine, you've got yourself a shiny new Mailserver :)