Segfault > IT > Webserver > Root server

First steps to take when setting up a root server

I just rented a Virtual Private Server (VPS) which allows me to install and run whatever I want as I have root access to the OS.
I'm really happy!! :o)) as until now I just had a shared server allowing me to run only plain HTML and PHP-stuff - still nice, but not as powerful as a root-server.

Security is a big concern when owning a server which is potentially accessible by anybody and is always on, I am therefore writing a kind of small guide about what I did to try to secure it hoping to help other people that want/have to do the same.

I don't expect to do this the "perfect" way, but it is at least as a start better than nothing.


The first hours

Already 3 hours after the first boot I started seeing all of a sudden the following messages pop up in /var/log/messages:

...
Apr 10 19:34:58 MYSERVER sshd[9844]: Invalid user administrator from 203.149.62.221
Apr 10 19:35:00 MYSERVER sshd[10552]: Invalid user oracle from 203.149.62.221
Apr 10 19:35:13 MYSERVER sshd[10608]: Invalid user test from 203.149.62.221
...

Well, I was still installing the main binaries in the OS, so after silently "thanking" 203.149.62.221 for the nice scan s/he performed on my system I quickly...

  • changed the listening port of the sshd daemon from the usual 22 to something more esotic (any port between 49152 and 65535)

    This can be set in /etc/ssh/ssd_config by altering the Port parameter.
    Doing this avoids that a "quick ports scan" (scans which look only for well-known open ports, like 21 for FTP, 22 for SSH, 80 for HTTP, etc..) has an easy time finding them on my server. I didn't have any other ports open.

    Of course as well the new port will be found if the aggressor performs a full scan on your server, but that will take time, and time is everything, right?
  • disabled SSH-access using a password.

    To do this you'll have to copy your own public key you previously generated (see Linuxssh), so e.g. id_dsa.pub into the file authorized_keys in your ~/.ssh/ directory on your server.

    After doing that check by ssh-ing to the server that you can login without entering a password and if that works (and ONLY IF THAT WORKS! Otherwise after continuing you won't be able to login anymore!!!) change in /etc/ssh/sshd_config the parameter ChallengeResponseAuthentication to no.
  • disabled root-logins.
    • Create on the server first of all another user (e.g. user_itsme):
         useradd -m user_itsme
    • Then, after having given him a password (passwd user_itsme - pls. a different one than the one root has) deploy the public certificate of the user you're using to connect to the server into the authorized_keys file of the user_itsme on the remote server - if in the previous step you used root to connect and login, you can actually copy his authorized_keys-file and change file owner to user_itsme (chown user_itsme authorized_keys) and file permissions (chmod -R 700 ~/.ssh).

      Now you should be able to connect to server serverX with ssh -p 123456 user_itsme@serverX without entering a password - make sure that works before continuing.
    • Now install sudo and give the user access to run sudo su - root (Sudo installation instructions). In my case I inserted (USING THE COMMAND VISUDO!!!) into /etc/sudoers the line user_itsme serverX=/bin/su - root. Again try if that works with sudo su - user_itsme [user_itsme password] and continue only if that works.
    • Final step: now that you're absolutely sure that you can use a different user to 1) login 2) sudo to root, change in /etc/ssh/sshd_config the parameter PermitRootLogin from yes to no and restart sshd (/etc/init.d/sshd restart). Now you shouldn't be able anymore to directly login as root, doesn't matter if the certificates you're using are valid or not.

Scan

  • After having increased the login security I did a scan for open ports on my server:
    nmap -p0-65535 [target server IP]
  • After a while it returned a list of open ports. After that I installed nessus:
    Run nessus-mkcert to generate the certificate.
    Run nessus-adduser to deploy a new nessus user. I chose the standard options.
    Then start up nessus and write in the Target tab the IP of your server.

In both cases nothing interesting was returned - very well.

Mail

Open the file /etc/mail/aliases and under the section # Well-known aliases -- these should be filled in! put your email address, e.g.:

root: itsme@myprovider.com
operator: itsme@myprovider.com

Now have a look at /etc/ssmtp/ssmtp.conf. Configure it accordingly, e.g.:

# /etc/ssmtp.conf -- a config file for sSMTP sendmail.

# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=[target@email.address]

mailhub=mail.[myprovider].com:25
rewriteDomain=
hostname=mail.[myprovider].com:25
UseSTARTTLS=YES
AuthUser=[myuserid]
AuthPass=[mypassword]
FromLineOverride=YES # optional

And, last but not least, configure /etc/ssmtp/revaliases the sender's address. In my case:

root:SERVERX@donotreply.com:mail.[myprovider].com:25

The above settings might or might not work - depends on the how the mailserver you're trying to connect to is configured. Start with a configuration with your real data and then modify it slowly.
I admin it's not the greatest mail configuration around, but it seem to work.

Logs

I installed logrotate to take care about the system logs that are written in /var/log/ - it will split the logs, compress the old ones and delete the very old ones.
Its configuration file is /etc/logrotate.conf and its syntax is very simple. In Gentoo a cron file should be created automatically upon emerging.

As system logger I used "syslog-ng". I configured it to have still as default all messages to be written to /var/log/messages, plus some filters to specialize some of them (e.g. with the below configuration the failed logins will be written to messages plus auth.log).

Here the config file I used (/etc/syslog-ng/syslog-ng.conf):

options {
        chain_hostnames(off);
        sync(0);
        # The default action of syslog-ng 1.6.0 is to log a STATS line
        # to the file every 10 minutes.  That's pretty ugly after a while.
        # Change it to every 12 hours so you get a nice daily update of
        # how many messages syslog-ng missed (0).
        stats(43200);
};

#2 sources: a global one (src) and a specialized one for kernel-thingies (kernsrc)
source src {
    unix-stream("/dev/log" max-connections(256));
    internal();
    file("/proc/kmsg");
};
source kernsrc { file("/proc/kmsg"); };

#Destinations
destination messages { file("/var/log/messages"); };
destination authlog { file("/var/log/auth.log"); };
destination syslog { file("/var/log/syslog"); };
destination cron { file("/var/log/cron.log"); };
destination daemon { file("/var/log/daemon.log"); };
destination kern { file("/var/log/kern.log"); };
destination lpr { file("/var/log/lpr.log"); };
destination user { file("/var/log/user.log"); };
destination mail { file("/var/log/mail.log"); };
destination mailinfo { file("/var/log/mail.info"); };
destination mailwarn { file("/var/log/mail.warn"); };
destination mailerr { file("/var/log/mail.err"); };
destination newscrit { file("/var/log/news/news.crit"); };
destination newserr { file("/var/log/news/news.err"); };
destination newsnotice { file("/var/log/news/news.notice"); };

#Filters
filter f_authpriv { facility(auth, authpriv); };
filter f_syslog { not facility(authpriv, mail); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_user { facility(user); };
filter f_info { level(info); };
filter f_warn { level(warn); };
filter f_crit { level(crit); };
filter f_err { level(err); };

#Links between source, filters and destinations
log { source(src); destination(messages); };
log { source(src); destination(console_all); };
log { source(src); filter(f_authpriv); destination(authlog); };
log { source(src); filter(f_syslog); destination(syslog); };
log { source(src); filter(f_cron); destination(cron); };
log { source(src); filter(f_daemon); destination(daemon); };
log { source(kernsrc); filter(f_kern); destination(kern); };
log { source(src); filter(f_lpr); destination(lpr); };
log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_user); destination(user); };
log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };
log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); };
log { source(src); filter(f_mail); filter(f_err); destination(mailerr); };

After organizing how logs got written I had to set up an automatic notification in case that something nasty shows up in them. I used Logcheck, which is contained in Gentoo in the logsentry package.
I configured it in /etc/logcheck/logcheck.sh just by setting my email address and left everything else as default:

SYSADMIN=[myemailaddress]

I tested it with /etc/logcheck/logcheck.sh and after a few seconds I got an email at home.
Refine the contents of /etc/logcheck/logcheck.ignore for the strings (e.g. *ciao*) that are supposed to be ignored - be careful not to set just "*" which would make logcheck ignore everything.

To run it automatically I used the usual vixie-cron. I inserted the name of the login user into /etc/cron.deny just to be sure that he doesn't have access to run any cron jobs.
Here there is a guide which explains how to set up new cron jobs. So, I ran crontab -e and as I wanted to have logcheck to run every 10 minutes I entered:

#Mi     HH      DD      Mo      Day of week
#START-logsentry/logcheck every 10 minutes
*/10    *       *       *       *       /etc/logcheck/logcheck.sh
#END-logsentry/logcheck every 10 minutes

One last thing I wanted to do was to set in /etc/security/limits.conf the maximum file size for the login user:
   youruser hard fsize 10240 #Max file size 10MB

Afterwards I checked if any files were writable by all with...
   find / -type f \( -perm -2 \) -exec ls -lg {} \; 2>/dev/null >output.txt
..., did a chmod -R 750 /etc/ssmtp directory and that's it.
I am sure that file security is not good, and well, I don't really know much concerning that area.

Firewall

As I don't have another server with multiple network cards available to make it act as a secure firewall, I had to install the firewall directly on this server.
I used iptables as firewall as it is the most used and tested firewall around.

To enable iptables it's best that you modify your kernel configuration and enable EVERYTHING as module under Networking support => Networking options => Network packet filtering framework (Netfilter) in both the Core netfilter configuration and IP: Netfilter configuration.

Be sure to check multiple times in both submenus as some modules pop up in one submenu only when others in the other submenu are selected.
I was missing some of them (e.g. the ip_conntrack_ftp module) and iptables was popping up all the time the stupid iptables: No chain/target/match by that name or iptables: Invalid argument error messages and only after several hours of investigating the syntax of my commands I thought that maybe I was missing some modules :o|

!!!Be sure to first test on a local machine what you do next as if you try it out directly on your remote server and the settings are wrong you'll be cut out. You'll have as well to prepare a startup script which executes this, because if you do it in a terminal you'll be cut out right at the beginning because the first commands you execute are the default behavours which will make you disconnect from the terminal!!!

In the end all you need is a script which configures the firewall each time you boot your server.
I created the following one manually - I advice you to do the same to know exactly how the server will react. To write the script I advice you to:

  • 1. buy (and read) the book Linux Server Security (or Linux Server-Sicherheit for the german version) written by Michael D. Bauer
  • 2. download and install on your local PC Firewall Builder (create a copy of your rules in there - it will help you when you'll get stuck when doing things manually)
  • 3. experiment on your local PC and when things don't work check with netstat -tn which ports are being used by whatever got stuck - lsof | grep [searchstring_e.g._ip] will help you too.

Ok, so here is the first version of a script I wrote (mostly taken from Michael D. Bauer s book) for my server (changes will vanish after a reboot - you'll have to write a startup script if you want to activate them automatically):

#/bin/bash
#Script iptables_set.sh

#REMINDERS-START
#List everything
#iptables --line-numbers --list
#iptables --line-numbers --list INPUT/OUTPUT/FORWARD

#Delete a rule
#iptables -D OUTPUT 3

#Delete everything
#iptables --flush
#iptables --delete-chain
#iptables -P INPUT ACCEPT
#iptables -P OUTPUT ACCEPT
#iptables -P FORWARD ACCEPT
#REMINDERS-END

#=============================
#START

#Reset everything
iptables --flush
iptables --delete-chain

#Setup the default behaviours
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

#Let everything to the loopback interface
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
#Now a "ping localhost" should work

#=============================
#INPUT-START (From outside world to the server)

#REJECT-START
#Anti-IP-Spoofing rules
iptables -A INPUT -s 255.0.0.0/8 -j LOG --log-prefix "Refused by rule in1\!"
iptables -A INPUT -s 255.0.0.0/8 -j DROP
iptables -A INPUT -s 0.0.0.0/8 -j LOG --log-prefix "Refused by rule in2\!"
iptables -A INPUT -s 0.0.0.0/8 -j DROP
iptables -A INPUT -s 127.0.0.0/8 -j LOG --log-prefix "Refused by rule in3\!"
iptables -A INPUT -s 127.0.0.0/8 -j DROP
iptables -A INPUT -s 192.168.0.0/16 -j LOG --log-prefix "Refused by rule in4\!"
iptables -A INPUT -s 192.168.0.0/16 -j DROP
iptables -A INPUT -s 172.16.0.0/12 -j LOG --log-prefix "Refused by rule in5\!"
iptables -A INPUT -s 172.16.0.0/12 -j DROP
iptables -A INPUT -s 10.0.0.0/8 -j LOG --log-prefix "Refused by rule in6\!"
iptables -A INPUT -s 10.0.0.0/8 -j DROP
#Anti-Stealth rules
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "Refused by rule in8\!"
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
#Auth ident requests
iptables -A INPUT -p tcp -m tcp  --dport 113  -m state --state NEW  -j LOG --log-prefix "Refusd by rule in9\!"
iptables -A INPUT -p tcp -m tcp  --dport 113  -m state --state NEW  -j REJECT
#REJECT-END


#ACCEPT-START
#Traffic that was previously tagged as being ok:
iptables -A INPUT -j ACCEPT -m state --state ESTABLISHED,RELATED
#SSH new traffic
iptables -A INPUT -p tcp -j ACCEPT --dport 22 -m state --state NEW
#SSH special port on server
iptables -A INPUT -p tcp -j ACCEPT --dport 12345 -m state --state NEW
#FTP new traffic
#iptables -A INPUT -p tcp -j ACCEPT --dport 21 -m state --state NEW
#HTTP new traffic
#iptables -A INPUT -p tcp -j ACCEPT --dport 80 -m state --state NEW
#PINGs to the server
iptables -A INPUT -p icmp  -m icmp  --icmp-type 8/0   -m state --state NEW  -j ACCEPT
#Log all the rest - must be the last one
iptables -A INPUT -j LOG --log-prefix "Default in-traffic rejection:"
#ACCEPT-END
#INPUT-END

#=============================

#OUTPUT-START (From the server to the outside world)
#Allow authorized connections - this will make e.g. external SSH connections work
iptables -I OUTPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT

#Allow outgoing SSH connections
#iptables -A OUTPUT -p tcp -j ACCEPT --dport 22 -m state --state NEW
#SSH special port on server
#iptables -A OUTPUT -p tcp -j ACCEPT --dport 12345 -m state --state NEW

#Allow outgoing pings
iptables -A OUTPUT -p icmp -j ACCEPT --icmp-type echo-request

#Allow outgoing DNS requests
iptables -A OUTPUT -p udp --dport 53 -m state --state NEW -j ACCEPT

#Make "whois" work
iptables -A OUTPUT -p tcp -m tcp --dport 43 -m state --state NEW -j ACCEPT

#Make "emerge --sync" (rsync) work:
iptables -A OUTPUT -p tcp -m tcp  --dport 873  -m state --state NEW  -j ACCEPT

#Make "emerge whatever" and HTTP-access  work:
iptables -A OUTPUT -p tcp -m tcp  --dport 80  -m state --state NEW  -j ACCEPT

#Make it possible to send out emails:
iptables -A OUTPUT -p tcp -m tcp  --dport 25  -m state --state NEW  -j ACCEPT

#Make it possible to check for incoming emails:
#iptables -A OUTPUT -p tcp -m tcp  --dport 110  -m state --state NEW  -j ACCEPT

#Log all the rest - must be the last one
iptables -A OUTPUT -j LOG --log-prefix "Default out-traffic rejection"
#OUTPUT-END

You see that I commented out some commands (like the incoming HTTP requests on port 80) as I don't have (yet) such services running.

Once you reviewed, modified and successfully tested the script on your local LAN, upload it to the server and prepare the root-user to execute it (non-root users won't be allowded to change firewall settings).
Remember that you cannot execute the single steps of the scripts one-by-one as after setting the default behaviours (which will reject EVERYTHING) you'll be cut out. It is therefore very important that you execute the script as a whole AND that you have a "rescue"-strategy in place if anything goes wrong.

One of the "rescue-strategies" is for sure the possibility to remotely restart your server using the external command console of your provider. I do have such a console but didn't want to restart the server each time (which takes quite some time), so I wrote this script...

#/bin/bash
#Script runme.sh

/home/myrootuser/iptables_set.sh > output_iptables.txt
sleep 120

#Undo everything
iptables --flush
iptables --delete-chain
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT

...which first runs the commands needed to set up the firewall and then gives me 2 minutes to see if I am still able to connect.
When the 2 minutes elapse the firewall is set to let everything through and you should get back connectivity if you were previously cut out.
Once you are ready you'll have to run the above script as "nohup ./runme & > output.txt".
"&" will run the script in the background, "nohup" will continue to run the script even if you disconnect from the terminal, and ">" will output all (hopefully none) errors to the file "output.txt".

Once you successfully installed the firewall script on your server and have disabled port 22 for incoming connections when running the command...
   nmap -sS -O [IP_of_your_server]
...you should get something like:

~ # nmap -sS -O [IP_of_your_server]

Starting Nmap 4.76 ( http://nmap.org ) at 2009-04-13 19:35 CEST
All 1000 scanned ports on YOURHOST (123.123.123.123) are filtered
Too many fingerprints match this host to give specific OS details

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 26.82 seconds

So, now it is time for you to set the firewall script in your startup scripts, reboot the server and keep fingers crossed. After the server has rebooted check (if you can still connect to it) with iptables --line-numbers --list if everything is still in place.

One last thing: if you're using the Bash shell, write the following into the ~/.bash_logout-file of your users to clear their history when they logout so that no traces which might eventually help an intruder knowing what you last did are left:

history -c
echo ciao > ~/.bash_history


References

http://www.gentoo.org/doc/en/security/security-handbook.xml?full=1

Notes

  • On Gentoo run from time to time emerge --sync followed by "glsa-check -l affected" to see the list of available security updates. Install them if anything pops up.
  • Guide to set up the time on the server using NTP here.
  • For a paravirtualized VM/domU running from within Xen, add...
       echo 1 > /proc/sys/xen/independent_wallclock
    ...to the file "/etc/conf.d/local.start".
  • (02.Feb.2011) If you're setting up your own mailserver, be prepared for a lot of action (lot of people trying to break in). After having a look at all options I went for postfix, courier-imap & Co.. A mandatory step is to set up iptables as firewall in combination with fail2ban.

    For "fail2ban" to work with smtp you will have to modify /etc/fail2ban/filter.d/sasl.conf from...
failregex = (?i): warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication
failed(: [A-Za-z0-9+/]*={0,2})?$
 
...to...
 
failregex = (?i): warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication
failed: authentication failure