ASCII Smiley Face Daniel Dickinson Mini Headshot
The C Shore

Yet Another Web Server HOWTO

Version 1.0.0~rc2

Note

This is currently in ‘recipe’ format and doesn’t explain why or go into depth. Future plan for this doc is to be more detailed in those areas.

What you get

  • Web Server

    • Static virtual hosting
    • Let’s Encrypt SSl certificates (also used by mail server)
    • Stats virtual host (authorized users only)
  • Admin

    • Some attack reduction and blocking
    • Some stats

Not in this document

Future work

  • Add alternate path for web app support (e.g. CGI, uwSGI, etc), e.g. for webmail

Out of Scope (i.e. Lots of Other Documentation Sources for These)

  • Initial host/instance setup
  • General admin utilities and convenience setup

Prerequisites

  • An internet accessible host
  • DNS Service (such as dyn.com, self-hosted, etc)
  • For this HOWTO: CentOS 7, 1 GiB RAM, 10 GiB HD (e.g. virtual HD)
  • Repos
    • Defaults + EPEL (to install epel do yum install epel-release)

Packages

The following packages need to be installed for this setup (e.g. yum install package1 package2 ...)

Admin Tools

  • policycoreutils
  • policycoreutils-python

Web Server

  • httpd-tools
  • lighttpd

Let’s Encrypt / ACME SSL Certificates

  • certbot

Stats

  • awstats

Attack Detection / Blocking

  • fail2ban
  • fail2ban-firewalld
  • fail2ban-server

First Steps

  1. Configure networking,admin users etc for your host/instance
  2. (Optional) Install your preferred admin/monitoring utilities etc.
  3. Install “Admin Tools” listed above
  4. Add ‘EPEL’ repository listed above

Web Server & Let’s Encrypt Configuration

If you are not serving web pages or apps, other Certbot configuration might be more suitable for getting Let’s Encrypt SSL certificates for your mail server.

Web Server (HTTP: only serves ACME / Let’s Encrypt verification)

  1. Install “Web Server” packages above
  2. In file /etc/lighttpd/lighttpd.conf

    1. below

        var.vhosts_dir = server_root + "/vhosts"
      

      add

        var.vhosts_acme_dir = server_root + "/vhosts-acme"
      
    2. At end, add:

        $HTTP["scheme"] == "http" {
          include "/etc/lighttpd/vhosts-acme.d/*.conf"
        }
      
  3. Configure IP binding

    1. Uncomment the server.bind = "localhost" line and replace localhost with your hostname (e.g. example.com)
    2. Below it add $SERVER["socket"] == "0.0.0.0:80" { }. Optionaly replace 0.0.0.0 with your ipv4 address.
  4. In file, /etc/lighttpd/conf.d/dirlisting.conf

    • set dir-listing.activate to "disable" (unless you want a browse-able list of files and directories for directories without an index file (e.g. index.html, index.htm, etc))
  5. In file /etc/lighttpd/modules.conf

    1. inside the directive server.modules =, uncomment or add the following
      • mod_alias,
      • mod_redirect,
    2. If you want to redirect example.com to www.example.com, after the closing parentheses for server-modules add

        $HTTP["scheme"] == "http" {
           $HTTP["host"] == "example.com" {
               url.redirect = (".*" => "http://www.example.com$0")
           }
        }
      
    3. Uncomment include "conf.d/compress.conf"

    4. Execute mkdir -p /var/cache/lighttpd/compress

    5. Peform restorecon -Rv /var/cache/lighttpd

  6. For each static virtual host to serve, under /etc/lighttpd/vhosts-acme.d/ include a file such as www.example.com.conf (NB: The filename must end in .conf):

        $HTTP["host"] == "www.example.com" {
          var.server_name = "www.example.com"
          server.name = server_name
    
          server.document-root = vhosts_acme_dir + "/www.example.com"
          accesslog.filename = log_root + "/" + server_name + "/access.log"
        }
        ##
        #######################################################################
    
  7. Run lighttpd-angel -f /etc/lighttpd/lighttpd.conf and correct any errors detected.

  8. Issue setsebool -P httpd_setrlimit on

  9. In /etc/lighttpd/lighttpd.conf set server.max-connections=512 or set server.max-fd=2048 (depending on traffic and resources)

  10. Make the log directories you pointed to above, e.g. mkdir -p /var/log/lighttpd/www.example.com

  11. Create an empty log file: e.g. touch /var/log/lighttpd/www.example.com/access.log

  12. Set ownership so lighttpd can write to it: chown -R lighttpd:adm /var/log/lighttpd/www.example.com

  13. Execute systemctl restart lighttpd

  14. Do systemctl enable lighttpd

Web Server (HTTPS: the real servers; requires SSL)

Prerequisites: Installed and configured web server as above.

Let’s Encrypt (Part 1)

  1. Install “Let’s Encrypt” package above (certbot)
  2. Add certbot user and group

      adduser -U --system -M --home-dir /etc/letsencrypt certbot
      passwd -l certbot
    
  3. mkdir -p /etc/letsencrypt/renewal-hooks/deploy && chown -R certbot:certbot /etc/letsencrypt

  4. Place the following script in /etc/letsencrypt/renewal-hooks/deploy as lighttpd and make it executable

      #!/bin/sh
    
      RET=0
      for CERTBOT_DOMAIN in $RENEWED_DOMAINS; do
        cd /etc/letsencrypt/live/${CERTBOT_DOMAIN} && cat fullchain.pem privkey.pem >lighttpd.pem && chmod 640 lighttpd.pem || RET=1
      done
      exit $RET
    
  5. Place the following script in /etc/cron.daily as certbot and make it executable

      #!/bin/sh
    
      /bin/su -c "certbot -q -n renew" certbot
    
  6. Issue mkdir -p /var/lib/letsencrypt && chown certbot:certbot /var/lib/letsencrypt

  7. Do mkdir -p /var/log/letsencrypt && chown certbot:adm /var/log/letsencrypt

  8. Perform certbot register

Lighttpd configuration (Part 2)

  1. Create ACME / Let’s Encrypt verification directories (these will be internet accessible from per-vhost directories down (e.g. www.example.com in the example below will be the web root))

      mkdir -p /var/www/vhosts-acme/www.example.com/.well-known
      chown -R certbot:certbot /var/www/vhosts-acme/www.example.com/.well-known
    
  2. Repeat for each vhost you want to enable

  3. Issue

       firewall-cmd --add-service http
       firewall-cmd --add-service https
       firewall-cmd --runtime-to-permanent
    

Let’s Encrypt (Part 2)

  1. Issue

      su - certbot
      certbot certonly --staging --webroot -w /var/www/vhosts-acme/www.example.com -d www.example.com -d example.com
    
  2. If the command completes successfully, repeat the command, without --staging.

      cd /etc/letsencrypt/live/<default-domain>     # In the example above <default-domain> is www.example.com
      cat fullchain.pem privkey.pem >lighttpd.pem
      chmod 640 lighttpd.pem
      exit
    

Note that Certbot recommendation is to put all hostnames on one certificate (see certbot --help --webroot for more information), unless one has specific reasons for having separate certificates. If you need separate certificates than you will need to issue the commands above for each certificate or set of certificates.

Lighttpd configuration (Part 3)

  1. In /etc/lighttpd/lighttpd.conf, before

      $HTTP["scheme"] == "http" {
          include "/etc/lighttpd/vhosts-acme.d/*.conf"
      }
    

which you added previously, add

     $SERVER["socket"] == "0.0.0.0:443" { include "ssl.conf" }
     $SERVER["socket"] == "[::]:443" { include "ssl.conf" }
  1. Add a file /etc/lighttpd/ssl.conf that looks like:

      ssl.engine                  = "enable"
      ssl.pemfile                 = "/etc/letsencrypt/live/www.example.com/lighttpd.pem"
      include "/etc/lighttpd/vhosts.d/*.conf"
    
  2. And add a file /etc/lighttpd/vhosts.d/www.example.com.conf for each virtual host you are adding (in this case www.example.com; and not including the Radicale vhost), that looks like:

        $HTTP["host"] == "www.example.com" {
        var.server_name = "www.example.com"
    
        server.name = server_name
        server.document-root = vhosts_dir + "/www.example.com"
        accesslog.filename          = log_root + "/" + server_name + "/access.log"
      }
    
  3. In file /etc/lighttpd/modules.conf

    1. inside the directive server.modules =, uncomment or add the following

      • "mod_openssl",
    2. And if you want to use HTTP authentication for some pages or sites, also uncomment or add the following

      • "mod_auth",
      • "mod_authn_file",
    3. If you want to redirect HTTP to HTTPS, before

        $HTTP["scheme"] == "http" {
           $HTTP["host"] == "example.com" {
               url.redirect = (".*" => "http://www.example.com$0")
           }
        }
      

      which you added above, add

        $HTTP["scheme"] == "http" {
            $HTTP["host"] == "example.com" {
                url.redirect = (".*" => "https://www.example.com$0")
            }
         }
      

      and after, add

         $HTTP["scheme"] == "http" {
             $HTTP["url"] !~ "^/.well-known/acme-challenge/" {
                 $HTTP["host"] =~ ".*" {
                     url.redirect = ( "^/(.*)" => "https://%0/$1" )
                 }
             }
         }
      
  4. If you uncommented/added an mod_auth* line, make sure /etc/lighttpd/conf.d/auth.conf is all commented out

  5. If you want to require HTTP authentication for a page or site

    1. in /etc/vhosts.d/<host-to-require-auth>, above the closing brace (}) in the <host>.conf listed previously, add something similar to:

        auth.backend = "htdigest"
        auth.backend.htdigest.userfile = "/etc/lighttpd/users/<host>"
        auth.require = ( "/" =>
          (
               "method" => "digest",
               "realm" => "[auth-realm]",
               "require" => "valid-user"
          )
        )
      
    2. Use htdigest to create /etc/lighttpd/users/<host> (man htdigest for details)

  6. Do mkdir -p /var/www/vhosts/www.example.com for each static vhost you are creating.

DNS Setup

  • Add a DNS A record for you ‘base hostname’ (e.g. for example.com with ).
  • Add a DNS CNAME record for any virtual hosts (e.g. for www.example.com add a CNAME record pointing to example.com)
  • Your bare domain name (e.g. example.com) should be A or AAAA records and not CNAMEs. This also helps if you move some subdomains, or the top level domain and some subdomains to services like Netlify.

Stats Configuration

SELinux Tweaks for AWStats Static Reports

  1. Create a file awstats_cron.te (e.g. in /root) as below

     module awstats_cron 1.0;
    
     require {
         type httpd_sys_content_t;
         type awstats_t;
         class capability { dac_override dac_read_search };
         class dir search;
         class file { open write getattr ioctl };
     }
    
     #============= awstats_t ==============
     allow awstats_t httpd_sys_content_t:dir search;
     allow awstats_t httpd_sys_content_t:file { open write getattr ioctl };
     allow awstats_t self:capability { dac_override dac_read_search };
    
    1. Execute checkmodule -M -m awstats_cron.te -o awstats_cron.mod
    2. Perform semodule_package -o awstats_cron.pp -m awstats_cron.mod
    3. Run semodule -i awstats_cron.pp

AWStats Static Reports

To avoid enabling CGI and therefore increasing attack surface when we are just serving static pages, we generate stats once an hour and put them in /awstats and serve them statically.

Lighttpd Configuration

  1. It is recommended that you use HTTP AUTH as described above in your vhost.
  2. You should also add the following to your /etc/lighttpd/vhosts.d/<vhost>.conf file

      alias.url = (
         "/icon/" => "/usr/share/awstats/wwwroot/icon/",
          "/css/" => "/usr/share/awstats/wwwroot/css/",
          "/js/" => "/usr/share/awstats/wwwroot/js/",
          "/classes/" => "/usr/share/awstats/wwwroot/classes/",
          "/awstats-docs/" => "/usr/share/doc/awstats-7.7/"
      )
    
  3. Make sure that /var/www/vhosts/<hostname>/awstats and /var/www/vhosts/<hostname>/awstats-year exist.

  4. Create /var/www/vhosts/<hostname>/index.html, such as:

      <html>
      <head>
        <title>mail.example.com</title>
      </head>
      <body>
      <h1>mail.example.com</h1>
      <h2>This Month</h2>
        <ul>
          <li><a href="awstats/awstats.mail.example.com.html">mail.example.com www stats</a></li>
          <li><a href="awstats/awstats.mail.html">mail.example.com mail stats</a></li>
      </ul>
      <h2>This Year</h2>
      <ul>
          <li><a href="awstats-year/awstats.mail.example.com.html">mail.example.com www stats</a></li>
          <li><a href="awstats-year/awstats.mail.html">mail.example.com mail stats</a></li>
      </ul>
      </body>
      </html>
    

    AWStats Configuration

  5. In /etc/awstats

    1. For web server log stats
      1. cp awstats.localhost.localdomain.conf awstats.mail.example.com.conf (assuming your primary hostname is mail.example.com)
      2. Set Logfile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/lighttpd/mail.example.com/access.log* |"
      3. Set LogFormat = "%host %virtualname %logname %time1 %methodurl %code %bytesd %referrerquot %uaquot"
      4. Set SiteDomain="mail.example.com"
      5. Set HostAliases="mail.example.com" (note you should include any aliases for this host as a space separate listed or look in the docs for how to use regular expressions here)
      6. Set DNSLookup=1
      7. Comment out DirIcons=/awstatsicons
      8. Set TrapInfosForHTTPErrorCodes = "400 403 404 407"
      9. Set ShowHTTPErrorsPageDetail=RH

Cron Configuration

  1. Replace /etc/cron.hourly/awstats with

      /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=mail.example.com -update -configdir="/etc/awstats" -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -dir=/var/www/vhosts/mail.example.com/awstats" >/dev/null
    
  2. And for full stats for the entire year, create /etc/cron.daily/awstats with

      /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=mail -update -month=all -configdir="/etc/awstats" -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -dir=/var/www/vhosts/mail.example.com/awstats-year" >/dev/null
      /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=oldwww.example.com -update -month=all -configdir="/etc/awstats" -awstatsprog="/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -dir=/var/www/vhosts/mail.example.com/awstats-year" >/dev/null
    
    1. Test updates by running /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=mail.example.com -update

Attack Detection/Blocking

  1. Install “Attack Detection/Blocking” packages listed above
  2. Create /etc/fail2bain/jail.local like the following:

      [sshd]
      port = ssh,<your-alternate-ssh-port-if-applicable>
      filter = sshd-aggressive
      enabled = true
    
      [selinux-ssh]
      port = ssh,<your-alternate-ssh-port-if-applicable>
      enabled = true
    
      [lighttpd-auth]
      enabled = true
    
      [recidive]
      enabled = true
    
  3. Run touch /var/log/fail2ban.log

  4. And finally execute systemctl enable fail2ban && systemctl start fail2ban