DenyHosts setup on Red Hat/Centos/Amazon AMIs

This is a really useful tutorial for setting up DenyHosts to protect against brute force ssh attacks.

http://www.cyberciti.biz/faq/rhel-linux-block-ssh-dictionary-brute-force-attacks/

DenyHosts homepage:

http://denyhosts.sourceforge.net/

Make script compatible with chkconfig

To make an init script compatible with chkconfig (to autostart on next runlevel change), add the following header to your script within it's /etc/init.d/ file, for example, /etc/init.d/myscript

#!/bin/bash

#
# myscript     init script to manage myserver
#
# chkconfig: 2345 90 35
# description: A server which does amazing things.
#


Now, add it with these commands:

/bin/chmod +x  /etc/init.d/myscript
/sbin/chkconfig myscript --add
/sbin/chkconfig myscript on
/sbin/chkconfig myscript --list
myserver       0:off    1:off    2:on    3:on    4:on    5:on    6:off


Within the chkconfig definition line, 2345 means the runlevels it will be started under, and 90 35 are the start and kill priorities, an integer between 0 and 99.  In this example, it would be started as one of the last services, and killed (stopped) as one of the earlier processes.

Install capistrano on RHEL or CentOS

Capistrano is great for automating system tasks. Here is how to install it on RPM-based systems:

# yum install ruby rubygems
# gem install mocha echoe rake capistrano

Please note that mocha, echoe are optional for tests, but I included them in these instructions.

Sample Linux interview questions

I have compiled some sample interview questions for use in testing a potential systems administrator or systems engineer. I have created most of them but have reused some of them from the recent interviews that I have had. It should be one way to separate candidates as well as prepare others for interviews.

Sample questions:

What is a way to find the current running kernel version level?

Various iterations of the uname command (uname -a or uname -r)

How do you update the system on Red Hat 4? Red Hat 5?

up2date -u (RHEL 4) and yum update (RHEL 5)

What is a way to see what service pack and version the Red Hat system is at? CentOS?

cat /etc/redhat-release (Same for both RHEL and CentOS)

What is a way to change a kernel parameter?

sysctl -w parameter=value (persistent)

or

echo 32768 > /proc/sys/fs/file-max (not reboot persistent)

or

change a kernel parameter in /boot/grub/menu.lst as such:

kernel /boot/vmlinuz-2.6.18-128.1.16.el5 ro root=LABEL=/ elevator=deadline (persistent)

or

change the parameter in /etc/sysctl.conf (persistent)

You have a 32 bit system but want to allow RHEL to be able to use more than 4GB of RAM. What kernel do you use to accomplish this task?

Install and boot into the PAE kernel.

Of these filesystems--XFS, EXT3, EXT4, reiserFS, what is the best for large files?

XFS

Of these filesystems--XFS, EXT3, EXT4, reiserFS, what is the best for small files?

reiserFS

Of these filesystems--XFS, EXT3, EXT4, reiserFS, which ones are supported as of RHEL 5.3?

EXT3 and EXT4 (technology preview)

What has Microsoft contributed to the Linux kernel (trivia)?

A kernel module which enabled better performance with its Hyper-V virtualization technology.

You have tried to install an RPM but it has failed because of broken dependencies. How do you override and force the installation anyway?

rpm -i myprogram.rpm --nodeps

How do you set the maximum interval between fsck checks on /dev/sda1 to be one week?

tune2fs -i 1w /dev/sda1

Define dom0 and domU.

dom0 (domain zero) is the server running the Xen, KVM, or QEMU hypervisor. domU (domain unprivileged) is a virtual machine within a Xen, KVM, or QEMU server.

By default, what is the first disk known as within a VMWare virtual machine?

/dev/sda

By default, what is the first disk known as within a Xen domU?

/dev/xvda

Hope this is helpful.

Comb through Red Hat hair after initial install

Yes, you can also do this with a kickstart file, but if you want to clean up some unnecessary pieces of a Red Hat install, use the command:

yum remove blue* autofs at* anacron* cups* hid* gpm firstboot* iptables isdn* lvm* md* nfs* oddjob pcsc* portmap rpc* sendmail ypbind winbind* wpa* nscd* samba* smb*

Of course, analyze what you are using the server for and whether you will need any of these packages. But, for a vanilla install, for me this command seems to be useful to clean up some unnecessary packages.

Determine BIOS settings within Linux

Use dmidecode to determine hardware information fro the system EFI or BIOS.

Process affinity

Here is how to bind a program to a CPU (process affinity):

Bind processes to a CPU core

An example would be:

taskset -pc 0,1,2 20509

to set processid #20509 to have affinity (bind) on Cpu0, Cpu1, and Cpu2. To bind it to a single core, use the command:

taskset -pc 0 20509

to bind it to Cpu0.

A VSFTP server configuration with virtual users

Here is how to implement a “Very Secure” FTP server with virtual users. This has maximum security as we are implementing virtual FTP users instead of system users.

1. Edit the file /etc/vsftpd/logins.txt and add usernames and passwords (one line each, no spaces) like this:

bob
bobpassword
tom
tompassword

2. Load it into a database file (using Berkley's DB4). Install it with "yum install db4" or "sudo apt-get install db4"

db_load -T -t hash -f /etc/vsftpd/logins.txt /etc/vsftpd/vsftpd_login.db

3. Tell pam to use this database file for logins. Comment out anything in /etc/pam.d/vsftpd and add the lines:

auth required /lib/security/pam_userdb.so db=/etc/vsftpd/vsftpd_login
account required /lib/security/pam_userdb.so db=/etc/vsftpd/vsftpd_login

4. Now, take care of some permissions. Virtual ftp users will be mapped to the system user virtualuser
mkdir /mnt/dev
useradd -d /mnt/dev/ virtualuser
chown virtualuser.virtualuser /mnt/dev
chmod 600 /etc/vsftpd/vsftp_login.db
mv /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.orig

5. Now, add the following to /etc/vsftpd/vsftpd.conf For more information and directives, see http://vsftpd.beasts.org/vsftpd_conf.html

#Don't run through xinetd, run standalone
listen=YES
#Best to put it on a seperate partition as /
local_root=/mnt/dev
# No anonymous login or writes
anonymous_enable=NO
#Restrict users to local_root (/mnt/dev)
chroot_local_user=YES

#Let local users login, essential for allowing the system user virtual user to login. The caveat is that other system users other than virtualuser can login. You can further lock this down with putting virtualuser as the only user within /etc/vsftpd/allowed_users As other virtual ftp users defined within /etc/vsftpd/vsftp_login.db are mapping to the system account virtualuser, this is a good method to lock down other local users.
#userlist_deny=NO
#userlist_enable=YES
#userlist_file=/etc/vsftpd/allowed_users
local_enable=YES

#Enable for compatibility
connect_from_port_20=YES
#Default is 21, define something else if running non-standard. Remember to configure iptables to allow incoming/outgoing access to port 21.
listen_port=21

# Write permissions
write_enable=YES
#Important as we are using virtual users
check_shell=NO
#Make sure that /etc/pam.d/vsftpd is present and correct from the previous steps
pam_service_name=vsftpd

#Virtual user setup is also defined at: ftp://vsftpd.beasts.org/users/cevans/untar/vsftpd-2.0.5/EXAMPLE/VIRTUAL_USERS/README
#Important as this is how to enable many ftp users to use the one guest, system user "virtualuser"
#This enhances security because if these accounts are compromised, only ftp is compromised, not a privileged system user.
guest_enable=YES
#System username defined earlier
guest_username=virtualuser
#Allows virtualuser to have more than anonymous access
virtual_use_local_privs=YES
#Everything appears as the user "ftp," disable if you want individual users to be shown as owners within their ftp client.
hide_ids=yes

# Connection limit for each IP, good security
max_per_ip=2
# Maximum number of clients, increase if you are expecting more.
max_clients=200
#Shows which files are uploaded to the server to xferlog_file
xferlog_enable=YES
#Defines where the file should reside
xferlog_file=/var/log/vsftpd.log
#What users will see when they login
ftpd_banner=This is a secure blah FTP server
#Logs commands are being ran on the server (uploads, deletes, etc.) to xferlog_file
log_ftp_protocol=YES
#Added security of tcp_wrappers
tcp_wrappers=YES

6. Restart vsftpd
service vsftpd restart
or
/etc/init.d/vsftpd restart

A quality PostgreSQL startup script

I have went through several iterations of PostgreSQL startup scripts. Most are less than useful. This one, modified for my use (changed some of the directory structure, and tailored for Postgres 8.3.6) is originally provided by the Postgres YUM repository, and is actually useful. This is in stark comparison to the quasi-useful one that is included in the source package in file postgresql-version/contrib/start-scripts/linux This is designed for Red Hat iterations of Linux, but may with small modifications, work with Debian, Ubuntu, and SuSE. Enjoy.


#!/bin/sh
# postgresql This is the init script for starting up the PostgreSQL
# server
#
# chkconfig: - 64 36
# description: Starts and stops the PostgreSQL backend daemon that handles \
# all database requests.
# processname: postmaster
# pidfile: /var/run/postmaster.pid
PGVERSION=8.3.6
# PGMAJORVERSION is major version, e.g., 8.0 (this should match PG_VERSION)
PGMAJORVERSION=`echo "$PGVERSION" | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/'`

# Source function library.
INITD=/etc/rc.d/init.d
. $INITD/functions

# Get function listing for cross-distribution logic.
TYPESET=`typeset -f|grep "declare"`

# Get config.
. /etc/sysconfig/network

# Find the name of the script
NAME=`basename $0`
if [ ${NAME:0:1} = "S" -o ${NAME:0:1} = "K" ]
then
NAME=${NAME:3}
fi

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]
then
SU=runuser
else
SU=su
fi


# Set defaults for configuration variables
PGENGINE=/app/pgsql/bin
PGPORT=5432
PGDATA=/app/pgsql/data

if [ -f "$PGDATA/PG_VERSION" ] && [ -d "$PGDATA/base/template1" ]
then
echo "Using old-style directory structure"
else
PGDATA=/app/pgsql/data
fi
PGLOG=/app/pgsql/data/pgstartup.log

# Override defaults from /etc/sysconfig/pgsql if file is present
[ -f /etc/sysconfig/pgsql/${NAME} ] && . /etc/sysconfig/pgsql/${NAME}

export PGDATA
export PGPORT

# Check that networking is up.
# Pretty much need it for postmaster.
[ "${NETWORKING}" = "no" ] && exit 0

[ -f "$PGENGINE/postmaster" ] || exit 1

script_result=0

start(){
PSQL_START=$"Starting ${NAME} service: "

# Make sure startup-time log file is valid
if [ ! -e "$PGLOG" -a ! -h "$PGLOG" ]
then
touch "$PGLOG" || exit 1
chown postgres:postgres "$PGLOG"
chmod go-rwx "$PGLOG"
[ -x /usr/bin/chcon ] && /usr/bin/chcon -u system_u -r object_r -t postgresql_log_t "$PGLOG" 2>/dev/null
fi

# Check for the PGDATA structure
if [ -f "$PGDATA/PG_VERSION" ] && [ -d "$PGDATA/base" ]
then
# Check version of existing PGDATA

if [ x`cat "$PGDATA/PG_VERSION"` != x"$PGMAJORVERSION" ]
then
SYSDOCDIR="(Your System's documentation directory)"
if [ -d "/usr/doc/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/doc
fi
if [ -d "/usr/share/doc/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/share/doc
fi
if [ -d "/usr/doc/packages/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/doc/packages
fi
if [ -d "/usr/share/doc/packages/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/share/doc/packages
fi
echo
echo $"An old version of the database format was found."
echo $"You need to upgrade the data format before using PostgreSQL."
echo $"See $SYSDOCDIR/postgresql-$PGVERSION/README.rpm-dist for more information."
exit 1
fi

# No existing PGDATA! Warn the user to initdb it.
else
echo
echo "$PGDATA is missing. Use \"service postgresql initdb\" to initialize the cluster first."
echo_failure
echo
exit 1
fi

echo -n "$PSQL_START"
$SU -l postgres -c "$PGENGINE/postmaster -p '$PGPORT' -D '$PGDATA' ${PGOPTS} &" >> "$PGLOG" 2>&1 < /dev/null
sleep 2
pid=`pidof -s "$PGENGINE/postmaster"`
if [ $pid ] && [ -f "$PGDATA/postmaster.pid" ]
then
success "$PSQL_START"
touch /var/lock/subsys/${NAME}
head -n 1 "$PGDATA/postmaster.pid" > "/var/run/postmaster.${PGPORT}.pid"
echo
else
failure "$PSQL_START"
echo
script_result=1
fi
}

stop(){
echo -n $"Stopping ${NAME} service: "
$SU -l postgres -c "$PGENGINE/pg_ctl stop -D '$PGDATA' -s -m fast" > /dev/null 2>&1 < /dev/null
ret=$?
if [ $ret -eq 0 ]
then
echo_success
else
echo_failure
script_result=1
fi
echo
rm -f "/var/run/postmaster.${PGPORT}.pid"
rm -f "/var/lock/subsys/${NAME}"
}

restart(){
stop
start
}

initdb(){
if [ -f "$PGDATA/PG_VERSION" ]
then
echo "Data directory is not empty!"
echo_failure
else
echo -n $"Initializing database: "
if [ ! -e "$PGDATA" -a ! -h "$PGDATA" ]
then
mkdir -p "$PGDATA" || exit 1
chown postgres:postgres "$PGDATA"
chmod go-rwx "$PGDATA"
fi
# Clean up SELinux tagging for PGDATA
[ -x /sbin/restorecon ] && /sbin/restorecon "$PGDATA"
# Initialize the database
$SU -l postgres -c "$PGENGINE/initdb --pgdata='$PGDATA' --auth='ident sameuser'" >> "$PGLOG" 2>&1 < /dev/null
# Create directory for postmaster log
mkdir "$PGDATA/pg_log"
chown postgres:postgres "$PGDATA/pg_log"
chmod go-rwx "$PGDATA/pg_log"

[ -f "$PGDATA/PG_VERSION" ] && echo_success
[ ! -f "$PGDATA/PG_VERSION" ] && echo_failure
echo
fi
}
condrestart(){
[ -e /var/lock/subsys/${NAME} ] && restart
}

condstop(){
[ -e /var/lock/subsys/${NAME} ] && stop
}

reload(){
$SU -l postgres -c "$PGENGINE/pg_ctl reload -D '$PGDATA' -s" > /dev/null 2>&1 < /dev/null
}

# This script is slightly unusual in that the name of the daemon (postmaster)
# is not the same as the name of the subsystem (postgresql)

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p /var/run/postmaster.${PGPORT}.pid
script_result=$?
;;
restart)
restart
;;
initdb)
initdb
;;
condrestart)
condrestart
;;
condstop)
condstop
;;
reload|force-reload)
reload
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|condstop|reload|force-reload|initdb}"
exit 1
esac

exit $script_result

Quick and dirty openLDAP replication

This is a cursory view of how to install and configure a master and slave openLDAP server pair. Unless specified, follow each step on both the master and the slave servers. The only real difference between the servers is the presence of a slurpd configuration on the master and the unique slapd configuration on both servers. In the end, you will have a syncing pair which will replicate changes from the master server to the slave server every three seconds.

A. Install OS and LDAP

1.Install your OS. I am assuming Linux, specifically RHEL or Fedora Core, but openLDAP will run on a variety of systems and these instructions can be adapted to your specific flavor.

2.On both the master and the slave, install openldap, php, httpd and the dependencies with the command (assuming RHEL or Fedora Core):
# yum -t -y install openldap-clients openldap-servers openldap php-ldap nss_ldap httpd php

B. Install and configure phpLDAPadmin (optional)

3.Install phpLDAPadmin from this website: http://phpldapadmin.sourceforge.net/wiki/index.php/Main_Page

4.Untar the download, and then copy the file phpldapadmin-<version>/config/config.php.example to phpldapadmin-/config/config.php

5.Move the phpldapadmin-<version> to /var/www/html/phpldapadmin

6.Restart httpd with the command
#chkconfig httpd on
#service httpd restart

C. Configure and test LDAP

7.Make sure that the master server's hostname is pingable from the slave and vice versa. If not, add the entries to /etc/hosts and restart networking with the command:
#service network restart

8.Copy /etc/openldap/ldap.conf to /etc/openldap/ldap.conf.orig

9.Copy /etc/openldap/sldapd.conf to /etc/openldap/slapd.conf.orig

10.Copy /etc/openldap/DB_CONFIG.example to /var/lib/ldap/DB_CONFIG

11.Copy the configuration files to the respective servers. These are located at the bottom of this document. Make sure to copy the correct ldap.conf and slapd.conf to their respective servers.

12.Import the base dn from the base.ldif file (included later in this document)
#slapadd < /etc/openldap/base.ldif

13.Start the LDAP service
# chkconfig ldap on
# service ldap start

14.Point to http://hostname/ or http://hostname/phpldapadmin If you get a “php memory too low” error, change the memory limit to something meaningful in /etc/php.ini
memory_limit = 128M ; For example

15.The login for the server is cn=root,dc=example,dc=com and the password needs to be set with slappasswd. Use slappasswd and change the hash in the /etc/openldap/slapd.conf file.
# slappasswd
New password:
Re-enter new password:
{SSHA}At/pOvtko2KXcKfM7t0o/OPedJrpXQM0

Now enter the line with the hashed password in the file /etc/openldap/slapd.conf as shown:
rootpw {SSHA}At/pOvtko2KXcKfM7t0o/OPedJrpXQM0

16.From the phpLDAPadmin GUI or using #slapadd similar to before, import the ldif file from a backup or existing server to the master server. If you have not created or do not have a backup of the ldif file of the old directory server, the other option is to copy the /var/lib/ldap directory over to the new server. If starting from scratch, this is a mute point.

17.If syncing is working, it will be replicated on the slave server. If not, the file /var/lib/ldap/replica/openldap-master-replog on the master server will tell you why.

18.For testing the syncing and replication of the master and slave servers, add a new entry to the master server and see if the entry appears on the slave server. For testing the subordination of the slave server, create an entry on the slave server and watch as it is not replicated on the master server.

/etc/openldap/base.ldif (For both servers)
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
dc: example
o: Example

/etc/openldap/ldap.conf (For master server)
URI ldap://127.0.0.1/
BASE dc=example,dc=com
TLS_CACERTDIR /etc/openldap/cacerts

/etc/openldap/slapd.conf (For master server)
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
#Note that this should be changed based upon the hostname or user for greater security
access to *
by * read
by anonymous auth
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn. (e.g., "access to * by * read")
# rootdn can always read and write EVERYTHING!
database bdb
suffix "dc=example,dc=com"
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}/mYjTZhwSR1hIGKt6qD0oBpHdRjeHSGh
directory /var/lib/ldap
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
# Replicas of this database
replogfile /var/lib/ldap/openldap-master-replog
replica host="slave:389"
suffix="dc=example,dc=com"
binddn="cn=root,dc=example,dc=com"
credentials=changeme
bindmethod=simple

/etc/openldap/ldap.conf (For slave server)
URI ldap://127.0.0.1
BASE dc=example,dc=com
TLS_CACERTDIR /etc/openldap/cacerts
updatedn "cn=root,dc=example,dc=com"
updateref ldap://master
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}nlb8tJHDUJCMqQMBMyMIIu26VF1ViVEu

/etc/openldap/slapd.conf (For slave server)
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
# Note that you should change this based upon the hostname of the master server.
access to *
by * write
by anonymous auth
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn. (e.g., "access to * by * read")
# rootdn can always read and write EVERYTHING!
database bdb
suffix "dc=example,dc=com"
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}/mYjTZhwsR1hIGKt6qD0oBpHdRjeHSGh
updatedn "cn=root,dc=example,dc=com"
updateref ldap://master
directory /var/lib/ldap
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
TLSCertificateFile /etc/openldap/ldap.cert
TLSCertificateKeyFile /etc/openldap/ldap.key

Install and configure NTP

NTP is great at keeping your Linux server or desktop's time synced. Not having the time synced can potentially cause issues with backup software, applications and some web applications. Here is a sample script to install and configure NTP on Linux. This was created for Red Hat, but it should work with other versions of Linux with few modifications (like the installation of the init scripts).

#NTP configuration script.

date
cat /var/lib/ntp/drift
chkconfig ntpd --list
service ntpd stop
ntpdate -u 0.rhel.pool.ntp.org
ntpdate -u 1.rhel.pool.ntp.org
ntpdate -u 2.rhel.pool.ntp.org
chkconfig ntpd on
cat /etc/ntp.conf | grep server
vi /etc/ntp.conf

#Based upon the output of those commands, add (or delete) the following lines in /etc/ntp.conf
server 0.rhel.pool.ntp.org
server 1.rhel.pool.ntp.org
server 2.rhel.pool.ntp.org

#Now save and test
service ntpd start
sleep 4
ntpq -p
cat /var/lib/ntp/drift
date

Coraid documentation

I was able to do quite a bit of technical writing with Coraid. It is useful for those of you using AoE technology. I wrote all of the quickstart documentation here:

http://www.coraid.com/RESOURCES/Quickstart-Documentation

Also, I wrote the driver installation guides here:

http://www.coraid.com/SUPPORT/AoE-Drivers

Execute any custom command on bootup (SuSE or Red Hat)

This command is for anything that you would like to execute after the system has been started and all of the scripts in /etc/init.d/ have been run. If you want to execute these commands before any of these scripts, but the commands in the new file:

/etc/init.d/before.local

If you want them to execute afterwards, put it in

/etc/init.d/after.local

This is rather useful for network mounts, changes to hardware configuration, or custom programs that need to be started at boot. Make sure to make the file executable through the command: chmod +x /etc/init.d/after.local

The other option is to edit the /etc/rc.d/rc.local and do that same there.

Why Webmin is the best tool for Systems Administrators

In the SysAdmin world, Webmin is the best invention next to sliced bread. It is a compete, web-based administration interface for dozens of different systems. Not only does it support a plethora of operating systems, but it is much better than Red Hat's system-config-* and almost as good as Novell's Yast. It knocks the socks off any other administration tool that I have seen. You can download the package for your distro at Webmin's Website I guess you can call me a Webmin fanboy.