Very useful bash $PS1 setting

My ~/.bashrc settings have a killer setting for how my shell prompt looks:

case $- in
   *i*)
export PS1="\[$(tput bold)\]\[$(tput setaf 6)\]\t \[$(tput setaf 2)\][\[$(tput setaf 4)\]\u\[$(tput setaf 1)\]@\[$(tput setaf 4)\]\H \w \[$(tput setaf 6)\]\[$(tput setaf 2)\]]\[$(tput setaf 4)\] \$? \\$ \[$(tput sgr0)\]" ;;

     *)  return ;;
esac


The result:

22:19:43 [myuser@myserver.example.com ~ ] 0 $

This shows the current system time, username@server FQDN, pwd, exit code of last command, and regular user or root prompt.

This works in both interactive and non-interactive shells (hence the case statement).  Enjoy.  Additional info can be found here:

Bash $PS1 generator
10 Bash $PS1 examples
Bashrc generator

Ruby Regex

Just wanted to pass along two very useful sites for regular expressions with Ruby

Ruby regex calculator:

http://rubular.com/

Ruby regex documentation:

http://www.tutorialspoint.com/ruby/ruby_regular_expressions.htm

MOTD on Amazon Linux

Just had to pass this along, very useful.

https://coderwall.com/p/hr_9pw

Google search plugin for terminator

This is going to save me 10 minutes a day.  Props to the developer.

https://github.com/choffee/terminator-plugins

This is a plugin which adds a function to the context menu of terminator (my favorite Linux-based terminal emulator) which will search Google for the selected text.  Before I discovered this, I would select the text, copy it to a browser like Chrome or Firefox, then searched Google.  This is going to make me much more efficient.

Capistrano with sudo

Capistrano with sudo

An easy way to execute commands with sudo from within a capistrano job is to add the prefix  #{sudo} to a command within your capistrano task.  For example:

task :make_protected_dir, :roles =>db do
  run "#{sudo} mkdir /root/mydir"
end

Another example:

task :restart_httpd, :roles =>app do
  run "#{sudo} /etc/init.d/httpd restart"
end

Then from the command line, load your file with these entries:

cap -f capfile

Then, execute the commands:

cap make_protected_dir

cap restart_httpd

More info:

https://github.com/capistrano/capistrano/wiki/2.x-DSL-Configuration-Tasks-Task

S3 Utility

Here is a great tool for working with Amazon S3.  It even has a wide variety of repos available (easier than compiling from source).  s3cmd is very easy to setup.  It is the easiest to work with of all the S3 tools I have used.

http://s3tools.org/s3cmd

Trace your python programs

Bash programming has a very helpful command of:

bash -x myscript.sh

This will show you what bash is doing while running through myscript.sh. Python has the same functionality, through this command:

python -m trace --trace myscript.py 

Very useful in debugging python programs.  Additional info:

http://docs.python.org/library/trace.html

Create ssh public key from private key

If you have a private key and want to create the public key from that (to put in ~/.ssh/authorized_keys, for example), do the following:

ssh-keygen -f <my_private_key> -y > <my_public_key>

For example:

ssh-keygen -f ~/.ssh/id_rsa -y > ~/.ssh/id_rsa.pub


If you are using this to ssh with keys, add my_public_key (eg. ~/.ssh/id_rsa.pub) to ~/.ssh/authorized_keys on the remote server.

Then ssh with the private key to the remote server to test.

EC2 Bundling an AMI -- Server.InternalError: An internal error has occurred

Once you try to re-bundle an OS image that has been previously bundled, it will complain about /etc/ec2/amitools/cert-ec2.pem not being present.  I figured that was my certificate, but it is  EC2's X.509 public key certificate, not mine (docs are somewhat confusing about that).  Since they were stripped out of the AMI last time it was bundled, I reinstalled the tools to restore these certs and no errors any more.

AWS API tools:

http://aws.amazon.com/developertools/368

Hope this helps.

Original EC2 forum posts:

https://forums.aws.amazon.com/thread.jspa?messageID=345021

https://forums.aws.amazon.com/message.jspa?messageID=344101

Ubuntu and EC2

Great blog on the nitty gritty of working with EC2:

http://alestic.com/

Install Cisco AnyConnect on Ubuntu/Debian/Mint 64 bit

Looks like there are no 64 bit Cisco AnyConect VPN Clients.  Here is a workaround to get them working on a Debian-based 64 bit OS:

http://www.virtualcorner.nl/?page_id=220

Linux Mint on Gigabyte GA-Z68X-UD3H-B3

I tried to install Linux Mint on a desktop with Gigabyte GA-Z68X-UD3H-B3 motherboard with its Intel raid array configured with two of three hard drives.  I wanted to install Linux Mint 12 x64 on the third drive.  When trying to boot, I got the error message:

udev timeout killing /sbin/modprobe -bv pci:

I had to add the grub option of nomodeset as a kernel paramater.  It then was able to install from there.  Here are more details:

http://forums.linuxmint.com/viewtopic.php?f=46&t=95351&p=547012#p545731

DenyHosts setup on Red Hat/Centos/Amazon AMIs

This is a really useful tutorial for setting up DenyHosts to protect against brute force ssh attacks.

http://www.cyberciti.biz/faq/rhel-linux-block-ssh-dictionary-brute-force-attacks/

DenyHosts homepage:

http://denyhosts.sourceforge.net/

pv

This was so good I had to repost it here.  It is about a useful utility called pv which shows a progress report of running processes:

http://www.dp.cx/blog/pv--one-of-the-most-useful-applications-around.html

Make script compatible with chkconfig

To make an init script compatible with chkconfig (to autostart on next runlevel change), add the following header to your script within it's /etc/init.d/ file, for example, /etc/init.d/myscript

#!/bin/bash

#
# myscript     init script to manage myserver
#
# chkconfig: 2345 90 35
# description: A server which does amazing things.
#


Now, add it with these commands:

/bin/chmod +x  /etc/init.d/myscript
/sbin/chkconfig myscript --add
/sbin/chkconfig myscript on
/sbin/chkconfig myscript --list
myserver       0:off    1:off    2:on    3:on    4:on    5:on    6:off


Within the chkconfig definition line, 2345 means the runlevels it will be started under, and 90 35 are the start and kill priorities, an integer between 0 and 99.  In this example, it would be started as one of the last services, and killed (stopped) as one of the earlier processes.

Just kill the process

/bin/kill -9 `/bin/cat /var/run/snmpd.pid`

To kill just the PID reported by the process as the active PID (with a shotgun, -9) and nothing else, use this shortcut.

sed add line at end of file

If you want to add a a line of text to the end of a file, and want to do the edit in place, use this syntax:

sed -i '$ a\This is some sample text.'  /foo/bar

Visual explanation of SQL joins

I wish I had these pictures in my head when I started learning about joins in SQL.  Might have been a DBA instead of a systems engineer.


http://www.codinghorror.com/blog/2007/10/a-visual-explanation-of-sql-joins.html

Charge iPad or iPad2 over usb in Linux

First, follow these instructions:

http://korenkov.info/ipad-ipad2-charge-ubuntu

For later versions of udev (I tested this on Ubuntu 11.04), make sure that /etc/udev/rules.d/95-ipad_charge.rules looks like the following examples.

For iPad:

ENV{DEVTYPE}=="usb_device", ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="129a", RUN+="/usr/bin/ipad_charge"

For iPad2:

ENV{DEVTYPE}=="usb_device", ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="05ac", ATTRS{idProduct}=="129f", RUN+="/usr/bin/ipad_charge"

Then restart udev or reboot.

Lexmark z645 install redux

This is the best way to install the printer driver for the Lexmark 600 or 640 series drivers on Ubuntu:

https://help.ubuntu.com/community/HardwareSupportComponentsPrinters/LexmarkPrinters

sed with variables

Here is an example of how to get sed to replace values within files with variables:

for a in {1..12} ; do sed -e s/0/"$a"/g myConfig > myConfig$a ; done;


This is useful if you need to have an iterated set of config files, for instance.

In place editing of files quickly

Use the sed -i option for in place editing of files:

sed -i 's/foo/bar/g' FILENAME

It will replace all occurances of foo with bar within FILENAME

Convert videos to iPad 2 format

Recorded programs on my MythTV are in a lossless HD mpg format, which doesn't play well on my iPad2 using AirVideo Server over wlan (via wine). So, instead of having AirVideo Server converting it through the wine abstraction layer (which requires more CPU/memory, and uses ffmpeg.exe anyway), I just run this user job from my MythTV server to convert the files that I care to watch on my iPad.

Step 1:
Create this file which I call /usr/bin/mythconverttoipad. If you have an issue with "too may threads," remove -threads 16. I am using this script on a 3 core machine, the -threads option speeds things up a bit.

#!/bin/bash
DIR=$1
FILE=$2

echo $DIR
echo $FILE

ffmpeg -y -i $DIR/$FILE -ab 320kb -vcodec libx264 -b 3200kb -s 1280x720 -threads 16 $DIR/$FILE.m4v


Step 2:
Make the script executable.

chmod +x /usr/bin/mythconverttoipad

Step 3:
Add script as user job within MythTV.

Within mythtv-setup add the following user job:

/usr/bin/mythconverttoipad %DIR% %FILE%

Step 4:
Enable it within rules.

Within a program recording rule (within mythfrontent), enable this user job to be run post-recording.

Step 5:
Access the files from Air Video client or other player.

If you have questions, please comment.

Control Roku player over telnet

My young boys have broken several remotes for my Roku player. I am tired of buying new ones. There is a way to control the Roku player over the network, though. Use telnet on port 8080, for example:

[root@mythtv3 ~]# telnet 192.168.1.8 8080
Trying 192.168.1.8...
Connected to 192.168.1.8.
Escape character is '^]'.
H0A0AC134945
ETHMAC 00:0d:4c:43:90:11
WIFIMAC 00:0d:4c:43:90:11
>press right
>press down
>press select

The available commands are:
press up
press down
press left
press right
press select
press home
press fwd
press back
press pause


Additional information:

http://forums.roku.com/viewtopic.php?t=20106

Install capistrano on RHEL or CentOS

Capistrano is great for automating system tasks. Here is how to install it on RPM-based systems:

# yum install ruby rubygems
# gem install mocha echoe rake capistrano

Please note that mocha, echoe are optional for tests, but I included them in these instructions.

TCP dump for only port 80

Here is a way to dump all tcp traffic on port 80 to and from the bond0 interface:

tcpdump -w tcpdumpPort80.pcap -i bond0 tcp port 80

If you wanted to use only the eth0 interface (more common), use this example:

tcpdump -w tcpdumpPort80.pcap -i eth0 tcp port 80

Sending files from the Linux command line

Here is a quick way to e-mail yourself files from a server using mutt.

$ mutt -a tcpdumpApril182011.pcap my_name@example.com < /dev/null

Mutt is great for sending MIME encoded files.

Quickly clear out a file's contents

If you ever have the need to quickly clear out the contents of a file while preserving its priviledges and creation date, use this command:

echo " " > myConfig.xml

or

echo " " > /etc/my.cnf

Grep with filenames

Sometimes you want grep to tell you what files have a pattern within a directory.  Here is how to do it:


grep -H JAVA_HOME=/  /usr/local/terracotta/bin/* /usr/local/terracotta/platform/bin/*

/usr/local/terracotta/bin/start-tc-server.sh:export JAVA_HOME=/usr/java/jdk1.6.0_21/
/usr/local/bin/stop-tc-server.sh:export JAVA_HOME=/usr/java/jdk1.6.0_21/
/usr/local/bin/tim-get.sh:export JAVA_HOME=/usr/java/jdk1.6.0_21/
/usr/local/platform/bin/make-boot-jar.sh:export JAVA_HOME=/usr/java/jdk1.6.0_21/


Enjoy!

Bash for loops with a series of numbers or letters

Bash scripting is great. Here is a quick trick to funnel a series of numbers (or letters) into a variable, and thus an argument of a script:


#!/bin/bash
for a in {1..18}
do
echo "The number $a"
done


#!/bin/bash
for a in {a..z}
do
echo " The letter $a "
done

Here is another example:

for a in {0..15};  do  /usr/sbin/xm vcpu-pin 0 $a 0-1,4-15;  done


Such a quick and easy trick!

Bring Netflix streaming to Linux!

Considering there are a host of devices (PS3, Roxee, TiVo, LG Blu-Ray players) that run Linux internally and support Netflix Streaming, it should be an easy technical transition to bring this to Linux.  Sign the petition to bring Netflix streaming to Linux. I am promoting it so I can use Netflix Streaming on my MythTV server and enhance the MythTV experience with Netflix streaming.

http://www.petitiononline.com/Linflix/petition.html

Create huge files fast with dd

Here is a quick way to create very large empty files without writing every byte using dd.

# dd if=/dev/zero of=largeemptyfile.img bs=1M count=1 seek=16999
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.001772 seconds, 592 MB/s
# ls -lh
total 1.1M
-rw-r--r-- 1 root root 17G Jul 16 09:07 largeemptyfile.img
# du -h
1.1M    .

Notice that the actual file is only 1.1 MB, but the file shows as 17GB.  This is because dd basically wrote the first and last parts of the file, and left the middle alone.  You can now use this file for anything, such as a Xen disk image.  Once it is in use, it will report as the full 17GB with du.

Killing zombie processes

Kill those persistent and annoying zombie processes.


ps -e -o ppid,stat | grep Z | cut -d" " -f2 | xargs kill -9

Tested on Fedora for accuracy.  

New t-shirt shop

I have partnered with CafePress.com to create some suave t-shirts about tech.  See my shop at:

http://www.cafepress.com/Peachfuzztech

Enjoy!

Sample /etc/dhcpd.conf configuration

Here is a basic dhcpd configuration.
For more info, look at /usr/share/doc/dhcp*/dhcpd.conf.sample.

ddns-update-style ad-hoc;

subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.200;
option routers 192.168.1.1 ;
option domain-name "mydomain.com" ;
option domain-name-servers 4.2.2.2 4.2.2.3 ;
}

Make a service persistant on Ubuntu

The latest versions of Ubuntu (10.04 is where I noticed this behavior) ignore LSB configuration, which makes chkconfig not work for some scripts.  Instead, use update-rc.d-insserv to enable a service after reboot as shown.  This example enables the Samba services after a reboot.

update-rc.d-insserv nmbd defaults
update-rc.d-insserv smbd defaults

Find the PID number with nothing else

Use this combination to find a PID number without anything else in the output:

ps -ef | grep -v grep | grep ssh | cut -d" " -f3
3012

Enjoy that kungfu knife-kick combo.

Write on my Linux wall

To display a text message to other users logged into a *nix system, use the wall command. For example:

wall "Deploying new .ear file in five minutes."


This will send the message to all users logged into the system, whether they be physically on a console, via ssh or ftp.

Comment lines in vi

If you need to comment the next 10 lines within vi, execute the command within the command mode:

:.,+10s/^/#

Sample Linux interview questions

I have compiled some sample interview questions for use in testing a potential systems administrator or systems engineer. I have created most of them but have reused some of them from the recent interviews that I have had. It should be one way to separate candidates as well as prepare others for interviews.

Sample questions:

What is a way to find the current running kernel version level?

Various iterations of the uname command (uname -a or uname -r)

How do you update the system on Red Hat 4? Red Hat 5?

up2date -u (RHEL 4) and yum update (RHEL 5)

What is a way to see what service pack and version the Red Hat system is at? CentOS?

cat /etc/redhat-release (Same for both RHEL and CentOS)

What is a way to change a kernel parameter?

sysctl -w parameter=value (persistent)

or

echo 32768 > /proc/sys/fs/file-max (not reboot persistent)

or

change a kernel parameter in /boot/grub/menu.lst as such:

kernel /boot/vmlinuz-2.6.18-128.1.16.el5 ro root=LABEL=/ elevator=deadline (persistent)

or

change the parameter in /etc/sysctl.conf (persistent)

You have a 32 bit system but want to allow RHEL to be able to use more than 4GB of RAM. What kernel do you use to accomplish this task?

Install and boot into the PAE kernel.

Of these filesystems--XFS, EXT3, EXT4, reiserFS, what is the best for large files?

XFS

Of these filesystems--XFS, EXT3, EXT4, reiserFS, what is the best for small files?

reiserFS

Of these filesystems--XFS, EXT3, EXT4, reiserFS, which ones are supported as of RHEL 5.3?

EXT3 and EXT4 (technology preview)

What has Microsoft contributed to the Linux kernel (trivia)?

A kernel module which enabled better performance with its Hyper-V virtualization technology.

You have tried to install an RPM but it has failed because of broken dependencies. How do you override and force the installation anyway?

rpm -i myprogram.rpm --nodeps

How do you set the maximum interval between fsck checks on /dev/sda1 to be one week?

tune2fs -i 1w /dev/sda1

Define dom0 and domU.

dom0 (domain zero) is the server running the Xen, KVM, or QEMU hypervisor. domU (domain unprivileged) is a virtual machine within a Xen, KVM, or QEMU server.

By default, what is the first disk known as within a VMWare virtual machine?

/dev/sda

By default, what is the first disk known as within a Xen domU?

/dev/xvda

Hope this is helpful.

Comb through Red Hat hair after initial install

Yes, you can also do this with a kickstart file, but if you want to clean up some unnecessary pieces of a Red Hat install, use the command:

yum remove blue* autofs at* anacron* cups* hid* gpm firstboot* iptables isdn* lvm* md* nfs* oddjob pcsc* portmap rpc* sendmail ypbind winbind* wpa* nscd* samba* smb*

Of course, analyze what you are using the server for and whether you will need any of these packages. But, for a vanilla install, for me this command seems to be useful to clean up some unnecessary packages.

Show module information

Use the command modinfo to find detailed module information.

# modinfo ext3
filename: /lib/modules/2.6.18-164.6.1.el5/kernel/fs/ext3/ext3.ko
license: GPL
description: Second Extended Filesystem with journaling extensions
author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others
srcversion: 51D84081C475FE078B1D891
depends: jbd
vermagic: 2.6.18-164.6.1.el5 SMP mod_unload 686 REGPARM 4KSTACKS gcc-4.1

Determine BIOS settings within Linux

Use dmidecode to determine hardware information fro the system EFI or BIOS.

Process affinity

Here is how to bind a program to a CPU (process affinity):

Bind processes to a CPU core

An example would be:

taskset -pc 0,1,2 20509

to set processid #20509 to have affinity (bind) on Cpu0, Cpu1, and Cpu2. To bind it to a single core, use the command:

taskset -pc 0 20509

to bind it to Cpu0.

Clone a virtual domU with virt-clone

Here is a simple way to clone a virtual image created with KVM, Xen, QEMU or others.

#virt-clone -o oldDomU -n newDomU -f /var/lib/xen/images/newDomU.img

You can them use xm list to see the new virtual machine:

#xm list

Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1985 8 r----- 10566.8
oldDomU 1 999 1 -b---- 191.0
newDomU 2 999 1 -b---- 138.7

Manual zone transfers with dig

Transferring zones with named (bind) can be done manually with this command:

dig <master_dns_server> <zone> axfr

Example:

dig 10.1.1.6 example.com axfr

If this is run on a slave named server, /var/named/db.example.com will be updated as well if zone transfers are enabled on both servers.

Find and replace within vi

Yes, vi is cryptic. But this command is useful for performing find and replace functions within the vi editor. First, hit ESC (escape), and the colon key (:) to get to the ex shell. Then enter:

%s/searchString/replaceString/g

An example will be:

:%s/WindowsNT/RedHatLinux/g

chmod calculator

The best that I have found for quickly calculating numerical permissions:

http://www.javascriptkit.com/script/script2/chmodcal.shtml

Install Legato client on HP-UX 11

Download the archive onto the HP-UX server.

tar -xvf nw75sp1_hpux11_ia64.tar.gz
swinstall -s /hpux11_ia64/NetWorker.pkg

Follow the directions from the install screen.

Then, to start the program, run the startup script of /sbin/init.d/networker or /opt/networker/bin/nsrexecd

Install Legato client on Solaris

Download the package onto the Solaris server.

gunzip nw75sp1_solaris_64.tar.gz
tar -xvf nw75sp1_solaris_64.tar
pkgadd -d .

Select which package you would like to install (most likely LGTOclnt, the Legato client and LGTOman, the Legato manual).

Remove old files

If you don't use logrotate to remove or archive old logs, here is a way to remove old logs using the -ctime directive within the bash command find.

/usr/bin/find /var/log/tomcat -name *.tgz -ctime +15 | xargs rm -rf

A VSFTP server configuration with virtual users

Here is how to implement a “Very Secure” FTP server with virtual users. This has maximum security as we are implementing virtual FTP users instead of system users.

1. Edit the file /etc/vsftpd/logins.txt and add usernames and passwords (one line each, no spaces) like this:

bob
bobpassword
tom
tompassword

2. Load it into a database file (using Berkley's DB4). Install it with "yum install db4" or "sudo apt-get install db4"

db_load -T -t hash -f /etc/vsftpd/logins.txt /etc/vsftpd/vsftpd_login.db

3. Tell pam to use this database file for logins. Comment out anything in /etc/pam.d/vsftpd and add the lines:

auth required /lib/security/pam_userdb.so db=/etc/vsftpd/vsftpd_login
account required /lib/security/pam_userdb.so db=/etc/vsftpd/vsftpd_login

4. Now, take care of some permissions. Virtual ftp users will be mapped to the system user virtualuser
mkdir /mnt/dev
useradd -d /mnt/dev/ virtualuser
chown virtualuser.virtualuser /mnt/dev
chmod 600 /etc/vsftpd/vsftp_login.db
mv /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.orig

5. Now, add the following to /etc/vsftpd/vsftpd.conf For more information and directives, see http://vsftpd.beasts.org/vsftpd_conf.html

#Don't run through xinetd, run standalone
listen=YES
#Best to put it on a seperate partition as /
local_root=/mnt/dev
# No anonymous login or writes
anonymous_enable=NO
#Restrict users to local_root (/mnt/dev)
chroot_local_user=YES

#Let local users login, essential for allowing the system user virtual user to login. The caveat is that other system users other than virtualuser can login. You can further lock this down with putting virtualuser as the only user within /etc/vsftpd/allowed_users As other virtual ftp users defined within /etc/vsftpd/vsftp_login.db are mapping to the system account virtualuser, this is a good method to lock down other local users.
#userlist_deny=NO
#userlist_enable=YES
#userlist_file=/etc/vsftpd/allowed_users
local_enable=YES

#Enable for compatibility
connect_from_port_20=YES
#Default is 21, define something else if running non-standard. Remember to configure iptables to allow incoming/outgoing access to port 21.
listen_port=21

# Write permissions
write_enable=YES
#Important as we are using virtual users
check_shell=NO
#Make sure that /etc/pam.d/vsftpd is present and correct from the previous steps
pam_service_name=vsftpd

#Virtual user setup is also defined at: ftp://vsftpd.beasts.org/users/cevans/untar/vsftpd-2.0.5/EXAMPLE/VIRTUAL_USERS/README
#Important as this is how to enable many ftp users to use the one guest, system user "virtualuser"
#This enhances security because if these accounts are compromised, only ftp is compromised, not a privileged system user.
guest_enable=YES
#System username defined earlier
guest_username=virtualuser
#Allows virtualuser to have more than anonymous access
virtual_use_local_privs=YES
#Everything appears as the user "ftp," disable if you want individual users to be shown as owners within their ftp client.
hide_ids=yes

# Connection limit for each IP, good security
max_per_ip=2
# Maximum number of clients, increase if you are expecting more.
max_clients=200
#Shows which files are uploaded to the server to xferlog_file
xferlog_enable=YES
#Defines where the file should reside
xferlog_file=/var/log/vsftpd.log
#What users will see when they login
ftpd_banner=This is a secure blah FTP server
#Logs commands are being ran on the server (uploads, deletes, etc.) to xferlog_file
log_ftp_protocol=YES
#Added security of tcp_wrappers
tcp_wrappers=YES

6. Restart vsftpd
service vsftpd restart
or
/etc/init.d/vsftpd restart

Quicky find what directories are using up space on your disk

The df -h command will tell you disk usage from a mountpoint perspective, but the command du tells you from a directory perspective. Use the command:

du -h / --max-depth=1

To show disk usage for each individual directory on the system, or go lower down to see usage on a particular directory:

du -h /var/log --max-depth=1

vi find and replace

To perform a find and replace with all entries of a file, enter the colon to invoke ex from vi. Then, from there, enter

:%s/find_string/replace_string/g

For example:

%s/oldHostname/newHostname/g

There are many other ways to do this, but I prefer this method. This is one of the reasons that vi stands for "Voodoo for Intellectuals."

Vacuumdb within crontab

Vacuumdb for postgres is best run in a cron. But when your database user needs a password, export it and the cron will run without a problem. Here is an example of a vacuumdb instance that exports the password and then does a full, quiet, and analyzing vacuumdb on the mydatabase database.

Clean, vacuum and analyze the tripplanning database
0 2 * * * export PGPASSWORD=mypassword && vacuumdb -f -q -z -U postgres -d mydatabase >> /var/log/messages 2>&1

Take a network trace on HP UX

I needed to take a trace on an HP UX server the other day. These are the commands that I used:

nettl -tn 0x30800000 -e ns_ls_ip -tracemax 99999 -f /tmp/networkTrace

Then to kill it before it got to 99999 lines, I used the command

nettl -tf -e all

Then I analyzed /tmp/networkTrace with Wireshark.

zipidey-do daw, zipidey-de day, what a wonderful day!

The zip command will create a zip file that can be used across disparate platforms, including Linux/Unix to Windows or Mac. In other words, if you need to send your Windows friend several ziped files and he can only deal with .zip files, keep it simple. Don't use bzip or tar, just use zip. Here is a command to create a highly compressed zip file to contain some log files. Then just get the produced zipfile to your Windows "without walls" friend.

zip -9 logServices.zip /var/log/messages myapp/logs/mylog.log /usr/local/tomcat/logs/catalina.out

Add an temporary user account

If you need an account for a set period of time, or an account that you don't want to deactivate later, add the -e option on useradd:

useradd -m -e 2009-12-01 -c "Temp Account" tempUser

This user account will expire on December 1st, 2009 and will lock the user and their password.

Grep entire directories

To find a phrase that could be found somewhere in the current directory, use the command:

grep -r -i searchString ./

For example:

grep -r -i splunk /etc

This would search the entire /etc filesystem for any line with the word "splunk" located therein. Another way to do this would be the command:

find / -type f -print | xargs grep splunk

It works well with HP-UX and other Unixes.

Tar with date and a twist of chocolate

Here is a way to tar up a set of important directories with the date. You can use this as a log archive, or with Splunk. This uses the highest compression of gzip (-9).

tar cp /myapp/logs /var/log/messages /var/log/httpd/ | gzip -9c > /tmp/oldlogs`date +"%Y%m%d"`.tgz

Reboot your computer after 4082.97902312 years

For some cool reason, the maximum time that the Linux shutdown command will accept is 2,147,483,647 minutes, which is 4082.97902312 years. If you think your hardware is going to last that long, execute the command:

#shutdown -r -F 2147483647

In a little over 4,000 years, it will reboot and check your disks. Hopefully by then you will be doing something cooler than counting down, like golfing on Mars.

A quality PostgreSQL startup script

I have went through several iterations of PostgreSQL startup scripts. Most are less than useful. This one, modified for my use (changed some of the directory structure, and tailored for Postgres 8.3.6) is originally provided by the Postgres YUM repository, and is actually useful. This is in stark comparison to the quasi-useful one that is included in the source package in file postgresql-version/contrib/start-scripts/linux This is designed for Red Hat iterations of Linux, but may with small modifications, work with Debian, Ubuntu, and SuSE. Enjoy.


#!/bin/sh
# postgresql This is the init script for starting up the PostgreSQL
# server
#
# chkconfig: - 64 36
# description: Starts and stops the PostgreSQL backend daemon that handles \
# all database requests.
# processname: postmaster
# pidfile: /var/run/postmaster.pid
PGVERSION=8.3.6
# PGMAJORVERSION is major version, e.g., 8.0 (this should match PG_VERSION)
PGMAJORVERSION=`echo "$PGVERSION" | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/'`

# Source function library.
INITD=/etc/rc.d/init.d
. $INITD/functions

# Get function listing for cross-distribution logic.
TYPESET=`typeset -f|grep "declare"`

# Get config.
. /etc/sysconfig/network

# Find the name of the script
NAME=`basename $0`
if [ ${NAME:0:1} = "S" -o ${NAME:0:1} = "K" ]
then
NAME=${NAME:3}
fi

# For SELinux we need to use 'runuser' not 'su'
if [ -x /sbin/runuser ]
then
SU=runuser
else
SU=su
fi


# Set defaults for configuration variables
PGENGINE=/app/pgsql/bin
PGPORT=5432
PGDATA=/app/pgsql/data

if [ -f "$PGDATA/PG_VERSION" ] && [ -d "$PGDATA/base/template1" ]
then
echo "Using old-style directory structure"
else
PGDATA=/app/pgsql/data
fi
PGLOG=/app/pgsql/data/pgstartup.log

# Override defaults from /etc/sysconfig/pgsql if file is present
[ -f /etc/sysconfig/pgsql/${NAME} ] && . /etc/sysconfig/pgsql/${NAME}

export PGDATA
export PGPORT

# Check that networking is up.
# Pretty much need it for postmaster.
[ "${NETWORKING}" = "no" ] && exit 0

[ -f "$PGENGINE/postmaster" ] || exit 1

script_result=0

start(){
PSQL_START=$"Starting ${NAME} service: "

# Make sure startup-time log file is valid
if [ ! -e "$PGLOG" -a ! -h "$PGLOG" ]
then
touch "$PGLOG" || exit 1
chown postgres:postgres "$PGLOG"
chmod go-rwx "$PGLOG"
[ -x /usr/bin/chcon ] && /usr/bin/chcon -u system_u -r object_r -t postgresql_log_t "$PGLOG" 2>/dev/null
fi

# Check for the PGDATA structure
if [ -f "$PGDATA/PG_VERSION" ] && [ -d "$PGDATA/base" ]
then
# Check version of existing PGDATA

if [ x`cat "$PGDATA/PG_VERSION"` != x"$PGMAJORVERSION" ]
then
SYSDOCDIR="(Your System's documentation directory)"
if [ -d "/usr/doc/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/doc
fi
if [ -d "/usr/share/doc/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/share/doc
fi
if [ -d "/usr/doc/packages/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/doc/packages
fi
if [ -d "/usr/share/doc/packages/postgresql-$PGVERSION" ]
then
SYSDOCDIR=/usr/share/doc/packages
fi
echo
echo $"An old version of the database format was found."
echo $"You need to upgrade the data format before using PostgreSQL."
echo $"See $SYSDOCDIR/postgresql-$PGVERSION/README.rpm-dist for more information."
exit 1
fi

# No existing PGDATA! Warn the user to initdb it.
else
echo
echo "$PGDATA is missing. Use \"service postgresql initdb\" to initialize the cluster first."
echo_failure
echo
exit 1
fi

echo -n "$PSQL_START"
$SU -l postgres -c "$PGENGINE/postmaster -p '$PGPORT' -D '$PGDATA' ${PGOPTS} &" >> "$PGLOG" 2>&1 < /dev/null
sleep 2
pid=`pidof -s "$PGENGINE/postmaster"`
if [ $pid ] && [ -f "$PGDATA/postmaster.pid" ]
then
success "$PSQL_START"
touch /var/lock/subsys/${NAME}
head -n 1 "$PGDATA/postmaster.pid" > "/var/run/postmaster.${PGPORT}.pid"
echo
else
failure "$PSQL_START"
echo
script_result=1
fi
}

stop(){
echo -n $"Stopping ${NAME} service: "
$SU -l postgres -c "$PGENGINE/pg_ctl stop -D '$PGDATA' -s -m fast" > /dev/null 2>&1 < /dev/null
ret=$?
if [ $ret -eq 0 ]
then
echo_success
else
echo_failure
script_result=1
fi
echo
rm -f "/var/run/postmaster.${PGPORT}.pid"
rm -f "/var/lock/subsys/${NAME}"
}

restart(){
stop
start
}

initdb(){
if [ -f "$PGDATA/PG_VERSION" ]
then
echo "Data directory is not empty!"
echo_failure
else
echo -n $"Initializing database: "
if [ ! -e "$PGDATA" -a ! -h "$PGDATA" ]
then
mkdir -p "$PGDATA" || exit 1
chown postgres:postgres "$PGDATA"
chmod go-rwx "$PGDATA"
fi
# Clean up SELinux tagging for PGDATA
[ -x /sbin/restorecon ] && /sbin/restorecon "$PGDATA"
# Initialize the database
$SU -l postgres -c "$PGENGINE/initdb --pgdata='$PGDATA' --auth='ident sameuser'" >> "$PGLOG" 2>&1 < /dev/null
# Create directory for postmaster log
mkdir "$PGDATA/pg_log"
chown postgres:postgres "$PGDATA/pg_log"
chmod go-rwx "$PGDATA/pg_log"

[ -f "$PGDATA/PG_VERSION" ] && echo_success
[ ! -f "$PGDATA/PG_VERSION" ] && echo_failure
echo
fi
}
condrestart(){
[ -e /var/lock/subsys/${NAME} ] && restart
}

condstop(){
[ -e /var/lock/subsys/${NAME} ] && stop
}

reload(){
$SU -l postgres -c "$PGENGINE/pg_ctl reload -D '$PGDATA' -s" > /dev/null 2>&1 < /dev/null
}

# This script is slightly unusual in that the name of the daemon (postmaster)
# is not the same as the name of the subsystem (postgresql)

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p /var/run/postmaster.${PGPORT}.pid
script_result=$?
;;
restart)
restart
;;
initdb)
initdb
;;
condrestart)
condrestart
;;
condstop)
condstop
;;
reload|force-reload)
reload
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|condstop|reload|force-reload|initdb}"
exit 1
esac

exit $script_result

Unique lines in a file

If you want to see all unique lines within a file, use the uniq command within Linux/Unix. To do so, just execute uniq against a file.

#uniq /var/log/messages

Or, less elegantly...

#cat /var/log/messages | uniq

This will also work with standard input.

Quick and dirty openLDAP replication

This is a cursory view of how to install and configure a master and slave openLDAP server pair. Unless specified, follow each step on both the master and the slave servers. The only real difference between the servers is the presence of a slurpd configuration on the master and the unique slapd configuration on both servers. In the end, you will have a syncing pair which will replicate changes from the master server to the slave server every three seconds.

A. Install OS and LDAP

1.Install your OS. I am assuming Linux, specifically RHEL or Fedora Core, but openLDAP will run on a variety of systems and these instructions can be adapted to your specific flavor.

2.On both the master and the slave, install openldap, php, httpd and the dependencies with the command (assuming RHEL or Fedora Core):
# yum -t -y install openldap-clients openldap-servers openldap php-ldap nss_ldap httpd php

B. Install and configure phpLDAPadmin (optional)

3.Install phpLDAPadmin from this website: http://phpldapadmin.sourceforge.net/wiki/index.php/Main_Page

4.Untar the download, and then copy the file phpldapadmin-<version>/config/config.php.example to phpldapadmin-/config/config.php

5.Move the phpldapadmin-<version> to /var/www/html/phpldapadmin

6.Restart httpd with the command
#chkconfig httpd on
#service httpd restart

C. Configure and test LDAP

7.Make sure that the master server's hostname is pingable from the slave and vice versa. If not, add the entries to /etc/hosts and restart networking with the command:
#service network restart

8.Copy /etc/openldap/ldap.conf to /etc/openldap/ldap.conf.orig

9.Copy /etc/openldap/sldapd.conf to /etc/openldap/slapd.conf.orig

10.Copy /etc/openldap/DB_CONFIG.example to /var/lib/ldap/DB_CONFIG

11.Copy the configuration files to the respective servers. These are located at the bottom of this document. Make sure to copy the correct ldap.conf and slapd.conf to their respective servers.

12.Import the base dn from the base.ldif file (included later in this document)
#slapadd < /etc/openldap/base.ldif

13.Start the LDAP service
# chkconfig ldap on
# service ldap start

14.Point to http://hostname/ or http://hostname/phpldapadmin If you get a “php memory too low” error, change the memory limit to something meaningful in /etc/php.ini
memory_limit = 128M ; For example

15.The login for the server is cn=root,dc=example,dc=com and the password needs to be set with slappasswd. Use slappasswd and change the hash in the /etc/openldap/slapd.conf file.
# slappasswd
New password:
Re-enter new password:
{SSHA}At/pOvtko2KXcKfM7t0o/OPedJrpXQM0

Now enter the line with the hashed password in the file /etc/openldap/slapd.conf as shown:
rootpw {SSHA}At/pOvtko2KXcKfM7t0o/OPedJrpXQM0

16.From the phpLDAPadmin GUI or using #slapadd similar to before, import the ldif file from a backup or existing server to the master server. If you have not created or do not have a backup of the ldif file of the old directory server, the other option is to copy the /var/lib/ldap directory over to the new server. If starting from scratch, this is a mute point.

17.If syncing is working, it will be replicated on the slave server. If not, the file /var/lib/ldap/replica/openldap-master-replog on the master server will tell you why.

18.For testing the syncing and replication of the master and slave servers, add a new entry to the master server and see if the entry appears on the slave server. For testing the subordination of the slave server, create an entry on the slave server and watch as it is not replicated on the master server.

/etc/openldap/base.ldif (For both servers)
dn: dc=example,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
dc: example
o: Example

/etc/openldap/ldap.conf (For master server)
URI ldap://127.0.0.1/
BASE dc=example,dc=com
TLS_CACERTDIR /etc/openldap/cacerts

/etc/openldap/slapd.conf (For master server)
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
#Note that this should be changed based upon the hostname or user for greater security
access to *
by * read
by anonymous auth
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn. (e.g., "access to * by * read")
# rootdn can always read and write EVERYTHING!
database bdb
suffix "dc=example,dc=com"
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}/mYjTZhwSR1hIGKt6qD0oBpHdRjeHSGh
directory /var/lib/ldap
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
# Replicas of this database
replogfile /var/lib/ldap/openldap-master-replog
replica host="slave:389"
suffix="dc=example,dc=com"
binddn="cn=root,dc=example,dc=com"
credentials=changeme
bindmethod=simple

/etc/openldap/ldap.conf (For slave server)
URI ldap://127.0.0.1
BASE dc=example,dc=com
TLS_CACERTDIR /etc/openldap/cacerts
updatedn "cn=root,dc=example,dc=com"
updateref ldap://master
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}nlb8tJHDUJCMqQMBMyMIIu26VF1ViVEu

/etc/openldap/slapd.conf (For slave server)
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
# Note that you should change this based upon the hostname of the master server.
access to *
by * write
by anonymous auth
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn. (e.g., "access to * by * read")
# rootdn can always read and write EVERYTHING!
database bdb
suffix "dc=example,dc=com"
rootdn "cn=root,dc=example,dc=com"
rootpw {SSHA}/mYjTZhwsR1hIGKt6qD0oBpHdRjeHSGh
updatedn "cn=root,dc=example,dc=com"
updateref ldap://master
directory /var/lib/ldap
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
TLSCertificateFile /etc/openldap/ldap.cert
TLSCertificateKeyFile /etc/openldap/ldap.key

DNS lookups

To find a hostname when you know the IP address:

nmblookup -A <ip_address>

eg. nmblookup -A 192.168.0.148

or

host <ip_address>

eg. host 192.168.0.148

To find an IP address when you know the hostname, try the following ways:

whois www.google.com
dig www.google.com
nslookup www.google.com

Install and configure NTP

NTP is great at keeping your Linux server or desktop's time synced. Not having the time synced can potentially cause issues with backup software, applications and some web applications. Here is a sample script to install and configure NTP on Linux. This was created for Red Hat, but it should work with other versions of Linux with few modifications (like the installation of the init scripts).

#NTP configuration script.

date
cat /var/lib/ntp/drift
chkconfig ntpd --list
service ntpd stop
ntpdate -u 0.rhel.pool.ntp.org
ntpdate -u 1.rhel.pool.ntp.org
ntpdate -u 2.rhel.pool.ntp.org
chkconfig ntpd on
cat /etc/ntp.conf | grep server
vi /etc/ntp.conf

#Based upon the output of those commands, add (or delete) the following lines in /etc/ntp.conf
server 0.rhel.pool.ntp.org
server 1.rhel.pool.ntp.org
server 2.rhel.pool.ntp.org

#Now save and test
service ntpd start
sleep 4
ntpq -p
cat /var/lib/ntp/drift
date

List files in an rpm file

rpm -qlp mysql5-0.rpm

Rename files which match specific criteria

This renames all files within the current working directory from .exe to .exeold. This is useful if you are trying to rename any file extension from one to another.

for file in *.exe ; do mv $file `echo $file | sed 's/\(.*\.\)exe/\1exeold/'` ; done

Simple bash for loop

for kernelrpm in kernel-smp-2.6.9-5.EL kernel-smp-2.6.9-34.EL kernel-smp-2.6.9-67.0.1.EL kernel-utils-2.4-14.1.117

do rpm -e $kernelrpm

done

Boot FreeBSD with Grub

If you have installed FreeBSD (or any operating system) and a partition and then later installed Linux (or any operating system with Grub) on another partition and can't get the latter to boot the former, add this line to /boot/grub/menu.lst

# For booting FreeBSD
title  FreeBSD 7.0
root   (hd0,1)
chainloader +1

In this case, the partition was /dev/sda2 which contained FreeBSD. Modify it for your needs.