Thursday, February 9, 2012

 Redirecting operators in Linux/ File Descriptor 


 In order to understand about Redirecting operators in Linux we should know how we communicate with a computer. When we are communicating with a computer there should be a way to do it and this is achieved by STDIN (0), STDOUT (1), STDERR (2) file descriptors. There are numbers assigned to these file descriptors as shown below.
TYPE        Symbol
STDIN       0< 
STDOUT      1>
STDERR      2>


 STDIN: stands for STandarD INput. By using this we can give an input to the computer to do some task. Whatever device we used to give input to a computer will come under STDIN.
 A STDIN can be
A keyboard 
A mouse 
A scanner 
A floppy 
A CD/DVD ROM 
A touch screen 
A barcode reader 
Or a card reader
 STDOUT: The abbreviation is StandarD OUTput. By using this we can see the output from a computer. Whatever device we used to get the output from a computer will come under this STDOUT.
 A STDOUT can be
A monitor 
A speaker 
Or a printer
 STDERR: This is abbreviated as STandarD ERRor. By using this, A computer can communicate with user to give him warning/error etc that something went wrong. A STDERR can be
A monitor 
A printer 
A log file 
A LED indicator 
Or a speaker
 Why we require redirecting operators?
 We require redirecting operators in some situations where our standard communication will not meet our requirement. For example sometimes we want to move an error which is popping on a screen to a file for future reference. So at that time we can use these redirecting operators to change the default way of communication between users and computers.
 In Linux there are many redirecting operators (File descriptors) as shown below
 Basic redirecting operators
 > --Output redirecting operator 
< --Input redirecting operator 
>> --Output appending operator. 
<< --Input appending operator (no significant practical use) 
| --Pipe operator, redirects output of a command to next command.
 Advanced redirecting operators

1> --Standard Output redirect operator (same as >) 
1>> --Standard output append redirect operator (same as >>) 
2> --Standard Error redirector 
2>> --Standard error redirector with append 
&> --Standard Output and Error 
tee command --To redirect output to the screen as well as a file 
xargs command --To feed output of command as input files to other command 
- -- (hyphen or Dash) redirection from standard input/output
 We will see above operators with some examples on how to use them. We are giving much details on << operator as it is rarely used and 1>, 1>> as these operators are equal to >, >>.
 Example1: Redirect output of fdisk command to a file for future reference.
 fdisk -l > file1.txt
 Example2: Search if Mitch username is present in /etc/passwd file or not
 grep mitch < /etc/passwd
 Example3: Use output redirect append operators to redirect fdisk command output once again so that I can see both the previous output and present one in single file.
 fdisk -l >> file1.txt
Note: Not discussing input append redirect operator as its very rarely used in practice.
Example4: How can I count all the files with in present working directory
ls -l | wc -l
Example5: Redirect error of a command to a file and output to the screen.
ls -hrtfd 2> file3.txt
 Exampl6: Redirect and append error to a file and output to the screen.

ls -wertdf 2>> file4.txt
 Example7: Redirect both output and error to a file

ls -ewrdter &> file5.txt
 Note: &> and 2>&1 is one and the same.
 tee command: tee(T) command is a special command which will redirect the output to a file as well as screen at same time, please consider it as capital T when it works.
 Example8: Redirect fdisk command to a file as well as on the screen at a time.
 For example if you want to execute fdisk –l command want to see output of the command then execute one more command to redirect output to a file for future reference. We can do that with following commands
 fdisk –l

fdisk –l > fdskout.txt
 But with tee command you no need to execute two command. Below is a single command for the above fdisk example.
 fdisk -l | tee file6.txt
 xargs command: xargs command is one more special command which will redirect output of a command and feed it as input to other commands.
 Example9: List all the files which contain .sh file extension and within them list all the files which contains bash in them.

ls -l *.sh | grep bash
 If I execute above command, this will not give all the files containing bash but will give file name containing bash. For this type of requirements we have to use xargs. Which will consider previous command output as file names and feed them to next command to args as shown below?

ls -l *.sh | xargs grep bash
 - (hyphen or dash) Operator: This operator will act as both input as well as output redirect operator. I did not find much usage on using this operator as input redirecting operator. There are numerous examples for using this as output redirect operation such as with tar, cut commands etc. I have taken following examples from tldp.org.
 Example10: Search for a character in a file1.txt file and feed that to diff command to see the difference between abc.txt file and this new temporary file (file2.txt) files. Normally we do that as follows..

Cat /temp/loc/file1.txt | grep xyz > /temp/loc/file2.txt
Now compare file2.txt with abc file.
diff /temp/loc/abc.txt /temp/loc/file2.txt
 This can be done in single command using – operator

grep xyz /temp/loc/file1.txt | diff /temp/loc/abc.txt –

Thursday, February 2, 2012


RPM Database Recovery

This document provides an overview of how to deal with RPM database corruption with RHEL-3 based RPM. With the switch from LinuxThreads to NPTL there have been an abnormal spike in database corruption in comparison to RHEL-2.1, where is was typically confined to /var/ partition becoming full.

Stale lock file cleanup

If an RPM command hangs, segfaults, or otherwise behaves abnormally during use then the first task is to check for stale lock files. This can be accomplished with -CA option to the rpmdb_stat command:
 # cd /var/lib/rpm
 # /usr/lib/rpm/rpmdb_stat -CA
Look at the output under the sections headed 'Locks grouped by lockers' and 'Locks grouped by object'. If no RPM command is executing, then there should be no entries. The RPM DB format allows many processes to be concurrently reading *AND* writing to the DB, so there is no safe way for an RPM command to identify & remove a stale lock. Stale locks are typically a result of a process being killed in an abnormal manner (power loss, kernel crash, 'kill' from an impatient admin). The locks are maintained in a handful of files named with two initial underscores.
Since there is generally no 100% reliable way to determine if an arbitrary application has a lock on the RPM db, the best time at which to clear the stale locks is while in single user mode. Thus, during the first stage of the boot process, while the system is still single user, the RHEL /etc/rc.d/rc.sysinit script performs a cleanup of existing lock files boot
 # grep rpm /etc/rc.d/rc.sysinit
 rm -f /var/lib/rpm/__db*
Thus to clean up stale lock files, the best course of action is to simply reboot the machine. If for some reason, it is not feasible to reboot the machine, then it is feasible to simply delete the files manually. Before doing this, one must ensure no application has any of the RPM database files open. This can be checked with the lsof command. For example, as root, ensure the following command shows no lines of output
 # lsof | grep /var/lib/rpm 
If this shows no output then it is safe to delete the lock files
 # rm -f /var/lib/rpm/__db*

DB corruption recovery process

If after cleaning up stale lock files, problems are still experianced, then it is likely some level of database corruption is present. The file that usually requires rebuilding is master package metadata file /var/lib/rpm/Packages, and following that the indexes will also need to be re-generated. Before doing any potentially destructive action *ALWAYS* take a backup of the RPM DB (you are doing that regularly anyway right ;-)
 # cd /var/lib
 # tar zcvf /var/preserve/rpmdb-[today's date].tar.gz rpm
Now verify the integrity of the Packages file
 # cd /var/lib/rpm
 # rm -f __db*      # to avoid stale locks
 # /usr/lib/rpm/rpmdb_verify Packages
Iff this reports any errors then a dump and load of the DB is required. After dumping, verify the integrity of the newly loaded Packages file
 # mv Packages Packages.orig
 # /usr/lib/rpm/rpmdb_dump Packages.orig |
      /usr/lib/rpm/rpmdb_load Packages
 # /usr/lib/rpm/rpmdb_verify Packages
Iff the Packages file now passes the verify step, then as an additional sanity check query all headers in the DB by doing, and watch for any messages sent to standard error (it helps to discard standard out when looking for this):
 # rpm -qa 1> /dev/null
Iff this query passes without generating any messages to standard error, then it is time to rebuild the indexes by invoking
 # rpm -v --rebuilddb
At this point you should have a functioning RPM database again. If any of the recovery steps failed, then a bug report should be filed at http://bugzilla.redhat.com/. When creating the report, provide the tar.gz backup of the RPM DB as an attachment, along with any daily package list log files named /var/log/rpm*.

Recovery without /usr/lib/rpm/rpmdb_* tools present

The above recovery steps assume access to various tools under /usr/lib/rpm/, however, prior to verion 4.2.3 of RPM, these tools were only provided by the sub-package rpm-devel which may well not be available. In addition, since the database is in a damaged state, it will not be possible to actually install this additional RPM. There are two ways around this apparent chicken-and-egg scenario.
  1. Use the rpm2cpio command to extract the contents of rpm-devel to a temporary scratch directory
    # cd /root
    # rpm2cpio rpm-devel-4.2.1-4.4.i386.rpm | cpio -idmv
    
    This will place the appropriate tools into /root/usr/lib/rpm
  2. Copy the database to a second machine, repair it there & then copy it back to the original machine & restore it.
     # scp /var/preserve/rpmdb-XXX.tar.gz root@other-machine:/root
     # ssh root@other-machine
     $ cd /root
     $ tar zxvf rpmdb-XXX.tar.gz
    
    Now proceed as per the original instructions, except apply them to the directory /root/rpm, instead of /var/lib/rpm. When recovery is complete, copy the database back
     $ cd /root
     $ tar zcvf rpmdb-recovered-XXX.tar.gz rpm
     $ scp rpmdb-recovered-XXX.tar.gz root@original-machine:/root
     $ ssh root@original-machine
     # cd /var/lib
     # rm -f rpm/__*
     # tar zxvf /root/rpmdb-recovered-XXX.tar.gz