List only directories in Linux

There are many ways to list only directories in a linux machine.

1) ls -ld /*

2) tree -di

3) ls -l | grep '^d' | awk '{print $8}' ### for Red Hat put $9 & for OpenSuse put $8

SVN book online

This is a place to read HTML and PDF versions of the book: http://svnbook.red-bean.com/

What is epoch time?

The Unix epoch (or Unix time or POSIX time or Unix timestamp) is the number of seconds that have elapsed since January 1, 1970 (midnight UTC/GMT), not counting leap seconds (in ISO 8601: 1970-01-01T00:00:00Z). Literally speaking the epoch is Unix time 0 (midnight 1-1-1970), but 'epoch' is often used as a synonym for 'Unix time'. Many Unix systems store epoch dates as a signed 32-bit integer, which might cause problems on January 19, 2038 (known as the Year 2038 problem or Y2038).

Human readable time	Seconds
1 minute	60 seconds
1 hour	3600 seconds
1 day	86400 seconds
1 week	604800 seconds
1 month (30.44 days)	2629743 seconds
1 year (365.24 days)	31556926 seconds

More info on epoch can be read at http://www.epochconverter.com/

Add Linux user through script

Run the below script as root.

#!/bin/bash

if [ $(id -u) -eq 0 ]; then
   read -p "Enter username : " username
   read -s -p "Enter password : " password
   egrep "^$username" /etc/passwd >/dev/null
   if [ $? -eq 0 ]; then
       echo "$username exists!"
       exit 1
   else
       pass=$(perl -e 'print crypt($ARGV[0], "password")' $password)
       useradd -m -p $pass $username
       [ $? -eq 0 ] && echo "User has been added to system!" || echo "Failed to add a user!"
   fi
else
   echo "Only root may add a user to the system"
   exit 2
fi

Tomcat Clustering

Many times it simply is not possible to satisfy all of your user load with a single server. When this is the case then your only option is to dedicate more than one server to satisfy your user requests, but dedicating more than one server to handle requests presents new challenges.

The primary challenge is how to respond when a server in the group goes down. Typically a server maintains stateful information about its users in a session object. It would be preferable if, when a server goes down, the user “falls over” to another server without being aware that he or she has changed servers. But in order for this to work, the user’s stateful information needs to be maintained in both the original server as well as the server that the user falls over to.

More……

Tools To Find Out Website Load Speed

Research shows that if your web pages take longer than 5 seconds to load, you lose 50% of your viewers and sales. As a UNIX admin often end users and web developers complain about website loading speed and timings. Usually, there is nothing wrong with my servers or server farm. Fancy java script and images / flash makes site pretty slow. These tools are useful to debug performance problems for sys admins, developers and end users. Here are six tools that can analyzes web pages and tells you why they are slow. Use the following tools to:

More…….

Memory Usage in Linux by Application

This Python script is written by Pádraig Brady , He can be contacted at P@draigBrady.com.

Usage:

For checking memory usage of Firefox, type the below command in console as root user.

./ps_mem.py | grep firefox

105.6 MiB +  8.4 MiB = 114.0 MiB      firefox-bin

Code : 

#!/usr/bin/env python
# Try to determine how much RAM is currently being used per program.
# Note the per program, not per process. So for example this script
# will report mem used by all httpd process together. In detail it reports:
# sum(all RSS for process instances) + max(shared mem for any process instance)
#
# The shared calculation below will factor out shared text and
# libs etc. within a program, but not between programs. So there
# will always be some overestimation. This will be the same for
# all processes that just use libc for e.g. but more for others
# that use larger shared libs like gnome, kde etc.


# Author: P@draigBrady.com

# V1.0      06 Jul 2005    Initial release
# V1.1      11 Aug 2006    root permission required for accuracy
# V1.2      08 Nov 2006    Add total to output
#                          Use KiB,MiB,... for units rather than K,M,...
# V1.3      22 Nov 2006    Ignore shared col from /proc/$pid/statm for
#                          2.6 kernels up to and including 2.6.9.
#                          There it represented the total file backed extent
# V1.4      23 Nov 2006    Remove total from output as it's meaningless
#                          (the shared values overlap with other programs).
#                          Display the shared column. This extra info is
#                          useful, especially as it overlaps between programs.
# V1.5      26 Mar 2007    Remove redundant recursion from human()
# V1.6      05 Jun 2007    Also report number of processes with a given name.
#                          Patch from riccardo.murri@gmail.com

# Notes:
#
# All interpreted programs where the interpreter is started
# by the shell or with env, will be merged to the interpreter
# (as that's what's given to exec). For e.g. all python programs
# starting with "#!/usr/bin/env python" will be grouped under python.
# You can change this by changing comm= to args= below but that will
# have the undesirable affect of splitting up programs started with
# differing parameters (for e.g. mingetty tty[1-6]).
#
# For 2.6 kernels up to and including 2.6.13 and later 2.4 redhat kernels
# (rmap vm without smaps) it can not be accurately determined how many pages
# are shared between processes in general or within a program in our case:
# http://lkml.org/lkml/2005/7/6/250
# A warning is printed if overestimation is possible.
# In addition for 2.6 kernels up to 2.6.9 inclusive, the shared
# value in /proc/$pid/statm is the total file-backed extent of a process.
# We ignore that, introducing more overestimation, again printing a warning.
#
# I don't take account of memory allocated for a program
# by other programs. For e.g. memory used in the X server for
# a program could be determined, but is not.
#
# This script assumes threads are already merged by ps

# TODO:
#
# use ps just to enumerate the pids and names
# so as to remove the race between reading rss and shared values

import sys, os, string

if os.geteuid() != 0:
    sys.stderr.write("Sorry, root permission required.\n");
    sys.exit(1)

PAGESIZE=os.sysconf("SC_PAGE_SIZE")/1024 #KiB
our_pid=os.getpid()

#(major,minor,release)
def kernel_ver():
    kv=open("/proc/sys/kernel/osrelease").readline().split(".")[:3]
    for char in "-_":
        kv[2]=kv[2].split(char)[0]
    return (int(kv[0]), int(kv[1]), int(kv[2]))

kv=kernel_ver()

def getShared(pid):
    if os.path.exists("/proc/"+str(pid)+"/smaps"):
        shared_lines=[line
                      for line in open("/proc/"+str(pid)+"/smaps").readlines()
                      if line.find("Shared")!=-1]
        return sum([int(line.split()[1]) for line in shared_lines])
    elif (2,6,1) <= kv <= (2,6,9):
        return 0 #lots of overestimation, but what can we do?
    else:
        return int(open("/proc/"+str(pid)+"/statm").readline().split()[2])*PAGESIZE

cmds={}
shareds={}
count={}
for line in os.popen("ps -e -o rss=,pid=,comm=").readlines():
    size, pid, cmd = map(string.strip,line.strip().split(None,2))
    if int(pid) == our_pid:
        continue #no point counting this process
    try:
        shared=getShared(pid)
    except:
        continue #ps gone away
    if shareds.get(cmd):
        if shareds[cmd] < shared:
            shareds[cmd]=shared
    else:
        shareds[cmd]=shared
    #Note shared is always a subset of rss (trs is not always)
    cmds[cmd]=cmds.setdefault(cmd,0)+int(size)-shared
    if count.has_key(cmd):
      count[cmd] += 1
    else:
      count[cmd] = 1

#Add max shared mem for each program
for cmd in cmds.keys():
    cmds[cmd]=cmds[cmd]+shareds[cmd]

sort_list = cmds.items()
sort_list.sort(lambda x,y:cmp(x[1],y[1]))
sort_list=filter(lambda x:x[1],sort_list) #get rid of zero sized processes (kernel threads)

#The following matches "du -h" output
#see also human.py
def human(num, power="Ki"):
    powers=["Ki","Mi","Gi","Ti"]
    while num >= 1000: #4 digits
        num /= 1024.0
        power=powers[powers.index(power)+1]
    return "%.1f %s" % (num,power)

def cmd_with_count(cmd, count):
    if count>1:
      return "%s (%u)" % (cmd, count)
    else:
      return cmd
print " Private  +  Shared  =  RAM used\tProgram \n"
for cmd in sort_list:
    print "%8sB + %8sB = %8sB\t%s" % (human(cmd[1]-shareds[cmd[0]]), human(shareds[cmd[0]]), human(cmd[1]),
                                      cmd_with_count(cmd[0], count[cmd[0]]))
print "\n Private  +  Shared  =  RAM used\tProgram \n"

#Warn of possible inaccuracies
#1 = accurate
#0 = some shared mem not reported
#-1= all shared mem not reported
def shared_val_accurate():
    """http://wiki.apache.org/spamassassin/TopSharedMemoryBug"""
    if kv[:2] == (2,4):
        if open("/proc/meminfo").read().find("Inact_") == -1:
            return 1
        return 0
    elif kv[:2] == (2,6):
        if os.path.exists("/proc/"+str(os.getpid())+"/smaps"):
            return 1
        if (2,6,1) <= kv <= (2,6,9):
            return -1
        return 0
    else:
        return 1

vm_accuracy = shared_val_accurate()
if vm_accuracy == -1:
    sys.stderr.write("Warning: Shared memory is not reported by this system.\n")
    sys.stderr.write("Values reported will be too large.\n")
elif vm_accuracy == 0:
    sys.stderr.write("Warning: Shared memory is not reported accurately by this system.\n")
    sys.stderr.write("Values reported could be too large.\n")

Kernel Compile

How to Share RedHat and Fedora Remote Directories with SSHFS

Lots of Linux and Unix power users know how to share remote directories with Samba or NFS.  Unfortunately,  more and more malicious hackers get access to servers through these ways of sharing. For example, if somebody got one of Trojan horse’s access to your Windows machine and you access your server through it, there is a fat chance that the remote directories may be infiltrated to.

There is a solution to that. You can share your remote directories through the SSH file system.  You just need to make sure that your remote server is running SSH (which it usually does) and that it is accessible to your user account on a client machine.

If all this is true, you will need to install with your yum software that is called fuse-sshfs.  Then, naturally, you will need to create a mount point - a directory on your client machine for mounting data from a remote server to your local directory.

When you are done with these simple tasks,  you can start mounting the remote directory like that:

sshfs alex@10.0.0.13:/var/yourremotefolder /mnt/yourlocalfolder

As soon as you finish your work and want to unmount the remote directory, you will need to use the following fusermount command:

fusermount -u /mnt/yourlocalfolder

This solutions will be much safer for communications between Linux machines due to the nature of SSH encryption. Try it, I guarantee that you will like it.

How to remove a file with a dash as first character?

If you accidentally created a file with a - in the beginning then you want to
remove it, you have to do :


# rm ./-thefile

Linux Dark Tips

Top 5 Tips for Migrating from Windows to Linux

Runlevels

Linux
Once booted, the kernel starts init which then starts rc which starts all scripts belonging to a runlevel. The scripts are stored in /etc/init.d and are linked into /etc/rc.d/rcN.d with N the runlevel number.
The default runlevel is configured in /etc/inittab. It is usually 3 or 5:

# grep default: /etc/inittab                                       
id:3:initdefault:

The actual runlevel can be changed with init. For example to go from 3 to 5:

# init 5                             # Enters runlevel 5

    * 0 Shutdown and halt
    * 1 Single-User mode (also S)
    * 2 Multi-user without network
    * 3 Multi-user with network
    * 5 Multi-user with X
    * 6 Reboot

Use chkconfig to configure the programs that will be started at boot in a runlevel.

# chkconfig --list                   # List all init scripts
# chkconfig --list sshd              # Report the status of sshd
# chkconfig sshd --level 35 on       # Configure sshd for levels 3 and 5
# chkconfig sshd off                 # Disable sshd for all runlevels

Debian and Debian based distributions like Ubuntu or Knoppix use the command update-rc.d to manage the runlevels scripts. Default is to start in 2,3,4 and 5 and shutdown in 0,1 and 6.

# update-rc.d sshd defaults          # Activate sshd with the default runlevels
# update-rc.d sshd start 20 2 3 4 5 . stop 20 0 1 6 .  # With explicit arguments
# update-rc.d -f sshd remove         # Disable sshd for all runlevels
# shutdown -h now (or # poweroff)    # Shutdown and halt the system

FreeBSD
The BSD boot approach is different from the SysV, there are no runlevels. The final boot state (single user, with or without X) is configured in /etc/ttys. All OS scripts are located in /etc/rc.d/ and in /usr/local/etc/rc.d/ for third-party applications. The activation of the service is configured in /etc/rc.conf and /etc/rc.conf.local. The default behavior is configured in /etc/defaults/rc.conf. The scripts responds at least to start|stop|status.

# /etc/rc.d/sshd status
sshd is running as pid 552.
# shutdown now                       # Go into single-user mode
# exit                               # Go back to multi-user mode
# shutdown -p now                    # Shutdown and halt the system
# shutdown -r now                    # Reboot

The process init can also be used to reach one of the following states level. For example # init 6 for reboot.

    * 0 Halt and turn the power off (signal USR2)
    * 1 Go to single-user mode (signal TERM)
    * 6 Reboot the machine (signal INT)
    * c Block further logins (signal TSTP)
    * q Rescan the ttys(5) file (signal HUP)

Reset root password
Linux method 1
At the boot loader (lilo or grub), enter the following boot option:

init=/bin/sh

The kernel will mount the root partition and init will start the bourne shell instead of rc and then a runlevel. Use the command passwd at the prompt to change the password and then reboot. Forget the single user mode as you need the password for that.
If, after booting, the root partition is mounted read only, remount it rw:

# mount -o remount,rw /
# passwd                             # or delete the root password (/etc/shadow)
# sync; mount -o remount,ro /        # sync before to remount read only
# reboot

FreeBSD method 1
On FreeBSD, boot in single user mode, remount / rw and use passwd. You can select the single user mode on the boot menu (option 4) which is displayed for 10 seconds at startup. The single user mode will give you a root shell on the / partition.

# mount -u /; mount -a               # will mount / rw
# passwd
# reboot

Unixes and FreeBSD and Linux method 2
Other Unixes might not let you go away with the simple init trick. The solution is to mount the root partition from an other OS (like a rescue CD) and change the password on the disk.

    * Boot a live CD or installation CD into a rescue mode which will give you a shell.
    * Find the root partition with fdisk e.g. fdisk /dev/sda
    * Mount it and use chroot:

# mount -o rw /dev/ad4s3a /mnt
# chroot /mnt                        # chroot into /mnt
# passwd
# reboot

Python Basics

Linux Limits

Some application require higher limits on open files and sockets (like a proxy web server, database). The default limits are usually too low.
Linux
Per shell/script
The shell limits are governed by ulimit. The status is checked with ulimit -a. For example to change the open files limit from 1024 to 10240 do:

# ulimit -n 10240 # This is only valid within the shell

The ulimit command can be used in a script to change the limits for the script only.
Per user/process
Login users and applications can be configured in /etc/security/limits.conf. For example:

# cat /etc/security/limits.conf
* hard nproc 250 # Limit user processes
asterisk hard nofile 409600 # Limit application open files

System wide
Kernel limits are set with sysctl. Permanent limits are set in /etc/sysctl.conf.

# sysctl -a # View all system limits
# sysctl fs.file-max # View max open files limit
# sysctl fs.file-max=102400 # Change max open files limit
# echo "1024 50000" > /proc/sys/net/ipv4/ip_local_port_range # port range
# cat /etc/sysctl.conf
fs.file-max=102400 # Permanent entry in sysctl.conf
# cat /proc/sys/fs/file-nr # How many file descriptors are in use

FreeBSD
Per shell/script
Use the command limits in csh or tcsh or as in Linux, use ulimit in an sh or bash shell.
Per user/process
The default limits on login are set in /etc/login.conf. An unlimited value is still limited by the system maximal value.
System wide
Kernel limits are also set with sysctl. Permanent limits are set in /etc/sysctl.conf or /boot/loader.conf. The syntax is the same as Linux but the keys are different.

# sysctl -a # View all system limits
# sysctl kern.maxfiles=XXXX # maximum number of file descriptors
kern.ipc.nmbclusters=32768 # Permanent entry in /etc/sysctl.conf
kern.maxfiles=65536 # Typical values for Squid
kern.maxfilesperproc=32768
kern.ipc.somaxconn=8192 # TCP queue. Better for apache/sendmail
# sysctl kern.openfiles # How many file descriptors are in use
# sysctl kern.ipc.numopensockets # How many open sockets are in use
# sysctl -w net.inet.ip.portrange.last=50000 # Default is 1024-5000
# netstat -m # network memory buffers statistics

See The FreeBSD handbook Chapter 11http://www.freebsd.org/handbook/configtuning-kernel-limits.html for details.
Solaris
The following values in /etc/system will increase the maximum file descriptors per proc:

set rlim_fd_max = 4096 # Hard limit on file descriptors for a single proc
set rlim_fd_cur = 1024 # Soft limit on file descriptors for a single proc

User Maintenance

# id # Show the active user id with login and group
# last # Show last logins on the system
# who # Show who is logged on the system
# groupadd admin # Add group "admin" and user colin (Linux/Solaris)
# useradd -c "Colin Barschel" -g admin -m colin
# usermod -a -G # Add existing user to group (Debian)
# groupmod -A # Add existing user to group (SuSE)
# userdel colin # Delete user colin (Linux/Solaris)
# adduser joe # FreeBSD add user joe (interactive)
# rmuser joe # FreeBSD delete user joe (interactive)
# pw groupadd admin # Use pw on FreeBSD
# pw groupmod admin -m newmember # Add a new member to a group
# pw useradd colin -c "Colin Barschel" -g admin -m -s /bin/tcsh
# pw userdel colin; pw groupdel admin
Encrypted passwords are stored in /etc/shadow for Linux and Solaris and /etc/master.passwd on FreeBSD. If the master.passwd is modified manually (say to delete a password), run
# pwd_mkdb -p master.passwd to rebuild the database.

To temporarily prevent logins system wide (for all users but root) use nologin. The message in nologin will be displayed (might not work with ssh pre-shared keys).

# echo "Sorry no login now" > /etc/nologin # (Linux)
# echo "Sorry no login now" > /var/run/nologin # (FreeBSD)

Load, statistics and messages

The following commands are useful to find out what is going on on the system.

# top # display and update the top cpu processes
# mpstat 1 # display processors related statistics
# vmstat 2 # display virtual memory statistics
# iostat 2 # display I/O statistics (2 s intervals)
# systat -vmstat 1 # BSD summary of system statistics (1 s intervals)
# systat -tcp 1 # BSD tcp connections (try also -ip)
# systat -netstat 1 # BSD active network connections
# systat -ifstat 1 # BSD network traffic through active interfaces
# systat -iostat 1 # BSD CPU and and disk throughput
# tail -n 500 /var/log/messages # Last 500 kernel/syslog messages
# tail /var/log/warn # System warnings messages see syslog.conf

Running kernel and system information

# uname -a # Get the kernel version (and BSD version)
# lsb_release -a # Full release info of any LSB distribution
# cat /etc/SuSE-release # Get SuSE version
# cat /etc/debian_version # Get Debian version

Use /etc/DISTR-release with DISTR= lsb (Ubuntu), redhat, gentoo, mandrake, sun (Solaris), and so on. See also /etc/issue.

# uptime # Show how long the system has been running + load
# hostname # system's host name
# hostname -i # Display the IP address of the host. (Linux only)
# man hier # Description of the file system hierarchy
# last reboot # Show system reboot history

Hardware Informations

Kernel detected hardware

# dmesg # Detected hardware and boot messages
# lsdev # information about installed hardware
# dd if=/dev/mem bs=1k skip=768 count=256 2>/dev/null | strings -n 8 # Read BIOS

Linux

# cat /proc/cpuinfo # CPU model
# cat /proc/meminfo # Hardware memory
# grep MemTotal /proc/meminfo # Display the physical memory
# watch -n1 'cat /proc/interrupts' # Watch changeable interrupts continuously
# free -m # Used and free memory (-m for MB)
# cat /proc/devices # Configured devices
# lspci -tv # Show PCI devices
# lsusb -tv # Show USB devices
# lshal # Show a list of all devices with their properties
# dmidecode # Show DMI/SMBIOS: hw info from the BIOS

FreeBSD

# sysctl hw.model # CPU model
# sysctl hw # Gives a lot of hardware information
# sysctl vm # Memory usage
# dmesg | grep "real mem" # Hardware memory
# sysctl -a | grep mem # Kernel memory settings and info
# sysctl dev # Configured devices
# pciconf -l -cv # Show PCI devices
# usbdevs -v # Show USB devices
# atacontrol list # Show ATA devices
# camcontrol devlist -v # Show SCSI devices