Bash Shell Script to search for textual string inside a specified directory


As a SysAdmin &/or Developer, going through the log files or code is a daily chore. We do it hundreds of times everyday while at work. Some of us do it even when we are not at work 😉

Many a time, we need to search for a particular piece of text/string in either the log-files or the code. There are many text-editors available that you can use to drill into log-files or your code. Some editors also allow you to search inside directories.

But, if you want a simple, command-line utility that does the job for you, I have written a Bash Shell script that allows you to search for your string inside any given directory. It also counts the total occurrences and shows the line-number for those entries.

My script takes care of the OS (Tested on Mac, Ubuntu & RedHat) & performs installation of a small utility called figlet (according the OS)  which I have used to produce an ASCII art for my script. You can press “n” when prompted, if you don’t want to install or use it.

You can download the script from my GitHub gist.

Below is the script in action (executed on Ubuntu).

If you don’t specify a directory, the script exits.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh

Error!! Please enter absolute path of the directory inside which you want to perform the search. Run this utility as ./textinfile /path/to/directory. Prefix sudo if needed.

Script after specifying a valid directory.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache2/

Detecting OS of your machine..

Your OS is : - Ubuntu.

Checking figlet insallation...

This utility requires a package called figlet in order to display a cool ASCII art ;-) Do you want to install it? Its a one time process only. You can safely ignore this insallation. Press y to install or n to cancel.
y
You chose to install figlet. Installing it for your OS now :-)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
linux-headers-4.4.0-81 linux-headers-4.4.0-81-generic
linux-image-4.4.0-81-generic linux-image-extra-4.4.0-81-generic
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
figlet
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/190 kB of archives.
After this operation, 744 kB of additional disk space will be used.
Selecting previously unselected package figlet.
(Reading database ... 1548998 files and directories currently installed.)
Preparing to unpack .../figlet_2.2.5-2_amd64.deb ...
Unpacking figlet (2.2.5-2) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up figlet (2.2.5-2) ...
update-alternatives: using /usr/bin/figlet-figlet to provide /usr/bin/figlet (figlet) in auto mode
_____ _ ___ _____ _ _
|_ _|____ _| |_|_ _|_ __ | ___(_) | ___
| |/ _ \ \/ / __|| || '_ \| |_ | | |/ _ \
| | __/> <| |_ | || | | | _| | | | __/
|_|\___/_/\_\\__|___|_| |_|_| |_|_|\___|

Welcome to TextInFile! A cute, little utility that allows you to search for the text inside directories from the command-line :-)

Enter the string that you want to search inside /var/log/apache2/...GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1

Searching for GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 in /var/log/apache2/...

GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 was found 2 time(s) inside /var/log/apache2/.

Below is your Search Result :-)

/var/log/apache2/access_reverse_proxy.log.1:7:192.168.0.50 - - [25/Jun/2017:05:35:51 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://192.168.0.51/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"
/var/log/apache2/access_reverse_proxy.log.1:2807:10.0.2.2 - - [27/Jun/2017:03:44:31 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

This utility took 84 seconds to run(including the time you took to type/paste your string).

That's all folks! Hope you liked TextInFile ;-)

If you provide an incorrect directory, the script shows an appropriate error & then exits.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache

The directory /var/log/apache that you specified does't exist. Please choose the correct directory.
Advertisements

Bash Shell Script To Create A Rich, Colorful Server Health Report(in HTML)


As you know,  my previous shell script produces an HTML Server Health Report displaying all vital stats in a plain format & I thought its magical to have a shell script create an HTML report even though it was a rather simple looking report 😉 Then I wrote a web-app that helps you track how much money you have saved & came up with the idea to enhance my script to output a rich, vibrant, vividly colored HTML report that has interactive 3D charts & other colorful visual elements 🙂 Sneak a peek below!

You can grab the script from my GitHub gist page.

Server-Health-Report-Shashank_Srivastava.png

So, I modified my script & made it fetch more server data and display that data in different tables & panels. These tables & panels include : –

Directories that eat up most of your disk space.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table showing the directories that take up most of your disk-space.

Top Memory consuming processes.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table displaying Top Memory consuming processes.

Top CPU consuming processes.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table displaying Top CPU consuming processes.

2 Speedometers displaying # of processes running & # of logged in users.

Server-Health-Report-Shashank_Srivastava.png

Speedometer 🙂

A bars system that displays the resources utilisation (RAM/CPU/Filesystem/INodes) & these bars change their color to red if threshold is crossed 🙂

Server-Health-Report-Shashank_Srivastava.png

Bars displaying vital system stats.

3D interactive pie-chart showing the break-up of filesystem utilisation.

Server-Health-Report-Shashank_Srivastava.png

3D chart with break-up of filesystem usage.

Create a Server Health Report (HTML) Using Shell Script


Shell scripts are insanely powerful & convenient. We all know it 😉 Much of the beauty in shell scripts lies in the way they can be used to automate many aspects of System Administration. As a SysAdmin, you might have been asked to prepare health-reports on a regular basis. Today, I wrote one such script that will generate an HTML health-report containing some vital system information. Lets see how it works 🙂

Lab Description : –

Ubuntu 14.04 Server. Environment : – Bash shell

Instructions : –

Download or clone my GitHub repository from below location.

https://github.com/shashank-ssriva/Linux-Server-HTML-Health-Report-Using-Shell-Script

Place the syshealth.sh file anywhere you want. I prefer keeping it under my home-directory but you may keep it anywhere.

Make it executable (if not already).

You may either run/execute it manually or you may also put it in a CRON job. I have chosen to generate the report twice a day, but its entirely upto you 🙂

Video Tutorial : –

To see the script in action, watch the video below on my YouTube Channel.

Additional Notes : –

I have kept the script & report minimal since I wrote it today only. You may customize it further so as to suit your needs. Sky is the limit 😉

Reset Apache HTTP User Passwords Using Expect In Linux


There are times when we need to reset HTTP user passwords. One of the common examples is Subversion (that uses Apache web front-end) users. There can be very many users who want to have their passwords reset. This can be very boring & tedious 😉 So, below is how I simplified it by writing a shell script using expect 🙂 If you recall, I had used expect earlier as well for this & this. Please read the script carefully. Its pretty self-explanatory 🙂

#Subversion password reset script
#Author : Shashank Srivastava
while read user; do
{
/usr/bin/expect << EOF
#/etc/svn-users is a flat-file that stores users & their encrypted passwords.
spawn htpasswd -m /etc/svn-users $user
puts "$user added"
expect "?New password: \r"
send "PASSWORD\r"
expect "?Re-type new password: \r"
send "PASSWORD\r"
expect eof
EOF
}
#/home/shashank/svn-users.txt is a text-file that stores the user-names
done</home/shashank/svn-users.txt

So, here is the output of my script 🙂

[root@server tcs_admin]# sh scripts/svnusers_pw_reset.sh
spawn htpasswd -m /etc/svn-users user1
user1 added
New password:
Re-type new password:
Updating password for user user1
spawn htpasswd -m /etc/svn-users user2
user2 added
New password:
Re-type new password:
Updating password for user user2

Simple Shell Script To Calculate Total Storage On a Linux Server


Lets face it. Linux doesn’t provide a very simple way to tell how many disks are attached to server. There is no simple way to know the total storage if you don’t have palimpsest tool installed, which is unlikely if you are accessing server through putty 😦

I was asked to check total storage available on the server & it was an overwhelming task since my server has 36 disks attached (of-course coming from SAN)!! So, I decided to write a shell script that would allow me to get all the details in one go 😉

Below is my script : –

DISKNUM=`fdisk -l | grep -e "^Disk /dev/sd" | wc -l`
i=1
while [ $i -le $DISKNUM ]
do
D=`fdisk -l | grep -e "^Disk /dev/sd" | awk '{print$3}' | cut -f1 -d, | head -$i | tail -1`
DISK="$DISK+$D"
SUM=`python -c "print $DISK" | cut -f1 -d.`
TB=$((SUM/1024))
let i=$i+1
done
echo "Total Disks Attached Are : $DISKNUM."
echo "Total Storage Attached is : $SUM GB or $TB TB."

Upon its execution, this is what I found as output.

[shashank@server ~]$ sudo sh disk.sh
Total Disks Attached Are : 36.
Total Storage Attached is : 5411 GB or 5 TB.

Using Expect Command To Automate User Creation on RHEL 6


Many a time my job requires me to add a number of users on a number of servers. The task can soon become painful once the number increases 😦

Thankfully, we have an amazing utility called Expect. Below is how I used Expect to automate user creation task. 🙂 This is basically a small script with self-explanatory tokens. I am also trying to incorporate a for /while loop so as to fully automate this task. This script is ideal for multiple users with same password.

#!/usr/bin/expect -f
#The script uses 2 arguments. user_name & password. Checking this condition.
if {[llength $argv] != 2} {
puts "usage: expect ./expect.sh user_name Password"
exit 1
}
set user [lindex $argv 0] #set user variable. Using first argument as username.
set pass [lindex $argv 1] #set password variable. Using second argument as username.
spawn useradd $user
puts "$user added" #print a message
expect "?server \r"
spawn passwd $user
expect "?Changing password for user $user. \r"
send "$pass\r"
expect "?Retype new password: \r"
send "$pass\r"
expect eof
puts "Password for $user set."

Below is the output 🙂

[shashank@server scripts]$ sudo expect ./expect.sh test4 Password
spawn useradd test4
test4 added
spawn passwd test4
Changing password for user test4.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.
Password for test4 set.

Bash Script To Check If A Device Is In A Specific Network | Cutting First Octet of IP Address


Had to configure a Print Server few week back. While the configurations was easy, its testing was way too cumbersome. Thanks to this trick that did the job 🙂

Now that the configuration was correct, my task was to configure many (62 to be precise) printers on my Print Server. A few printers were in a different network. So, the task was to first ping them & determine their location. Doing it manually would have been a pain (pinging 62 devices) 😦 So I chose to write a script to help me achieve it 😉

Below is that script.

while read line
do
ping_response=`ping -c1 $line|grep PING|cut -d"(" -f2|cut -d")" -f1 2>/dev/null`
ipAddressFirstOctet=`echo "$ping_response"|cut -d"." -f1`
re='^[0-9]+$' //Any output other than ping response. Happens when ping doesn't find that host.
if [[ $ipAddressFirstOctet =~ $re ]]
then
if [ "$ipAddressFirstOctet" -eq 10 ] //10 is first octet for our DMZ
then
echo "$line ($ping_response) is TCS Printer"
else
echo "$line ($ping_response) is CSC Printer"
fi
fi
done<~shashank/printer_list