Bash Shell Script to search for textual string inside a specified directory


As a SysAdmin &/or Developer, going through the log files or code is a daily chore. We do it hundreds of times everyday while at work. Some of us do it even when we are not at work 😉

Many a time, we need to search for a particular piece of text/string in either the log-files or the code. There are many text-editors available that you can use to drill into log-files or your code. Some editors also allow you to search inside directories.

But, if you want a simple, command-line utility that does the job for you, I have written a Bash Shell script that allows you to search for your string inside any given directory. It also counts the total occurrences and shows the line-number for those entries.

My script takes care of the OS (Tested on Mac, Ubuntu & RedHat) & performs installation of a small utility called figlet (according the OS)  which I have used to produce an ASCII art for my script. You can press “n” when prompted, if you don’t want to install or use it.

You can download the script from my GitHub gist.

Below is the script in action (executed on Ubuntu).

If you don’t specify a directory, the script exits.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh

Error!! Please enter absolute path of the directory inside which you want to perform the search. Run this utility as ./textinfile /path/to/directory. Prefix sudo if needed.

Script after specifying a valid directory.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache2/

Detecting OS of your machine..

Your OS is : - Ubuntu.

Checking figlet insallation...

This utility requires a package called figlet in order to display a cool ASCII art ;-) Do you want to install it? Its a one time process only. You can safely ignore this insallation. Press y to install or n to cancel.
y
You chose to install figlet. Installing it for your OS now :-)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
linux-headers-4.4.0-81 linux-headers-4.4.0-81-generic
linux-image-4.4.0-81-generic linux-image-extra-4.4.0-81-generic
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
figlet
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/190 kB of archives.
After this operation, 744 kB of additional disk space will be used.
Selecting previously unselected package figlet.
(Reading database ... 1548998 files and directories currently installed.)
Preparing to unpack .../figlet_2.2.5-2_amd64.deb ...
Unpacking figlet (2.2.5-2) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up figlet (2.2.5-2) ...
update-alternatives: using /usr/bin/figlet-figlet to provide /usr/bin/figlet (figlet) in auto mode
_____ _ ___ _____ _ _
|_ _|____ _| |_|_ _|_ __ | ___(_) | ___
| |/ _ \ \/ / __|| || '_ \| |_ | | |/ _ \
| | __/> <| |_ | || | | | _| | | | __/
|_|\___/_/\_\\__|___|_| |_|_| |_|_|\___|

Welcome to TextInFile! A cute, little utility that allows you to search for the text inside directories from the command-line :-)

Enter the string that you want to search inside /var/log/apache2/...GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1

Searching for GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 in /var/log/apache2/...

GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 was found 2 time(s) inside /var/log/apache2/.

Below is your Search Result :-)

/var/log/apache2/access_reverse_proxy.log.1:7:192.168.0.50 - - [25/Jun/2017:05:35:51 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://192.168.0.51/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"
/var/log/apache2/access_reverse_proxy.log.1:2807:10.0.2.2 - - [27/Jun/2017:03:44:31 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

This utility took 84 seconds to run(including the time you took to type/paste your string).

That's all folks! Hope you liked TextInFile ;-)

If you provide an incorrect directory, the script shows an appropriate error & then exits.

root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache

The directory /var/log/apache that you specified does't exist. Please choose the correct directory.
Advertisements

Bash Shell Script To Create A Rich, Colorful Server Health Report(in HTML)


As you know,  my previous shell script produces an HTML Server Health Report displaying all vital stats in a plain format & I thought its magical to have a shell script create an HTML report even though it was a rather simple looking report 😉 Then I wrote a web-app that helps you track how much money you have saved & came up with the idea to enhance my script to output a rich, vibrant, vividly colored HTML report that has interactive 3D charts & other colorful visual elements 🙂 Sneak a peek below!

You can grab the script from my GitHub gist page.

Server-Health-Report-Shashank_Srivastava.png

So, I modified my script & made it fetch more server data and display that data in different tables & panels. These tables & panels include : –

Directories that eat up most of your disk space.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table showing the directories that take up most of your disk-space.

Top Memory consuming processes.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table displaying Top Memory consuming processes.

Top CPU consuming processes.

Server-Health-Report-Shashank_Srivastava.png

Dynamic table displaying Top CPU consuming processes.

2 Speedometers displaying # of processes running & # of logged in users.

Server-Health-Report-Shashank_Srivastava.png

Speedometer 🙂

A bars system that displays the resources utilisation (RAM/CPU/Filesystem/INodes) & these bars change their color to red if threshold is crossed 🙂

Server-Health-Report-Shashank_Srivastava.png

Bars displaying vital system stats.

3D interactive pie-chart showing the break-up of filesystem utilisation.

Server-Health-Report-Shashank_Srivastava.png

3D chart with break-up of filesystem usage.

Create a Server Health Report (HTML) Using Shell Script


Shell scripts are insanely powerful & convenient. We all know it 😉 Much of the beauty in shell scripts lies in the way they can be used to automate many aspects of System Administration. As a SysAdmin, you might have been asked to prepare health-reports on a regular basis. Today, I wrote one such script that will generate an HTML health-report containing some vital system information. Lets see how it works 🙂

Lab Description : –

Ubuntu 14.04 Server. Environment : – Bash shell

Instructions : –

Download or clone my GitHub repository from below location.

https://github.com/shashank-ssriva/Linux-Server-HTML-Health-Report-Using-Shell-Script

Place the syshealth.sh file anywhere you want. I prefer keeping it under my home-directory but you may keep it anywhere.

Make it executable (if not already).

You may either run/execute it manually or you may also put it in a CRON job. I have chosen to generate the report twice a day, but its entirely upto you 🙂

Video Tutorial : –

To see the script in action, watch the video below on my YouTube Channel.

Additional Notes : –

I have kept the script & report minimal since I wrote it today only. You may customize it further so as to suit your needs. Sky is the limit 😉

Automate PostgreSQL Database Instance Creation


At times there are needs to setup multiple PostgreSQL instances. I have explained you how to install PostgreSQL & how to automate its installation on Linux. This time I will show you how to automate PostgreSQL instances creation using a shell-script. Instances are PostgreSQL installations to different directories running on different ports.

Below is the script that I wrote to automate this task 🙂 It will prompt you to enter the location & port.

#set -x
#PostgreSQL instance creation script version 1.0
#Author : - Shashank Srivastava
echo "Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path."
echo ""
ls -lf /opt | grep -i postgre | sort
echo ""
echo -n "Enter the absolute path for this instance. "
read ans
echo ""
echo "You chose $ans. Below are the Postgre ports currently in use. Please choose a unique port when prompted."
echo ""
netstat -tulnp | grep postgre
echo ""
echo -n "Enter port for this new instance to run on. "
read PORTNUM
echo ""
mkdir -p $ans
sleep 2s
mkdir -p $ans/data
sleep 2s
chown -R postgres $ans/data
cd /home/shashank/postgresql-9.3.5
./configure -prefix=$ans
make
make install
su - postgres -c "$ans/bin/initdb -D $ans/data"
su - postgres -c "cp -p $ans/data/postgresql.conf $ans/data/postgresql.conf.bkp"
sed -i "s/#port = 5432/port = $PORTNUM/g" $ans/data/postgresql.conf
su - postgres -c "$ans/bin/pg_ctl -D $ans/data -l logfile start"
echo ""
echo "Checking PostgreSQL verison..."
echo ""
$ans/bin/psql --version
echo ""
echo "Postgre instance is running on port $PORTNUM"
netstat -tulnp | grep $PORTNUM
echo ""
echo "Installation Successful."

Here’s the script in action 😉

[root@server shashank]# sh postgre_instance_setup.sh
Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path.

PostgreSQL
PostgreSQL2
PostgreSQL3
PostgreSQL4
PostgreSQL5
PostgreSQL6
PostgreSQL7

Enter the absolute path for this instance. abc

You chose abc. Below are the Postgre ports currently in use. Please choose a unique port when prompted.

tcp        0      0 127.0.0.1:5437              0.0.0.0:*                   LISTEN      785/postgres
tcp        0      0 127.0.0.1:5438              0.0.0.0:*                   LISTEN      8863/postgres
tcp        0      0 0.0.0.0:5432                0.0.0.0:*                   LISTEN      6723/postgres
tcp        0      0 0.0.0.0:5433                0.0.0.0:*                   LISTEN      51663/postgres
tcp        0      0 0.0.0.0:5434                0.0.0.0:*                   LISTEN      35999/postgres
tcp        0      0 127.0.0.1:5435              0.0.0.0:*                   LISTEN      48190/postgres
tcp        0      0 127.0.0.1:5436              0.0.0.0:*                   LISTEN      57141/postgres
tcp        0      0 ::1:5437                    :::*                        LISTEN      785/postgres
tcp        0      0 ::1:5438                    :::*                        LISTEN      8863/postgres
tcp        0      0 :::5432                     :::*                        LISTEN      6723/postgres
tcp        0      0 :::5433                     :::*                        LISTEN      51663/postgres
tcp        0      0 :::5434                     :::*                        LISTEN      35999/postgres
tcp        0      0 ::1:5435                    :::*                        LISTEN      48190/postgres
tcp        0      0 ::1:5436                    :::*                        LISTEN      57141/postgres

Enter port for this new instance to run on. 

Reset Apache HTTP User Passwords Using Expect In Linux


There are times when we need to reset HTTP user passwords. One of the common examples is Subversion (that uses Apache web front-end) users. There can be very many users who want to have their passwords reset. This can be very boring & tedious 😉 So, below is how I simplified it by writing a shell script using expect 🙂 If you recall, I had used expect earlier as well for this & this. Please read the script carefully. Its pretty self-explanatory 🙂

#Subversion password reset script
#Author : Shashank Srivastava
while read user; do
{
/usr/bin/expect << EOF
#/etc/svn-users is a flat-file that stores users & their encrypted passwords.
spawn htpasswd -m /etc/svn-users $user
puts "$user added"
expect "?New password: \r"
send "PASSWORD\r"
expect "?Re-type new password: \r"
send "PASSWORD\r"
expect eof
EOF
}
#/home/shashank/svn-users.txt is a text-file that stores the user-names
done</home/shashank/svn-users.txt

So, here is the output of my script 🙂

[root@server tcs_admin]# sh scripts/svnusers_pw_reset.sh
spawn htpasswd -m /etc/svn-users user1
user1 added
New password:
Re-type new password:
Updating password for user user1
spawn htpasswd -m /etc/svn-users user2
user2 added
New password:
Re-type new password:
Updating password for user user2

Automating PostgreSQl Installation Using Shell Script


This post taught you how to install PostgreSQL Database on Linux using source code. That approach is good when number of servers is less. But what if it is to be installed on a large number of servers? 😉  Such as in a clustered environment? Here’s how to achieve it using a shell script. 🙂

Better enable password-less SSH from your master server/jumpbox to all destination servers to avoid typing passwords. Then copy the installer tarball from jumpbox to all servers using below command.

for hst in `cat /home/shashank/hosts.txt`; do scp $hst:/home/shashank; done

Now that installer has been copied to destination servers, its time to run the script from jumpbox to install it on all destination servers in one go 🙂 Issue below command to execute automation script & install PostgreSQL on all our target servers.

for hst in `cat /home/shashank/hosts.txt`; do ssh -t $hst 'bash -s' < /home/shashank/postgre_installer.sh; done

Below is the installer script.

#PostgreSQL Automation Installer Script version 1
#Author - Shashank Srivastava
#set -x
echo "Checking availability of pre-required packages"
sleep 1s
sudo rpm -q readline-devel.x86_64
if [ $? != 0 ]
then
echo "Readline package not found. Installing it...."
sudo yum install readline-devel.x86_64 -y
fi
sudo rpm -q zlib-devel-1.2.3-29.el6.x86_64
if [ $? != 0 ]
then
echo "zlib package not found. Installing it...."
sudo yum install zlib-devel.x86_64 -y
fi
echo "Unpacking PostgreSQL tarball..."
sleep 2s
sudo tar -zxf /home/shashank/postgresql-9.3.5.tar.gz
echo ""
echo "Tarball unpacked. Creating destination directory for Postgre...."
sudo mkdir -p /opt/PostgreSQL/9.3
cd /home/shashank/postgresql-9.3.5/
echo ""
echo "Destination directory created. Building installer from source coude...."
sleep 1s
sudo ./configure --prefix=/opt/PostgreSQL/9.3
sudo make
sleep 2s
echo ""
echo "Installer built. Installing it now...."
sudo make install
sleep 1s
id -a postgres
if [ $? != 0 ]
then
echo "postgres user not found. Adding it...."
sudo useradd postgres
else
echo "postgres user already exists. Proceeding to next step...."
fi
echo ""
echo "Configuring PostgreSQL...."
sudo mkdir -p /opt/PostgreSQL/9.3/data
sudo chown postgres /opt/PostgreSQL/9.3/data
echo ""
echo "Defining data directory for Postgre & starting it...."
sudo su - postgres -c "/opt/PostgreSQL/9.3/bin/initdb /opt/PostgreSQL/9.3/data/; sleep 2s; /opt/PostgreSQL/9.3/bin/pg_ctl -D /opt/PostgreSQL/9.3/data/ -l logfile start"
echo ""
echo "Checking version...."
echo "-------------------------------------------"
/opt/PostgreSQL/9.3/bin/psql --version
/opt/PostgreSQL/9.3/bin/postgres --version
echo ""
echo "Script executed successfully!"

Automate Apache Tomcat Installation on Linux Server Using Shell Script


My work involves installing & managing a lot of Linux servers. At times I have to do same work on numerous servers. One such task is installing Tomcat. In these posts I showed you how to install Apache Tomcat & how to check its version on Linux servers.
In this post, I will show you how you can automate the installation process across any number of servers using a shell script 🙂

My approach is to : –

  • Setup password-less SSH on all servers from one jumpbox machine. Jumpbox is a server from which you will execute this script.
  • Create a list of servers in a text file. All servers are typed line by line.
  • Copy Java & Tomcat installers to all servers in one go from this jumpbox.
  • And finally execute the script.

Below are the commands & script through which I accomplished this. All of these are self-explanatory 😉

for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/jdk-8u25-linux-x64.tar.gz $hst:/home/shashank; done
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/apache-tomcat-8.0.15.tar.gz $hst:/home/shashank; done

What it does above? For loop reads the contents of text file (server IPs) line by line & copies tarballs there.

for hst in `cat /home/shashank/hosts.txt`; do ssh $hst -t 'bash -s' < ~shashank/tomcat_installer.sh; done

The trick above is to use ‘bash -s’. It will accept the script as an argument & execute it. After execution, it will move to next server. Below is my installer script.

Do let me know if it was helpful or not 🙂 I will come up with some more posts soon 🙂

#Apache Tomcat Installer Script
#Author : Shashank Srivastava
#set -x
echo "Logged into `hostname`. Installing here."
#Checking if installer tarballs are present or not. If they are not found, script will print error message & quit.
if [ -f /home/shashank/jdk-8u25-linux-x64.tar.gz ] && [ -f /home/shashank/apache-tomcat-8.0.15.tar.gz ]
then
echo "Unpacking Java installer tarball.......";
sleep 2s
#waiting for 2 seconds to show you what is being done.
sudo tar -xzvf /home/shashank/jdk-8u25-linux-x64.tar.gz;
echo ""
echo "Java tarball unpacked.";
echo ""
echo "Unpacking Tomcat installer tarball.......";
sleep 2s;
sudo tar -xzvf /home/shashank/apache-tomcat-8.0.15.tar.gz;
echo ""
echo "Tomcat tarball unpacked.";
echo ""
echo "Installing Java & Tomcat to /opt/app directory....";
sudo cp -rp /home/shashank/jdk1.8.0_25 /opt/app;
sudo cp -rp /home/shashank/apache-tomcat-8.0.15 /opt/app;
echo ""
echo "Exporting necessary variables......";
export JAVA_HOME=/opt/app/jdk1.8.0_25;
export PATH=$PATH:/opt/app/jdk1.8.0_25/bin:$PATH;
export CATALINA_HOME=/opt/app/apache-tomcat-8.0.15;
echo "Variables exported.";
echo ""
echo $JAVA_HOME;
echo ""
echo $PATH;
echo ""
echo $CATALINA_HOME;
sleep 2s;
echo ""
echo "Checking Java & Tomcat versions.";
echo ""
echo "Java is installed at `which java` Directory";
echo ""
java -version;
echo ""
java -cp $CATALINA_HOME/lib/catalina.jar org.apache.catalina.util.ServerInfo;
echo ""
echo "Starting Tomcat server"
echo ""
cd $CATALINA_HOME/bin
./startup.sh
echo "'
echo "Removing unpacked tarballs from PWD.";
sudo rm -rf /home/shashank/jdk1.8.0_25;
sudo rm -rf /home/shashank/apache-tomcat-8.0.15;
else
echo "Installer tarballs not found in /home/shashank. Please make sure they exist there. Exiting installation process now."
exit
fi