Bash Shell Script to search for textual string inside a specified directory

As a SysAdmin &/or Developer, going through the log files or code is a daily chore. We do it hundreds of times everyday while at work. Some of us do it even when we are not at work πŸ˜‰

Many a time, we need to search for a particular piece of text/string in either the log-files or the code. There are many text-editors available that you can use to drill into log-files or your code. Some editors also allow you to search inside directories.

But, if you want a simple, command-line utility that does the job for you, I have written a Bash Shell script that allows you to search for your string inside any given directory. It also counts the total occurrences and shows the line-number for those entries.

My script takes care of the OS (Tested on Mac, Ubuntu & RedHat) & performs installation of a small utility called figlet (according the OS) Β which I have used to produce an ASCII art for my script. You can press “n” when prompted, if you don’t want to install or use it.

You can download the script from my GitHub gist.

Below is the script in action (executed on Ubuntu).

If you don’t specify a directory, the script exits.

root@shashank-dbserver:/home/shashank/Desktop# ./

Error!! Please enter absolute path of the directory inside which you want to perform the search. Run this utility as ./textinfile /path/to/directory. Prefix sudo if needed.

Script after specifying a valid directory.

root@shashank-dbserver:/home/shashank/Desktop# ./ /var/log/apache2/

Detecting OS of your machine..

Your OS is : - Ubuntu.

Checking figlet insallation...

This utility requires a package called figlet in order to display a cool ASCII art ;-) Do you want to install it? Its a one time process only. You can safely ignore this insallation. Press y to install or n to cancel.
You chose to install figlet. Installing it for your OS now :-)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
linux-headers-4.4.0-81 linux-headers-4.4.0-81-generic
linux-image-4.4.0-81-generic linux-image-extra-4.4.0-81-generic
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/190 kB of archives.
After this operation, 744 kB of additional disk space will be used.
Selecting previously unselected package figlet.
(Reading database ... 1548998 files and directories currently installed.)
Preparing to unpack .../figlet_2.2.5-2_amd64.deb ...
Unpacking figlet (2.2.5-2) ...
Processing triggers for man-db ( ...
Setting up figlet (2.2.5-2) ...
update-alternatives: using /usr/bin/figlet-figlet to provide /usr/bin/figlet (figlet) in auto mode
_____ _ ___ _____ _ _
|_ _|____ _| |_|_ _|_ __ | ___(_) | ___
| |/ _ \ \/ / __|| || '_ \| |_ | | |/ _ \
| | __/> <| |_ | || | | | _| | | | __/
|_|\___/_/\_\\__|___|_| |_|_| |_|_|\___|

Welcome to TextInFile! A cute, little utility that allows you to search for the text inside directories from the command-line :-)

Enter the string that you want to search inside /var/log/apache2/...GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1

Searching for GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 in /var/log/apache2/...

GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 was found 2 time(s) inside /var/log/apache2/.

Below is your Search Result :-)

/var/log/apache2/access_reverse_proxy.log.1:7: - - [25/Jun/2017:05:35:51 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"
/var/log/apache2/access_reverse_proxy.log.1:2807: - - [27/Jun/2017:03:44:31 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

This utility took 84 seconds to run(including the time you took to type/paste your string).

That's all folks! Hope you liked TextInFile ;-)

If you provide an incorrect directory, the script shows an appropriate error & then exits.

root@shashank-dbserver:/home/shashank/Desktop# ./ /var/log/apache

The directory /var/log/apache that you specified does't exist. Please choose the correct directory.

Automate PostgreSQL Database Instance Creation

At times there are needs to setup multiple PostgreSQL instances. I have explained you how to install PostgreSQL & how to automate its installation on Linux. This time I will show you how to automate PostgreSQL instances creation using a shell-script. Instances are PostgreSQL installations to different directories running on different ports.

Below is the script that I wrote to automate this task πŸ™‚ It will prompt you to enter the location & port.

#set -x
#PostgreSQL instance creation script version 1.0
#Author : - Shashank Srivastava
echo "Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path."
echo ""
ls -lf /opt | grep -i postgre | sort
echo ""
echo -n "Enter the absolute path for this instance. "
read ans
echo ""
echo "You chose $ans. Below are the Postgre ports currently in use. Please choose a unique port when prompted."
echo ""
netstat -tulnp | grep postgre
echo ""
echo -n "Enter port for this new instance to run on. "
echo ""
mkdir -p $ans
sleep 2s
mkdir -p $ans/data
sleep 2s
chown -R postgres $ans/data
cd /home/shashank/postgresql-9.3.5
./configure -prefix=$ans
make install
su - postgres -c "$ans/bin/initdb -D $ans/data"
su - postgres -c "cp -p $ans/data/postgresql.conf $ans/data/postgresql.conf.bkp"
sed -i "s/#port = 5432/port = $PORTNUM/g" $ans/data/postgresql.conf
su - postgres -c "$ans/bin/pg_ctl -D $ans/data -l logfile start"
echo ""
echo "Checking PostgreSQL verison..."
echo ""
$ans/bin/psql --version
echo ""
echo "Postgre instance is running on port $PORTNUM"
netstat -tulnp | grep $PORTNUM
echo ""
echo "Installation Successful."

Here’s the script in action πŸ˜‰

[root@server shashank]# sh
Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path.


Enter the absolute path for this instance. abc

You chose abc. Below are the Postgre ports currently in use. Please choose a unique port when prompted.

tcp        0      0    *                   LISTEN      785/postgres
tcp        0      0    *                   LISTEN      8863/postgres
tcp        0      0      *                   LISTEN      6723/postgres
tcp        0      0      *                   LISTEN      51663/postgres
tcp        0      0      *                   LISTEN      35999/postgres
tcp        0      0    *                   LISTEN      48190/postgres
tcp        0      0    *                   LISTEN      57141/postgres
tcp        0      0 ::1:5437                    :::*                        LISTEN      785/postgres
tcp        0      0 ::1:5438                    :::*                        LISTEN      8863/postgres
tcp        0      0 :::5432                     :::*                        LISTEN      6723/postgres
tcp        0      0 :::5433                     :::*                        LISTEN      51663/postgres
tcp        0      0 :::5434                     :::*                        LISTEN      35999/postgres
tcp        0      0 ::1:5435                    :::*                        LISTEN      48190/postgres
tcp        0      0 ::1:5436                    :::*                        LISTEN      57141/postgres

Enter port for this new instance to run on. 

Reset Apache HTTP User Passwords Using Expect In Linux

There are times when we need to reset HTTP user passwords. One of the common examples is Subversion (that uses Apache web front-end) users. There can be very many users who want to have their passwords reset. This can be very boring & tedious πŸ˜‰ So, below is how I simplified it by writing a shell script using expect πŸ™‚ If you recall, I had used expect earlier as well for this & this. Please read the script carefully. Its pretty self-explanatory πŸ™‚

#Subversion password reset script
#Author : Shashank Srivastava
while read user; do
/usr/bin/expect << EOF
#/etc/svn-users is a flat-file that stores users & their encrypted passwords.
spawn htpasswd -m /etc/svn-users $user
puts "$user added"
expect "?New password: \r"
send "PASSWORD\r"
expect "?Re-type new password: \r"
send "PASSWORD\r"
expect eof
#/home/shashank/svn-users.txt is a text-file that stores the user-names

So, here is the output of my script πŸ™‚

[root@server tcs_admin]# sh scripts/
spawn htpasswd -m /etc/svn-users user1
user1 added
New password:
Re-type new password:
Updating password for user user1
spawn htpasswd -m /etc/svn-users user2
user2 added
New password:
Re-type new password:
Updating password for user user2

Automating PostgreSQl Installation Using Shell Script

This post taught you how to install PostgreSQL Database on Linux using source code. That approach is good when number of servers is less. But what if it is to be installed on a large number of servers? πŸ˜‰ Β Such as in a clustered environment? Here’s how to achieve it using a shell script. πŸ™‚

Better enable password-less SSH from your master server/jumpbox to all destination servers to avoid typing passwords. Then copy the installer tarball from jumpbox to all servers using below command.

for hst in `cat /home/shashank/hosts.txt`; do scp $hst:/home/shashank; done

Now that installer has been copied to destination servers, its time to run the script from jumpbox to install it on all destination servers in one go πŸ™‚ Issue below command to execute automation script & install PostgreSQL on all our target servers.

for hst in `cat /home/shashank/hosts.txt`; do ssh -t $hst 'bash -s' < /home/shashank/; done

Below is the installer script.

#PostgreSQL Automation Installer Script version 1
#Author - Shashank Srivastava
#set -x
echo "Checking availability of pre-required packages"
sleep 1s
sudo rpm -q readline-devel.x86_64
if [ $? != 0 ]
echo "Readline package not found. Installing it...."
sudo yum install readline-devel.x86_64 -y
sudo rpm -q zlib-devel-1.2.3-29.el6.x86_64
if [ $? != 0 ]
echo "zlib package not found. Installing it...."
sudo yum install zlib-devel.x86_64 -y
echo "Unpacking PostgreSQL tarball..."
sleep 2s
sudo tar -zxf /home/shashank/postgresql-9.3.5.tar.gz
echo ""
echo "Tarball unpacked. Creating destination directory for Postgre...."
sudo mkdir -p /opt/PostgreSQL/9.3
cd /home/shashank/postgresql-9.3.5/
echo ""
echo "Destination directory created. Building installer from source coude...."
sleep 1s
sudo ./configure --prefix=/opt/PostgreSQL/9.3
sudo make
sleep 2s
echo ""
echo "Installer built. Installing it now...."
sudo make install
sleep 1s
id -a postgres
if [ $? != 0 ]
echo "postgres user not found. Adding it...."
sudo useradd postgres
echo "postgres user already exists. Proceeding to next step...."
echo ""
echo "Configuring PostgreSQL...."
sudo mkdir -p /opt/PostgreSQL/9.3/data
sudo chown postgres /opt/PostgreSQL/9.3/data
echo ""
echo "Defining data directory for Postgre & starting it...."
sudo su - postgres -c "/opt/PostgreSQL/9.3/bin/initdb /opt/PostgreSQL/9.3/data/; sleep 2s; /opt/PostgreSQL/9.3/bin/pg_ctl -D /opt/PostgreSQL/9.3/data/ -l logfile start"
echo ""
echo "Checking version...."
echo "-------------------------------------------"
/opt/PostgreSQL/9.3/bin/psql --version
/opt/PostgreSQL/9.3/bin/postgres --version
echo ""
echo "Script executed successfully!"

Automate Apache Tomcat Installation on Linux Server Using Shell Script

My work involves installing & managing a lot of Linux servers. At times I have to do same work on numerous servers. One such task is installing Tomcat. In these posts I showed you how to install Apache Tomcat & how to check its version on Linux servers.
In this post, I will show you how you can automate the installation process across any number of servers using a shell script πŸ™‚

My approach is to : –

  • Setup password-less SSH on all servers from one jumpbox machine. Jumpbox is a server from which you will execute this script.
  • Create a list of servers in a text file. All servers are typed line by line.
  • Copy Java & Tomcat installers to all servers in one go from this jumpbox.
  • And finally execute the script.

Below are the commands & script through which I accomplished this. All of these are self-explanatory πŸ˜‰

for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/jdk-8u25-linux-x64.tar.gz $hst:/home/shashank; done
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/apache-tomcat-8.0.15.tar.gz $hst:/home/shashank; done

What it does above? For loop reads the contents of text file (server IPs) line by line & copies tarballs there.

for hst in `cat /home/shashank/hosts.txt`; do ssh $hst -t 'bash -s' < ~shashank/; done

The trick above is to use ‘bash -s’. It will accept the script as an argument & execute it. After execution, it will move to next server. Below is my installer script.

Do let me know if it was helpful or not πŸ™‚ I will come up with some more posts soon πŸ™‚

#Apache Tomcat Installer Script
#Author : Shashank Srivastava
#set -x
echo "Logged into `hostname`. Installing here."
#Checking if installer tarballs are present or not. If they are not found, script will print error message & quit.
if [ -f /home/shashank/jdk-8u25-linux-x64.tar.gz ] && [ -f /home/shashank/apache-tomcat-8.0.15.tar.gz ]
echo "Unpacking Java installer tarball.......";
sleep 2s
#waiting for 2 seconds to show you what is being done.
sudo tar -xzvf /home/shashank/jdk-8u25-linux-x64.tar.gz;
echo ""
echo "Java tarball unpacked.";
echo ""
echo "Unpacking Tomcat installer tarball.......";
sleep 2s;
sudo tar -xzvf /home/shashank/apache-tomcat-8.0.15.tar.gz;
echo ""
echo "Tomcat tarball unpacked.";
echo ""
echo "Installing Java & Tomcat to /opt/app directory....";
sudo cp -rp /home/shashank/jdk1.8.0_25 /opt/app;
sudo cp -rp /home/shashank/apache-tomcat-8.0.15 /opt/app;
echo ""
echo "Exporting necessary variables......";
export JAVA_HOME=/opt/app/jdk1.8.0_25;
export PATH=$PATH:/opt/app/jdk1.8.0_25/bin:$PATH;
export CATALINA_HOME=/opt/app/apache-tomcat-8.0.15;
echo "Variables exported.";
echo ""
echo $JAVA_HOME;
echo ""
echo $PATH;
echo ""
sleep 2s;
echo ""
echo "Checking Java & Tomcat versions.";
echo ""
echo "Java is installed at `which java` Directory";
echo ""
java -version;
echo ""
java -cp $CATALINA_HOME/lib/catalina.jar org.apache.catalina.util.ServerInfo;
echo ""
echo "Starting Tomcat server"
echo ""
echo "'
echo "Removing unpacked tarballs from PWD.";
sudo rm -rf /home/shashank/jdk1.8.0_25;
sudo rm -rf /home/shashank/apache-tomcat-8.0.15;
echo "Installer tarballs not found in /home/shashank. Please make sure they exist there. Exiting installation process now."

How to Run sqlplus as a cron Job & Fix “sqlplus: error while loading shared libraries: cannot open shared object file: No such file or directory” Error?

If you have tried running Oracle sqlplus (of course through a script) in a cron job, then you might have encountered this error: –

sqlplus: error while loading shared libraries: cannot open shared object file: No such file or directory

What happens is that, despite defining all the environment variables in /etc/profile or bash_profile, cron doesn’t recognise that & throws above error. Close inspection after debugging the script took me to a conclusion that sqlplus is not able to connect to database, because its not finding its shared library.

If you have encountered this issue, here is how to fix it? πŸ˜‰

Make sure you have exported required variables. Below are my variables in my .bash_profile. Change yours accordingly.

export ORACLE_HOME=/u01/oracle/product/11.2.0/client_1/
export LD_LIBRARY_PATH=/u01/oracle/product/11.2.0/client_1/lib:usr/local/lib

Make sure you source your .bash_profile before executing the script in cron. Edit your script & add below line at the top.

source /home/shashank/.bash_profile

Now try executing the script in cron. It will be successful πŸ˜‰ cron will first source the bash_profile, initialise all the variables & then execute the script πŸ™‚


Add Multiple Users to a Group in Linux

Ever found yourself in a situation to add multiple users to a single group? πŸ˜‰ I know you have πŸ˜‰

Just issue below command as root & it will add all the required users to a group πŸ˜‰

for user in user1 user2 user3 user4 user5; do usermod -a -G group_nameΒ $user; done

If the number of users is too long, then create a text file containing all the users to be added (make sure user names are white-space separated, just like above example) & then pass this text file in for loop. Like this πŸ˜‰

for user in `cat ~shashank/users_list.txt`;Β do usermod -a -G group_nameΒ $user; done