As a SysAdmin &/or Developer, going through the log files or code is a daily chore. We do it hundreds of times everyday while at work. Some of us do it even when we are not at work 😉
Many a time, we need to search for a particular piece of text/string in either the log-files or the code. There are many text-editors available that you can use to drill into log-files or your code. Some editors also allow you to search inside directories.
But, if you want a simple, command-line utility that does the job for you, I have written a Bash Shell script that allows you to search for your string inside any given directory. It also counts the total occurrences and shows the line-number for those entries.
My script takes care of the OS (Tested on Mac, Ubuntu & RedHat) & performs installation of a small utility called figlet (according the OS) which I have used to produce an ASCII art for my script. You can press “n” when prompted, if you don’t want to install or use it.
You can download the script from my GitHub gist.
Below is the script in action (executed on Ubuntu).
If you don’t specify a directory, the script exits.
Error!! Please enter absolute path of the directory inside which you want to perform the search. Run this utility as ./textinfile /path/to/directory. Prefix sudo if needed.
Script after specifying a valid directory.
root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache2/
Detecting OS of your machine..
Your OS is : - Ubuntu.
Checking figlet insallation...
This utility requires a package called figlet in order to display a cool ASCII art ;-) Do you want to install it? Its a one time process only. You can safely ignore this insallation. Press y to install or n to cancel.
You chose to install figlet. Installing it for your OS now :-)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/190 kB of archives.
After this operation, 744 kB of additional disk space will be used.
Selecting previously unselected package figlet.
(Reading database ... 1548998 files and directories currently installed.)
Preparing to unpack .../figlet_2.2.5-2_amd64.deb ...
Unpacking figlet (2.2.5-2) ...
Processing triggers for man-db (184.108.40.206-1ubuntu1) ...
Setting up figlet (2.2.5-2) ...
update-alternatives: using /usr/bin/figlet-figlet to provide /usr/bin/figlet (figlet) in auto mode
_____ _ ___ _____ _ _
|_ _|____ _| |_|_ _|_ __ | ___(_) | ___
| |/ _ \ \/ / __|| || '_ \| |_ | | |/ _ \
| | __/> <| |_ | || | | | _| | | | __/
|_|\___/_/\_\\__|___|_| |_|_| |_|_|\___|
Welcome to TextInFile! A cute, little utility that allows you to search for the text inside directories from the command-line :-)
Enter the string that you want to search inside /var/log/apache2/...GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1
Searching for GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 in /var/log/apache2/...
GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 was found 2 time(s) inside /var/log/apache2/.
Below is your Search Result :-)
/var/log/apache2/access_reverse_proxy.log.1:7:192.168.0.50 - - [25/Jun/2017:05:35:51 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://192.168.0.51/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"
/var/log/apache2/access_reverse_proxy.log.1:2807:10.0.2.2 - - [27/Jun/2017:03:44:31 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
This utility took 84 seconds to run(including the time you took to type/paste your string).
That's all folks! Hope you liked TextInFile ;-)
If you provide an incorrect directory, the script shows an appropriate error & then exits.
root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache
The directory /var/log/apache that you specified does't exist. Please choose the correct directory.
As you know, my previous shell script produces an HTML Server Health Report displaying all vital stats in a plain format & I thought its magical to have a shell script create an HTML report even though it was a rather simple looking report 😉 Then I wrote a web-app that helps you track how much money you have saved & came up with the idea to enhance my script to output a rich, vibrant, vividly colored HTML report that has interactive 3D charts & other colorful visual elements 🙂 Sneak a peek below!
You can grab the script from my GitHub gist page.
So, I modified my script & made it fetch more server data and display that data in different tables & panels. These tables & panels include : –
Directories that eat up most of your disk space.
Dynamic table showing the directories that take up most of your disk-space.
Top Memory consuming processes.
Dynamic table displaying Top Memory consuming processes.
Top CPU consuming processes.
Dynamic table displaying Top CPU consuming processes.
2 Speedometers displaying # of processes running & # of logged in users.
A bars system that displays the resources utilisation (RAM/CPU/Filesystem/INodes) & these bars change their color to red if threshold is crossed 🙂
Bars displaying vital system stats.
3D interactive pie-chart showing the break-up of filesystem utilisation.
3D chart with break-up of filesystem usage.
Shell scripts are insanely powerful & convenient. We all know it 😉 Much of the beauty in shell scripts lies in the way they can be used to automate many aspects of System Administration. As a SysAdmin, you might have been asked to prepare health-reports on a regular basis. Today, I wrote one such script that will generate an HTML health-report containing some vital system information. Lets see how it works 🙂
Lab Description : –
Ubuntu 14.04 Server. Environment : – Bash shell
Instructions : –
Download or clone my GitHub repository from below location.
syshealth.sh file anywhere you want. I prefer keeping it under my home-directory but you may keep it anywhere.
Make it executable (if not already).
You may either run/execute it manually or you may also put it in a CRON job. I have chosen to generate the report twice a day, but its entirely upto you 🙂
Video Tutorial : –
To see the script in action, watch the video below on my YouTube Channel.
Additional Notes : –
I have kept the script & report minimal since I wrote it today only. You may customize it further so as to suit your needs. Sky is the limit 😉
At times there are needs to setup multiple PostgreSQL instances. I have explained you how to install PostgreSQL & how to automate its installation on Linux. This time I will show you how to automate PostgreSQL instances creation using a shell-script. Instances are PostgreSQL installations to different directories running on different ports.
Below is the script that I wrote to automate this task 🙂 It will prompt you to enter the location & port.
#PostgreSQL instance creation script version 1.0
#Author : - Shashank Srivastava
echo "Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path."
ls -lf /opt | grep -i postgre | sort
echo -n "Enter the absolute path for this instance. "
echo "You chose $ans. Below are the Postgre ports currently in use. Please choose a unique port when prompted."
netstat -tulnp | grep postgre
echo -n "Enter port for this new instance to run on. "
mkdir -p $ans
mkdir -p $ans/data
chown -R postgres $ans/data
su - postgres -c "$ans/bin/initdb -D $ans/data"
su - postgres -c "cp -p $ans/data/postgresql.conf $ans/data/postgresql.conf.bkp"
sed -i "s/#port = 5432/port = $PORTNUM/g" $ans/data/postgresql.conf
su - postgres -c "$ans/bin/pg_ctl -D $ans/data -l logfile start"
echo "Checking PostgreSQL verison..."
echo "Postgre instance is running on port $PORTNUM"
netstat -tulnp | grep $PORTNUM
echo "Installation Successful."
Here’s the script in action 😉
[root@server shashank]# sh postgre_instance_setup.sh
Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path.
Enter the absolute path for this instance. abc
You chose abc. Below are the Postgre ports currently in use. Please choose a unique port when prompted.
tcp 0 0 127.0.0.1:5437 0.0.0.0:* LISTEN 785/postgres
tcp 0 0 127.0.0.1:5438 0.0.0.0:* LISTEN 8863/postgres
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 6723/postgres
tcp 0 0 0.0.0.0:5433 0.0.0.0:* LISTEN 51663/postgres
tcp 0 0 0.0.0.0:5434 0.0.0.0:* LISTEN 35999/postgres
tcp 0 0 127.0.0.1:5435 0.0.0.0:* LISTEN 48190/postgres
tcp 0 0 127.0.0.1:5436 0.0.0.0:* LISTEN 57141/postgres
tcp 0 0 ::1:5437 :::* LISTEN 785/postgres
tcp 0 0 ::1:5438 :::* LISTEN 8863/postgres
tcp 0 0 :::5432 :::* LISTEN 6723/postgres
tcp 0 0 :::5433 :::* LISTEN 51663/postgres
tcp 0 0 :::5434 :::* LISTEN 35999/postgres
tcp 0 0 ::1:5435 :::* LISTEN 48190/postgres
tcp 0 0 ::1:5436 :::* LISTEN 57141/postgres
Enter port for this new instance to run on.
There are times when we need to reset HTTP user passwords. One of the common examples is Subversion (that uses Apache web front-end) users. There can be very many users who want to have their passwords reset. This can be very boring & tedious 😉 So, below is how I simplified it by writing a shell script using
expect 🙂 If you recall, I had used
expect earlier as well for this & this. Please read the script carefully. Its pretty self-explanatory 🙂
#Subversion password reset script
#Author : Shashank Srivastava
while read user; do
/usr/bin/expect << EOF
#/etc/svn-users is a flat-file that stores users & their encrypted passwords.
spawn htpasswd -m /etc/svn-users $user
puts "$user added"
expect "?New password: \r"
expect "?Re-type new password: \r"
#/home/shashank/svn-users.txt is a text-file that stores the user-names
So, here is the output of my script 🙂
[root@server tcs_admin]# sh scripts/svnusers_pw_reset.sh
spawn htpasswd -m /etc/svn-users user1
Re-type new password:
Updating password for user user1
spawn htpasswd -m /etc/svn-users user2
Re-type new password:
Updating password for user user2
This post taught you how to install PostgreSQL Database on Linux using source code. That approach is good when number of servers is less. But what if it is to be installed on a large number of servers? 😉 Such as in a clustered environment? Here’s how to achieve it using a shell script. 🙂
Better enable password-less SSH from your master server/jumpbox to all destination servers to avoid typing passwords. Then copy the installer tarball from jumpbox to all servers using below command.
for hst in `cat /home/shashank/hosts.txt`; do scp $hst:/home/shashank; done
Now that installer has been copied to destination servers, its time to run the script from jumpbox to install it on all destination servers in one go 🙂 Issue below command to execute automation script & install PostgreSQL on all our target servers.
for hst in `cat /home/shashank/hosts.txt`; do ssh -t $hst 'bash -s' < /home/shashank/postgre_installer.sh; done
Below is the installer script.
#PostgreSQL Automation Installer Script version 1
#Author - Shashank Srivastava
echo "Checking availability of pre-required packages"
sudo rpm -q readline-devel.x86_64
if [ $? != 0 ]
echo "Readline package not found. Installing it...."
sudo yum install readline-devel.x86_64 -y
sudo rpm -q zlib-devel-1.2.3-29.el6.x86_64
if [ $? != 0 ]
echo "zlib package not found. Installing it...."
sudo yum install zlib-devel.x86_64 -y
echo "Unpacking PostgreSQL tarball..."
sudo tar -zxf /home/shashank/postgresql-9.3.5.tar.gz
echo "Tarball unpacked. Creating destination directory for Postgre...."
sudo mkdir -p /opt/PostgreSQL/9.3
echo "Destination directory created. Building installer from source coude...."
sudo ./configure --prefix=/opt/PostgreSQL/9.3
echo "Installer built. Installing it now...."
sudo make install
id -a postgres
if [ $? != 0 ]
echo "postgres user not found. Adding it...."
sudo useradd postgres
echo "postgres user already exists. Proceeding to next step...."
echo "Configuring PostgreSQL...."
sudo mkdir -p /opt/PostgreSQL/9.3/data
sudo chown postgres /opt/PostgreSQL/9.3/data
echo "Defining data directory for Postgre & starting it...."
sudo su - postgres -c "/opt/PostgreSQL/9.3/bin/initdb /opt/PostgreSQL/9.3/data/; sleep 2s; /opt/PostgreSQL/9.3/bin/pg_ctl -D /opt/PostgreSQL/9.3/data/ -l logfile start"
echo "Checking version...."
echo "Script executed successfully!"
My work involves installing & managing a lot of Linux servers. At times I have to do same work on numerous servers. One such task is installing Tomcat. In these posts I showed you how to install Apache Tomcat & how to check its version on Linux servers.
In this post, I will show you how you can automate the installation process across any number of servers using a shell script 🙂
My approach is to : –
- Setup password-less SSH on all servers from one jumpbox machine. Jumpbox is a server from which you will execute this script.
- Create a list of servers in a text file. All servers are typed line by line.
- Copy Java & Tomcat installers to all servers in one go from this jumpbox.
- And finally execute the script.
Below are the commands & script through which I accomplished this. All of these are self-explanatory 😉
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/jdk-8u25-linux-x64.tar.gz $hst:/home/shashank; done
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/apache-tomcat-8.0.15.tar.gz $hst:/home/shashank; done
What it does above? For loop reads the contents of text file (server IPs) line by line & copies tarballs there.
for hst in `cat /home/shashank/hosts.txt`; do ssh $hst -t 'bash -s' < ~shashank/tomcat_installer.sh; done
The trick above is to use ‘bash -s’. It will accept the script as an argument & execute it. After execution, it will move to next server. Below is my installer script.
Do let me know if it was helpful or not 🙂 I will come up with some more posts soon 🙂
#Apache Tomcat Installer Script
#Author : Shashank Srivastava
echo "Logged into `hostname`. Installing here."
#Checking if installer tarballs are present or not. If they are not found, script will print error message & quit.
if [ -f /home/shashank/jdk-8u25-linux-x64.tar.gz ] && [ -f /home/shashank/apache-tomcat-8.0.15.tar.gz ]
echo "Unpacking Java installer tarball.......";
#waiting for 2 seconds to show you what is being done.
sudo tar -xzvf /home/shashank/jdk-8u25-linux-x64.tar.gz;
echo "Java tarball unpacked.";
echo "Unpacking Tomcat installer tarball.......";
sudo tar -xzvf /home/shashank/apache-tomcat-8.0.15.tar.gz;
echo "Tomcat tarball unpacked.";
echo "Installing Java & Tomcat to /opt/app directory....";
sudo cp -rp /home/shashank/jdk1.8.0_25 /opt/app;
sudo cp -rp /home/shashank/apache-tomcat-8.0.15 /opt/app;
echo "Exporting necessary variables......";
echo "Variables exported.";
echo "Checking Java & Tomcat versions.";
echo "Java is installed at `which java` Directory";
java -cp $CATALINA_HOME/lib/catalina.jar org.apache.catalina.util.ServerInfo;
echo "Starting Tomcat server"
echo "Removing unpacked tarballs from PWD.";
sudo rm -rf /home/shashank/jdk1.8.0_25;
sudo rm -rf /home/shashank/apache-tomcat-8.0.15;
echo "Installer tarballs not found in /home/shashank. Please make sure they exist there. Exiting installation process now."