As a SysAdmin &/or Developer, going through the log files or code is a daily chore. We do it hundreds of times everyday while at work. Some of us do it even when we are not at work 😉
Many a time, we need to search for a particular piece of text/string in either the log-files or the code. There are many text-editors available that you can use to drill into log-files or your code. Some editors also allow you to search inside directories.
But, if you want a simple, command-line utility that does the job for you, I have written a Bash Shell script that allows you to search for your string inside any given directory. It also counts the total occurrences and shows the line-number for those entries.
My script takes care of the OS (Tested on Mac, Ubuntu & RedHat) & performs installation of a small utility called figlet (according the OS) which I have used to produce an ASCII art for my script. You can press “n” when prompted, if you don’t want to install or use it.
You can download the script from my GitHub gist.
Below is the script in action (executed on Ubuntu).
If you don’t specify a directory, the script exits.
Error!! Please enter absolute path of the directory inside which you want to perform the search. Run this utility as ./textinfile /path/to/directory. Prefix sudo if needed.
Script after specifying a valid directory.
root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache2/
Detecting OS of your machine..
Your OS is : - Ubuntu.
Checking figlet insallation...
This utility requires a package called figlet in order to display a cool ASCII art ;-) Do you want to install it? Its a one time process only. You can safely ignore this insallation. Press y to install or n to cancel.
You chose to install figlet. Installing it for your OS now :-)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/190 kB of archives.
After this operation, 744 kB of additional disk space will be used.
Selecting previously unselected package figlet.
(Reading database ... 1548998 files and directories currently installed.)
Preparing to unpack .../figlet_2.2.5-2_amd64.deb ...
Unpacking figlet (2.2.5-2) ...
Processing triggers for man-db (22.214.171.124-1ubuntu1) ...
Setting up figlet (2.2.5-2) ...
update-alternatives: using /usr/bin/figlet-figlet to provide /usr/bin/figlet (figlet) in auto mode
_____ _ ___ _____ _ _
|_ _|____ _| |_|_ _|_ __ | ___(_) | ___
| |/ _ \ \/ / __|| || '_ \| |_ | | |/ _ \
| | __/> <| |_ | || | | | _| | | | __/
|_|\___/_/\_\\__|___|_| |_|_| |_|_|\___|
Welcome to TextInFile! A cute, little utility that allows you to search for the text inside directories from the command-line :-)
Enter the string that you want to search inside /var/log/apache2/...GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1
Searching for GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 in /var/log/apache2/...
GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1 was found 2 time(s) inside /var/log/apache2/.
Below is your Search Result :-)
/var/log/apache2/access_reverse_proxy.log.1:7:192.168.0.50 - - [25/Jun/2017:05:35:51 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://192.168.0.51/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"
/var/log/apache2/access_reverse_proxy.log.1:2807:10.0.2.2 - - [27/Jun/2017:03:44:31 +0200] "GET /static/4d441666/scripts/yui/menu/assets/skins/sam/menu.css HTTP/1.1" 200 1654 "http://localhost:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
This utility took 84 seconds to run(including the time you took to type/paste your string).
That's all folks! Hope you liked TextInFile ;-)
If you provide an incorrect directory, the script shows an appropriate error & then exits.
root@shashank-dbserver:/home/shashank/Desktop# ./textinfile.sh /var/log/apache
The directory /var/log/apache that you specified does't exist. Please choose the correct directory.
At times there are needs to setup multiple PostgreSQL instances. I have explained you how to install PostgreSQL & how to automate its installation on Linux. This time I will show you how to automate PostgreSQL instances creation using a shell-script. Instances are PostgreSQL installations to different directories running on different ports.
Below is the script that I wrote to automate this task 🙂 It will prompt you to enter the location & port.
#PostgreSQL instance creation script version 1.0
#Author : - Shashank Srivastava
echo "Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path."
ls -lf /opt | grep -i postgre | sort
echo -n "Enter the absolute path for this instance. "
echo "You chose $ans. Below are the Postgre ports currently in use. Please choose a unique port when prompted."
netstat -tulnp | grep postgre
echo -n "Enter port for this new instance to run on. "
mkdir -p $ans
mkdir -p $ans/data
chown -R postgres $ans/data
su - postgres -c "$ans/bin/initdb -D $ans/data"
su - postgres -c "cp -p $ans/data/postgresql.conf $ans/data/postgresql.conf.bkp"
sed -i "s/#port = 5432/port = $PORTNUM/g" $ans/data/postgresql.conf
su - postgres -c "$ans/bin/pg_ctl -D $ans/data -l logfile start"
echo "Checking PostgreSQL verison..."
echo "Postgre instance is running on port $PORTNUM"
netstat -tulnp | grep $PORTNUM
echo "Installation Successful."
Here’s the script in action 😉
[root@server shashank]# sh postgre_instance_setup.sh
Welcome to PostgreSQL instance creation script. Below instances are already created. Use a unique path.
Enter the absolute path for this instance. abc
You chose abc. Below are the Postgre ports currently in use. Please choose a unique port when prompted.
tcp 0 0 127.0.0.1:5437 0.0.0.0:* LISTEN 785/postgres
tcp 0 0 127.0.0.1:5438 0.0.0.0:* LISTEN 8863/postgres
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 6723/postgres
tcp 0 0 0.0.0.0:5433 0.0.0.0:* LISTEN 51663/postgres
tcp 0 0 0.0.0.0:5434 0.0.0.0:* LISTEN 35999/postgres
tcp 0 0 127.0.0.1:5435 0.0.0.0:* LISTEN 48190/postgres
tcp 0 0 127.0.0.1:5436 0.0.0.0:* LISTEN 57141/postgres
tcp 0 0 ::1:5437 :::* LISTEN 785/postgres
tcp 0 0 ::1:5438 :::* LISTEN 8863/postgres
tcp 0 0 :::5432 :::* LISTEN 6723/postgres
tcp 0 0 :::5433 :::* LISTEN 51663/postgres
tcp 0 0 :::5434 :::* LISTEN 35999/postgres
tcp 0 0 ::1:5435 :::* LISTEN 48190/postgres
tcp 0 0 ::1:5436 :::* LISTEN 57141/postgres
Enter port for this new instance to run on.
There are times when we need to reset HTTP user passwords. One of the common examples is Subversion (that uses Apache web front-end) users. There can be very many users who want to have their passwords reset. This can be very boring & tedious 😉 So, below is how I simplified it by writing a shell script using
expect 🙂 If you recall, I had used
expect earlier as well for this & this. Please read the script carefully. Its pretty self-explanatory 🙂
#Subversion password reset script
#Author : Shashank Srivastava
while read user; do
/usr/bin/expect << EOF
#/etc/svn-users is a flat-file that stores users & their encrypted passwords.
spawn htpasswd -m /etc/svn-users $user
puts "$user added"
expect "?New password: \r"
expect "?Re-type new password: \r"
#/home/shashank/svn-users.txt is a text-file that stores the user-names
So, here is the output of my script 🙂
[root@server tcs_admin]# sh scripts/svnusers_pw_reset.sh
spawn htpasswd -m /etc/svn-users user1
Re-type new password:
Updating password for user user1
spawn htpasswd -m /etc/svn-users user2
Re-type new password:
Updating password for user user2
This post taught you how to install PostgreSQL Database on Linux using source code. That approach is good when number of servers is less. But what if it is to be installed on a large number of servers? 😉 Such as in a clustered environment? Here’s how to achieve it using a shell script. 🙂
Better enable password-less SSH from your master server/jumpbox to all destination servers to avoid typing passwords. Then copy the installer tarball from jumpbox to all servers using below command.
for hst in `cat /home/shashank/hosts.txt`; do scp $hst:/home/shashank; done
Now that installer has been copied to destination servers, its time to run the script from jumpbox to install it on all destination servers in one go 🙂 Issue below command to execute automation script & install PostgreSQL on all our target servers.
for hst in `cat /home/shashank/hosts.txt`; do ssh -t $hst 'bash -s' < /home/shashank/postgre_installer.sh; done
Below is the installer script.
#PostgreSQL Automation Installer Script version 1
#Author - Shashank Srivastava
echo "Checking availability of pre-required packages"
sudo rpm -q readline-devel.x86_64
if [ $? != 0 ]
echo "Readline package not found. Installing it...."
sudo yum install readline-devel.x86_64 -y
sudo rpm -q zlib-devel-1.2.3-29.el6.x86_64
if [ $? != 0 ]
echo "zlib package not found. Installing it...."
sudo yum install zlib-devel.x86_64 -y
echo "Unpacking PostgreSQL tarball..."
sudo tar -zxf /home/shashank/postgresql-9.3.5.tar.gz
echo "Tarball unpacked. Creating destination directory for Postgre...."
sudo mkdir -p /opt/PostgreSQL/9.3
echo "Destination directory created. Building installer from source coude...."
sudo ./configure --prefix=/opt/PostgreSQL/9.3
echo "Installer built. Installing it now...."
sudo make install
id -a postgres
if [ $? != 0 ]
echo "postgres user not found. Adding it...."
sudo useradd postgres
echo "postgres user already exists. Proceeding to next step...."
echo "Configuring PostgreSQL...."
sudo mkdir -p /opt/PostgreSQL/9.3/data
sudo chown postgres /opt/PostgreSQL/9.3/data
echo "Defining data directory for Postgre & starting it...."
sudo su - postgres -c "/opt/PostgreSQL/9.3/bin/initdb /opt/PostgreSQL/9.3/data/; sleep 2s; /opt/PostgreSQL/9.3/bin/pg_ctl -D /opt/PostgreSQL/9.3/data/ -l logfile start"
echo "Checking version...."
echo "Script executed successfully!"
My work involves installing & managing a lot of Linux servers. At times I have to do same work on numerous servers. One such task is installing Tomcat. In these posts I showed you how to install Apache Tomcat & how to check its version on Linux servers.
In this post, I will show you how you can automate the installation process across any number of servers using a shell script 🙂
My approach is to : –
- Setup password-less SSH on all servers from one jumpbox machine. Jumpbox is a server from which you will execute this script.
- Create a list of servers in a text file. All servers are typed line by line.
- Copy Java & Tomcat installers to all servers in one go from this jumpbox.
- And finally execute the script.
Below are the commands & script through which I accomplished this. All of these are self-explanatory 😉
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/jdk-8u25-linux-x64.tar.gz $hst:/home/shashank; done
for hst in `cat /home/shashank/hosts.txt`; do scp ~shashank/apache-tomcat-8.0.15.tar.gz $hst:/home/shashank; done
What it does above? For loop reads the contents of text file (server IPs) line by line & copies tarballs there.
for hst in `cat /home/shashank/hosts.txt`; do ssh $hst -t 'bash -s' < ~shashank/tomcat_installer.sh; done
The trick above is to use ‘bash -s’. It will accept the script as an argument & execute it. After execution, it will move to next server. Below is my installer script.
Do let me know if it was helpful or not 🙂 I will come up with some more posts soon 🙂
#Apache Tomcat Installer Script
#Author : Shashank Srivastava
echo "Logged into `hostname`. Installing here."
#Checking if installer tarballs are present or not. If they are not found, script will print error message & quit.
if [ -f /home/shashank/jdk-8u25-linux-x64.tar.gz ] && [ -f /home/shashank/apache-tomcat-8.0.15.tar.gz ]
echo "Unpacking Java installer tarball.......";
#waiting for 2 seconds to show you what is being done.
sudo tar -xzvf /home/shashank/jdk-8u25-linux-x64.tar.gz;
echo "Java tarball unpacked.";
echo "Unpacking Tomcat installer tarball.......";
sudo tar -xzvf /home/shashank/apache-tomcat-8.0.15.tar.gz;
echo "Tomcat tarball unpacked.";
echo "Installing Java & Tomcat to /opt/app directory....";
sudo cp -rp /home/shashank/jdk1.8.0_25 /opt/app;
sudo cp -rp /home/shashank/apache-tomcat-8.0.15 /opt/app;
echo "Exporting necessary variables......";
echo "Variables exported.";
echo "Checking Java & Tomcat versions.";
echo "Java is installed at `which java` Directory";
java -cp $CATALINA_HOME/lib/catalina.jar org.apache.catalina.util.ServerInfo;
echo "Starting Tomcat server"
echo "Removing unpacked tarballs from PWD.";
sudo rm -rf /home/shashank/jdk1.8.0_25;
sudo rm -rf /home/shashank/apache-tomcat-8.0.15;
echo "Installer tarballs not found in /home/shashank. Please make sure they exist there. Exiting installation process now."
If you have tried running Oracle sqlplus (of course through a script) in a cron job, then you might have encountered this error: –
sqlplus: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory
What happens is that, despite defining all the environment variables in /etc/profile or bash_profile, cron doesn’t recognise that & throws above error. Close inspection after debugging the script took me to a conclusion that sqlplus is not able to connect to database, because its not finding its shared library.
If you have encountered this issue, here is how to fix it? 😉
Make sure you have exported required variables. Below are my variables in my .bash_profile. Change yours accordingly.
Make sure you source your .bash_profile before executing the script in cron. Edit your script & add below line at the top.
Now try executing the script in cron. It will be successful 😉 cron will first source the bash_profile, initialise all the variables & then execute the script 🙂
Ever found yourself in a situation to add multiple users to a single group? 😉 I know you have 😉
Just issue below command as root & it will add all the required users to a group 😉
for user in user1 user2 user3 user4 user5; do usermod -a -G group_name $user; done
If the number of users is too long, then create a text file containing all the users to be added (make sure user names are white-space separated, just like above example) & then pass this text file in
for loop. Like this 😉
for user in `cat ~shashank/users_list.txt`; do usermod -a -G group_name $user; done