linux BASH - replacing text in a file

If you want to replace some text within a file, that is too large to open in a GUI application then, this is when you fall back on the command line tools like sed.

But to make the change permanent, you have to add -i to the command:

Code:   

Syntax:
sed -i 's/<old_word>/<new_word>/g' <file>
sed 's/<old_word>/<new_word>/g' input.txt > output.txt

sed -i 's/Hello/hello/g' input.txt

   

Sed can also use regular expressions, so it's a very powerful tool.

The above searches for 'Hello' and replaces it with 'hello'.

Thanks to Eric_T for this which, I put here as a reminder to myself.

Code:   

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character
A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path "/nfs/apache/logs/rawlogs/access.log" with "__DOMAIN_LOG_FILE__":

Code:   

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"

## Escape path for sed using bash find and replace
_r1="${_r1//\//\\/}"

# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf

# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

Thanks to Vivek Gite
http://www.cyberciti.biz/faq/unix-linux-replace-string-words-in-many-files/

Very useful for replacing repetitive text string within a .csv file!

Convert a Tab separated to Comma separated file

If you want to replace some text within a file, that is too large to open in a GUI application then, this is when you fall back on the command line tools.

Sometimes I get a Tab separated (.tsv/.txt) file and I need to convert it to a Comma separated (.csv) file before I can import it into a database table.

On a Linux server:
Code:   

tr -s '\t' <input | tr '\t' ',' >output

   
Substitute 'input' for the original filename and 'output' for the desired output filename.

MySQL distinct count of data across multiple tables

You want to do a unique count of data across multiple tables.

Below is an example of getting a count of unique email addresses across two tables. Actually, I am looking for Females, aged 25 to 45 in the East Midlands, UK. I don't want duplicates because, I would not want to email the same person twice!

Code:   

SELECT COUNT(DISTINCT email) As East_Midlands
FROM (
SELECT email
FROM table_a
WHERE (gender = 'F' OR LOWER(title IN ('ms', 'miss', 'mrs')))
AND str_to_date(date, '%d/%m/%Y') BETWEEN '1967-01-01' AND '1987-08-13'
AND (SUBSTRING(postcode,1,2) IN ('CB','CO','DE','DN','IP','LE','LN','NG','NR','PE') OR postcode LIKE 'S[1-9]%')
UNION
SELECT email
FROM table_b
WHERE (gender = 'F' OR LOWER(title IN ('ms', 'miss', 'mrs')))
AND str_to_date(DOB, '%Y%m%d') BETWEEN '1967-01-01' AND '1987-08-13'
AND (SUBSTRING(postcode,1,2) IN ('CB','CO','DE','DN','IP','LE','LN','NG','NR','PE') OR postcode LIKE 'S[1-9]%')
) As SubQueryAlias;

   
I hope that makes sense. So, you have multiple selects, joined by the 'UNION', wrapped by the outer query which counts the distinct email addresses of the tables it embodies (as many as you want).

The 'As SubQueryAlias' at the end is needed to pull it together or, SQL will throw an error.

Thwart attacks to your SFTP server with DenyHosts

For automated blocking of attackers try denyhosts. DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

http://www.cyberciti.biz/faq/block-ssh-attacks-with-denyhosts/

ClamAV 97.4 error while loading shared libraries

I installed ClamAV 0.97.4 on my Kubuntu system today, the installation went OK and no error were shown but when I tried to run virus scanning I got this error message:

Quote:   

clamscan: error while loading shared libraries: libclamav.so.6: cannot open shared object file: No such file or directory

   


A quick browse and found that the following fixed this:
Code:   

sudo ldconfig

   

20 Linux System Monitoring Tools Every SysAdmin Should Know

Very good article on some essential monitoring tools.

http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

iptables: Blocking attacking IP addresses

Excellent article on using iptables to block an attacking IP address.

http://www.cyberciti.biz/faq/linux-iptables-drop/

I needed to block a known russian SSH/SFTP hacker.

Code:   

iptables -A INPUT -s 91.205.189.27/255.255.255.255 -j DROP

   


Even better for automated blocking of attackers is denyhosts. DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

http://www.cyberciti.biz/faq/block-ssh-attacks-with-denyhosts/