Convert a Tab separated to Comma separated file

If you want to replace some text within a file, that is too large to open in a GUI application then, this is when you fall back on the command line tools.

Sometimes I get a Tab separated (.tsv/.txt) file and I need to convert it to a Comma separated (.csv) file before I can import it into a database table.

On a Linux server:
Code:   

tr -s '\t' <input | tr '\t' ',' >output

   
Substitute 'input' for the original filename and 'output' for the desired output filename.

MySQL distinct count of data across multiple tables

You want to do a unique count of data across multiple tables.

Below is an example of getting a count of unique email addresses across two tables. Actually, I am looking for Females, aged 25 to 45 in the East Midlands, UK. I don't want duplicates because, I would not want to email the same person twice!

Code:   

SELECT COUNT(DISTINCT email) As East_Midlands
FROM (
SELECT email
FROM table_a
WHERE (gender = 'F' OR LOWER(title IN ('ms', 'miss', 'mrs')))
AND str_to_date(date, '%d/%m/%Y') BETWEEN '1967-01-01' AND '1987-08-13'
AND (SUBSTRING(postcode,1,2) IN ('CB','CO','DE','DN','IP','LE','LN','NG','NR','PE') OR postcode LIKE 'S[1-9]%')
UNION
SELECT email
FROM table_b
WHERE (gender = 'F' OR LOWER(title IN ('ms', 'miss', 'mrs')))
AND str_to_date(DOB, '%Y%m%d') BETWEEN '1967-01-01' AND '1987-08-13'
AND (SUBSTRING(postcode,1,2) IN ('CB','CO','DE','DN','IP','LE','LN','NG','NR','PE') OR postcode LIKE 'S[1-9]%')
) As SubQueryAlias;

   
I hope that makes sense. So, you have multiple selects, joined by the 'UNION', wrapped by the outer query which counts the distinct email addresses of the tables it embodies (as many as you want).

The 'As SubQueryAlias' at the end is needed to pull it together or, SQL will throw an error.

Thwart attacks to your SFTP server with DenyHosts

For automated blocking of attackers try denyhosts. DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

http://www.cyberciti.biz/faq/block-ssh-attacks-with-denyhosts/

ClamAV 97.4 error while loading shared libraries

I installed ClamAV 0.97.4 on my Kubuntu system today, the installation went OK and no error were shown but when I tried to run virus scanning I got this error message:

Quote:   

clamscan: error while loading shared libraries: libclamav.so.6: cannot open shared object file: No such file or directory

   


A quick browse and found that the following fixed this:
Code:   

sudo ldconfig

   

20 Linux System Monitoring Tools Every SysAdmin Should Know

Very good article on some essential monitoring tools.

http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

iptables: Blocking attacking IP addresses

Excellent article on using iptables to block an attacking IP address.

http://www.cyberciti.biz/faq/linux-iptables-drop/

I needed to block a known russian SSH/SFTP hacker.

Code:   

iptables -A INPUT -s 91.205.189.27/255.255.255.255 -j DROP

   


Even better for automated blocking of attackers is denyhosts. DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

http://www.cyberciti.biz/faq/block-ssh-attacks-with-denyhosts/

Copy table between two Databases

Code:   

CREATE TABLE DB2.new_copy LIKE DB1.master_copy;
INSERT INTO DB2.new_copy SELECT * FROM DB1.master_copy;

The first line creates the table using the same engine as the existing table and creates the primary keys and/or indexes.
The second line copies the data across.

If the original table has many keys then it may help to speed the INSERT if you turn off the keys using the following before the INSERT:

Code:   

ALTER TABLE new_table_name DISABLE KEYS;

   
And then after the INSERT:

Code:   

ALTER TABLE new_table_name ENABLE KEYS;

   
These two statements are not supported by all MySQL Engine types though. InnoDB being one of them in MySQL ver 5.x