Wednesday, 14 September 2011

MYSQL count email domains

In a previous job where clients supplied their own data, I had to be careful how of their email campaign went to a particular ESP/FQDN in one go.

This count gives an idea of how much of the email data is for what domains.

Code:   
SELECT SUBSTRING_INDEX(email, '@', -1) as Domain, count(email) as Total
FROM members
WHERE (email != '' AND email NOT NULL)
GROUP BY Domain
ORDER BY Total DESC
LIMIT 100;
   


MySQL query for geographical UK postcode selections

First, the basics.
Code:   
SELECT * FROM contacts WHERE postcode = 'MK3 6ZZ';
   
will match any records with a single postcode in Milton Keynes.

Code:   
SELECT * FROM contacts WHERE postcode LIKE 'MK3 %';
   
will match records with the postcode area MK3 in Milton Keynes. Care needs to be taken when using LIKE. Notice the space after the '3'.

Code:   
SELECT * FROM contacts WHERE postcode LIKE 'MK3%';
   
is not the same as the above. Notice that there is no space after the '3'. The would return records, not only in the MK3 postcode area but also, MK31 to MK39 (if they existed).

Code:   
SELECT * FROM contacts WHERE postcode REGEXP ('^E[1-9]');
   
will match any postcodes in the E1 to E9 (London) postcode areas.

As I will show, you can mix these examples to search for more than a single postcode area in one query. Useful if you want the query to return a larger geographical area.

If you want to target members/customers/recipients within a geographical location then here are a few snippets I have gleaned over the years.

Code:   
-- London
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('BR','CR','DA','EC','EN','HA','IG','KT','NW','RM','SE','SM','SW','TW','UB','WC','WD')
OR postcode REGEXP ('^E[1-9]')
OR postcode REGEXP ('^N[1-9]')
OR postcode REGEXP ('^W[1-9]'));

-- Scotland
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('KW','IV','AB','PH','DD','PA','FK','KY','ML','KA','EH','DG','TD')
OR postcode REGEXP ('^G[1-9]'));

-- England
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN
('BR','CR','DA','EC','EN','HA','IG','KT','NW','RM','SE','SM','SW','TW','UB','WC','WD','BN','CM','CT','GU','HP','LU','ME','MK',
'PO','RG','RH','SG','SL','SO','SS','TN','CB','CO','DE','DN','IP','LE','LN','NG','NR','PE','CV','DY','HR','NN','ST','TF','WR','WS',
'WV','DH','DL','HG','HU','LS','NE','SR','TS','WF','YO','BB','BD','BL','CA','CH','CW','FY','HD','HX','LA','OL','PR','SK','WA',
'WN','BA','BH','BS','DT','EX','GL','AL','PL','SN','SP','TA','TQ','TR','OX')
OR postcode REGEXP ('^E[1-9]')
OR postcode REGEXP ('^N[1-9]')
OR postcode REGEXP ('^W[1-9]')
OR postcode REGEXP ('^S[1-9]')
OR postcode REGEXP ('^B[1-9]')
OR postcode REGEXP ('^L[1-9]')
OR postcode REGEXP ('^M[1-9]'));

-- Wales
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('LL','SY','SA','LD','SA','CF','NP'));

-- Northern Ireland
SELECT COUNT(*)
FROM members
WHERE postcode LIKE 'BT%';

-- South East
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('BN','CM','CT','GU','HP','LU','ME','MK','PO','RG','RH','SG','SL','SO','SS','TN'));

-- East Midlands
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('CB','CO','DE','DN','IP','LE','LN','NG','NR','PE')
OR postcode REGEXP ('^S[1-9]'));

-- West Midlands
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('CV','DY','HR','NN','ST','TF','WR','WS','WV')
OR postcode REGEXP ('^B[1-9]'));

-- North East
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('DH','DL','HG','HU','LS','NE','SR','TS','WF','YO'));

-- North West
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('BB','BD','BL','CA','CH','CW','FY','HD','HX','LA','OL','PR','SK','WA','WN')
OR postcode REGEXP ('^L[1-9]%'
OR postcode REGEXP ('^M[1-9]%'));

-- South West
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,2) IN ('BA','BH','BS','DT','EX','GL','AL','PL','SN','SP','TA','TQ','TR','OX'));

I do use LIKE for small queries but REGEXP (Regular Expression) is better to avoid any ambiguity. LIKE 'MK1%' would be a mistake because it would select not only MK1 but, MK11, MK12, etc.

Say that you want to target UK members/customers/recipients within a radius of a postcode, how do you work it out? Well, there are websites that can help. Do a search for something like, 'Find UK Postcodes Inside a Radius' and you will find a few. Type in a postcode for the center of the search radio and then the mile/kilometers radius and then you can feed the results into an SQL query.

Example:
Code:   
SELECT COUNT(*)
FROM members
WHERE (SUBSTRING(postcode,1,4) IN ('MK1 ','MK17','MK2 ','MK3 ','MK4 ','MK5 ');
   



Saturday, 13 August 2011

HOW-TO: Convert .aac .m4a file to .mp3

You first extract the audio from the .mp4 or .m4a to .wav using faad. Then convert the .wav to .mp3

You need faad2 and lame installed.

Code:   
$ faad foobar.m4a       # to convert to wav
$ lame foobar.wav foobar.mp3          # to convert to mp3
   

Friday, 5 August 2011

HOW-TO: Email notification of SFTP file upload

Following on from my post, 'HOW-TO: Chrooted SFTP only access' I wanted to find a way to get an email notification of when a client had uploaded a file to my work's SFTP area. I had Sendmail already installed.

Each client has their own subdirectory under /home/sftp/ and I wanted to monitor them all and trigger an email when a new file was created.

Ok, I don't know how much use this will be to others but I am just going to throw down what I did to achieve what I wanted. These are personal notes and use them at your peril.

Code:   
sudo apt-get install iwatch
   

installs iWatch, written in Perl and based on inotify, a file change notification system, a kernel feature that allows applications to request the monitoring of a set of files against a list of events.

Code:   
sudo touch /etc/init.d/sftp-iwatch
   

creates an empty file. I edited the file and typed

Code:   
iwatch -r -e create -c "(w;ps -ef)|/home/david/file_notify_email.sh %f '$($(which date))'" /home/sftp >> /dev/null 2>&1 &
   

which basically executes a script /home/david/file_notify_email.sh when a new file is created under /home/sftp/.

/home/david/file_notify_email.sh
Code:   
#!/bin/bash
# 20110723 David Humble

FILE="$1"
CREATED="$2"
DATE=$($(which date))
FROM="aadfiler"
SUBJECT="File upload notification"
EMAIL="technical@xxx.co.uk"
EMAILMESSAGE="/var/upload.txt"

echo "Date: $DATE" > /var/upload.txt
echo "To: $EMAIL" >> /var/upload.txt
echo "From: $FROM">>/var/upload.txt
echo "Subject: $SUBJECT">>/var/upload.txt
echo "New file $FILE created at $CREATED on aadfiler SFTP Server.">>/var/upload.txt
echo ".">>/var/upload.txt

cat /var/upload.txt | /usr/sbin/sendmail -t
   


Code:   
sudo chmod +x /etc/init.d/sftp-iwatch
sudo chmod +x /home/david/file_notify_email.sh
   

makes both files executable.

Code:   
cd /etc/init.d
update-rc.d sftp-iwatch defaults
   

Adds sftp-iwatch to the servers startup processes.

now you can manually start sftp-iwatch by typing
Code:   
sudo /etc/init.d/sftp-iwatch
   


Now whenever a client uploads a file to any subdirectory of /home/sftp/ I get a notification thus:
Quote:   
Date: Wed Aug 3 16:27:02 BST 2011
To: technical@xxx.co.uk
From: aadfiler@xxx.local
Subject: File upload notification

New file /home/sftp/clientsname/upload/AAD013 GE eNewsletter Day 3.csv created at Sat Jul 23 13:36:37 BST 2011 on aadfiler SFTP Server.



References
http://iwatch.sourceforge.net/index.html

Categories: How-To, Linux, OS, Open-SSH, SFTP server, Chrooted SFTP only access
Tags: kde, ubuntu server, Linux, open-ssh, chroot, sftp, File upload email notification, iwatch, notify, dnotify

Tuesday, 14 June 2011

Copy PuTTY sessions to another computer

Copy PuTTY sessions (conn. profiles) to another computer

 
1. From Run or a command prompt, paste and execute the following:

Code:
regedit /e "%userprofile%\desktop\putty-registry.reg" HKEY_CURRENT_USER\Software\Simontatham


Simon Tatham is the author of PuTTY and thus the registry entry above! Bet you were worried that the above script ran something to mess with your Registry!?

2. This produces a file, putty-registry.reg on your desktop. Copy on to the computer you want the saved sessions added to and double click on it to execute. This adds the PuTTY registry to the computer's registry.

Done!

Thanks to Anton Perez for this. I copied here in case his page ever disappears and it helps my ageing grey-cells!

Sunday, 12 June 2011

Batch file to convert RAW to JPEG using dcraw

This is more a note for myself really, so that I can't quickly batch convert my Olympus E-500 .ORF (raw) images quickly to .jpg (JPEG) using dcraw and exiftool.

Using dcraw you can convert from most RAW formats to lesser formats, like JPEG.

Code:   
for i in *.ORF; do newname=$(echo $(basename "$i" ".ORF").jpg); dcraw -c -w -W -v -h "$i" | cjpeg -quality 95 -optimize -progressive > "$newname"; exiftool -overwrite_original -tagsFromFile "$i" "$newname"; done
   

Sunday, 5 June 2011

TIP: Installing XenServer an XenServer Tools on Ubuntu 10.04

Excellent XenServer on Ubuntu Server 10.04 How-to at
http://www.jansipke.nl/installing-xenserver-tools-on-ubuntu-10-04

I followed his clear and excellent how-to, to get Ubuntu Server going in paravirtualization mode. I in fact made the boot partition Ext3.

jansipke has a load of XenServer tips on his blog so check them out!