Check it out here…

http://www.2brightsparks.com/welcome/backup/freeware.html

Check it out here…

http://www.slysoft.com/en/virtual-clonedrive.html

Check it out here…

http://www.faststone.org/FSViewerDetail.htm

Since my old work machine is a Windows machine I actually used Cygwin for it’s ssh, sshpass (note: you will have to compile this one yourself since there is no port) and rsync. It worked pretty well.

Rsync Command

So here is the rsync command that I ended up with. I’ve left the long command line switches in here for clarity…

rsync --verbose  --progress --stats --compress --recursive --delete --times --perms --links --rsh='sshpass -p PASSWORD ssh -l USERNAME' /local/dir/ ssh.remote.com:/remote/dir/

So, lets explain those options…

PASSWORD     : Swap out for your ssh password on the remote host
USERNAME     : Swap out for your ssh username on the remote host
/local/dir/  : Change to appropriate local directory
/remote/dir/ : Change to appropriate remote directory

--verbose    : Give us lots of information
--progress   : And some information on the progress during transfer (useful on large files)
--stats      : And some stats since I am a stats junkie
--compress   : Compress the file data during the transfer
--recursive  : Recurse into all of the directories and process them also
--delete     : Delete files from the destination dir that do not exist amymore
--times      : Preserve the modification times for the file when placing it on the remote server
--perms      : Preserve the permissions for the file when placing it on the remote server
--links      : Copy the symlinks as symlinks and do not resolve them

I kicked this off overnight first time and then occasionally during the last week I worked. Once on the last day and then I formatted my machine. If I were running a *nix machine I would just have added it to the cron. Jobs a good ‘un.

Code

We want to mock something of medium size (for a web service response) and so we use the following code…

<?php

for($i = 0; $i < 1000; $i++) {
    $data[count($data)] = array("title" => "some title $i", "author" => "author number $i", "section" => "filed under section $i");
}

?>

Output

For my purposes here I am going to output in XML and also in JSON…

// JSON output
header('Content-type: application/json');
echo json_encode($data);

...

// XML output
header('Content-type: text/xml');
echo "<data>";
foreach($data as $item) {
    echo "<item>";
        echo "<title>{$item["title"]}</title>";
        echo "<author>{$item["author"]}</author>";
        echo "<section>{$item["section"]}</section>";
    echo "</item>";
}
echo "</data>";

Metrics

So bear in mind that all of this calling is from my local machine without anything unusual running and nothing scheduled to kick off in the middle of the capture. I guess this would be the best case scenario since it will essentially cut out any network related anomalies. For most load and stress testing I use NeoLoad from these guys…

http://www.neotys.com/

It seems to do the job just nicely. I highly recommend you download the 30 day trial and check it out. So, lets get testing and see what we can figure out (though I am sure that you are guessing the result already).

The Test

Very simple, get the page from the server 1000 times (roughly an hour’s worth of an occasional service) with 10 concurrent users. We can see how long it takes and also how much data is transferred.

Results

XML

Time  : 5:02 
Size  : 112.57 MB

JSON

Time  : 5:01 
Size  : 87.67 MB

Projecting Forward

Does that seem like a lot or a little to you? Since the time is pretty much the same (might be more of a difference with a real host) lets have a look and see what you might save in terms of bandwidth course of a month…

XML   : 112.57 MB * 24 hours * 30 days = 81050.4 MB
JSON  : 87.67 MB  * 24 hours * 30 days = 63122.4 MB

So, about 3/4 of the bandwidth cost for writing less code. Sounds like a good deal to me! Check out the code below and have a play around yourselves.

Download: JSON vs XML, some metrics

There are many reasons for finding out where the user is located physically on this planet of ours but I guess there are two main reasons…

• You want to track the user somehow in realtime and gain some knowledge based on this.
• You have regionalised your site and want to redirect the user to the most appropriate area to maximise their experience. This can be as simple as a currency change to a totally different site.

Download

First thing to do is to get the files from MaxMind. The free ones can be found here…

http://www.maxmind.com/app/geolitecity

Bear in mind that the free version is slightly less accurate than the paid for version. It’s such a valuable resource though that I would say go for the paid version if you are going to use this in any commercial way.

Decompress

If you are on a Linux machine then the .gz is going to be familiar to you and you can just run a ‘gzip -d ‘ on it. If your cruel company is forcing you to use a Windows machine that has no inbuilt support for one of the most widely used compression formats in the world today then your next stop is going to be downloading and installing 7-Zip…

http://www.7-zip.org/

This is an excellent little archive utility and I have already written about it here.

Download the API code for PHP

For our purposes here I am just going to use the Pure PHP API though if you have complete control of the machine I would recommend you check out the other options also. The Pure PHP API can be downloaded here…

http://www.maxmind.com/app/php

First steps

Decompress the API code and search for the geoip.inc, geoipcity.inc and the geoipregionvars.inc files. They will be in the root folder with all of the samples. Copy them into the same folder as the GeoLiteCity.dat file which should already be decomparessed.

Code

MaxMind have made this a real no brainer with with the API code that they provide…

<?php

include 'geoipcity.inc';
include 'geoipregionvars.php';

$gi = geoip_open("GeoLiteCity.dat",GEOIP_STANDARD);

$info = geoip_record_by_addr($gi,"130.167.0.0");

print_r($info);

geoip_close($gi);

?>

What’s the incoming client IP address to a website?

You will note from the above that I have just statically set the IP address to be 130.167.0.0 (a famous IP address range). This code really becomes useful when you place it on your site. So how are we going to get the IP address for the visiting user? Easy, PHP already has it in the $_SERVER['REMOTE_ADDR'] superglobal. Substitute that in for the static IP address and play around a bit. It can be interesting to see this information happening in (near) real time.

You might want to redirect to somewhere now based on the country / city…

header("Location: http://www.redirect.com/");

...or...

echo "<script>location.href='http://www.redirect.com'</script>"; // If you have already made output

Download: Get the physical location of an IP address with the free GeoIP / GeoLite library

http://www.7-zip.org/

I rarely even open it and just use the right click Explorer extensions. The 7z compression format comes highly recommended also for archive since it used LZMA / LZMA2 and can help you make the most of your backup space.

# Give a 404 when any of the .svn directories are accessed
RedirectMatch 404 "/.svn"

# If they are going to show the directory listing in the browser then filter out all of the .svn folders
<IfModule autoindex_module>
    IndexIgnore .svn
</IfModule>

…and for a bonus while you are at it I would also deny them from browsing the wp-config.php file…

# Deny browse to the wp-config.php file
RedirectMatch 404 "/wp-config.php"

You are now a little bit more secure

Enjoy…

http://virtuawin.sourceforge.net/

Step 1

First thing you are going to have to have a quick scan of are the Google Analytics reporting dimensions…

https://developers.google.com/analytics/devguides/reporting/core/dimsmets

This is a listing of the ways you can slice the data that Google Analytics captures on each page hit. It’s important to note that not all combinations are valid though there are so many that are you’re not going to be stuck for some metrics data!

Step 2

Next on the list is to go and grab a copy of the Google Analytics PHP Interface.

Step 3

I am going to assume for this step that you have an account already with Google Analytics and that you are monitoring something. You need to get the profileId for the specific domain. Fortunately this is pretty easy within the code…

<?php

require 'config/ga_define.php';
require 'api/gapi.class.php';

$ga = new gapi(ga_email,ga_password);

$ga->requestAccountData();
print_r($ga);

?>

View source if it looks like a muddled mess. Make sure you change the values in ga_define.php first (check for the download at the end of the post). This just gives you a dump of all the accounts (plus some more information) and their associated data. Pick the profileId for the domain you want to check the page from.

Step 4

So now that we have the profileId we can query Google Analytics for the page that is under that profile…

<?php

require 'config/ga_define.php';
require 'api/gapi.class.php';

$ga = new gapi(ga_email,ga_password);

$filter = 'pagePath == /2012/04/combination-rss-feed-for-wordpress-multisite/';

$ga->requestReportData(ga_profile_id,array('pagePath'),array('pageviews','visits'),null,$filter);
print_r($ga->getMetrics());

?>

Have some coffee. You deserve it. Actually hold on, you don’t really since letting this code run wild on any server that doesn’t need absolutely up to the minute pageview data (and 99% wont) is bad practice. It should be cached somewhere. This is where Memcached comes in.

Step 5

Installing Memcached is beyond the scope of this post but suffice to say you should be up and running in minutes. Please refer to the documentation…

http://code.google.com/p/memcached/wiki/NewStart

It will involve including a new PHP extension in the php.ini file and bouncing Apache.

Step 6

Wrap your call to Google Analytics in a Memcache object…

<?php

require 'config/ga_define.php';
require 'api/gapi.class.php';

$memcache = new Memcache;
$memcache->connect('localhost', 11211);

function get_key($key) {
    global $memcache;

    return ($memcache) ? $memcache->get($key) : false;
}

function set_key($key, $object, $timeout = 3600) {
    global $memcache;

    return ($memcache) ? $memcache->set($key, $object, 0, $timeout) : false;
}

$ga = new gapi(ga_email,ga_password);

$page_path = '/2012/04/combination-rss-feed-for-wordpress-multisite/';
$filter = 'pagePath == '.$page_path;

$cache_verison = get_key(md5($page_path));
if ($cache_version)
    print_r($cache_version);
else {
    $ga->requestReportData(ga_profile_id,array('pagePath'),array('pageviews','visits'),null,$filter);
    print_r($ga->getMetrics());

    set_key(md5($page_path), $ga->getMetrics());
}

?>

Done! Now you can have that cup of coffee. All code can be downloaded here…

Download: Google Analytics API example with Memcached

This is the content of the config.sh

#!/bin/bash

PROD_USERNAME=produser
PROD_PASSWORD=prodpass
PROD_DBNAME=proddb
PROD_URL=www.something.com

TEST_USERNAME=testsuer
TEST_PASSWORD=testpass
TEST_DBNAME=testdb
TEST_URL=test.something.com

OUTPUT_DIR=/home/user/sql

This is the content of the dump.sh

#!/bin/bash

if [ -z $OUTPUT_DIR ]; then
    source ./config.sh
fi

OUTPUT_FILENAME=$OUTPUT_DIR/<code>date +'%Y%m'</code>/<code>date +'%Y%m%d'</code>_blogs_db.sql

# Make sure that the directory is in place before we make the output
mkdir $OUTPUT_DIR/<code>date +'%Y%m'</code> 1>/dev/null 2>&1

# Dump the database
echo Dumping $PROD_DBNAME to $OUTPUT_FILENAME.gz ...
mysqldump -c --skip-extended-insert -u $PROD_USERNAME -p$PROD_PASSWORD $PROD_DBNAME > $OUTPUT_FILENAME
gzip -9 $OUTPUT_FILENAME

This is the content of the refresh_test.sh

#!/bin/bash

source ./config.sh

OUTPUT_FILENAME=$OUTPUT_DIR/<code>date +'%Y%m'</code>/<code>date +'%Y%m%d'</code>_blogs-test_db.sql
OUTPUT_FILENAME_BACKUP=$OUTPUT_DIR/<code>date +'%Y%m'</code>/<code>date +'%Y%m%d'</code>_blogs-test_backup_db.sql

# Make sure that the directory is in place before we make the output
mkdir $OUTPUT_DIR/<code>date +'%Y%m'</code> 1>/dev/null 2>&1

# Dump the file changing the url through the pipe
echo Dumping and converting $PROD_DBNAME to $OUTPUT_FILENAME ...
mysqldump -c --skip-extended-insert -u $PROD_USERNAME -p$PROD_PASSWORD $PROD_DBNAME | sed -e "s/$PROD_URL/$TEST_URL/g" > $OUTPUT_FILENAME

# Create a backup of the test database which you are going to nuke in the next paragraph
echo Backing up $TEST_DBNAME to $OUTPUT_FILENAME_BACKUP ...
mysqldump -c --skip-extended-insert -u $TEST_USERNAME -p$TEST_PASSWORD $TEST_DBNAME > $OUTPUT_FILENAME_BACKUP

# Wipe the contents of the test with the new changed prod
echo Reading and overwriting $TEST_DBNAME with $OUTPUT_FILENAME ...
mysql -u $TEST_USERNAME -p$TEST_PASSWORD $TEST_DBNAME < $OUTPUT_FILENAME

So, now I just cron a dump.sh nightly (using rsync later to shift it off to another machine) and also cron a refresh_test.sh nightly so that I am always testing with some fresh data. Perfect!

Download: WordPress database dump and refresh scripts

if ($_SERVER['DOCUMENT_ROOT'] == 'C:/xampp/htdocs/MyBlogDirectory') { // Localhost development
    define('DB_NAME', 'my-blog-local');
    define('DB_USER', 'user');
    define('DB_PASSWORD', 'pass');
}
else { // Production
    define('DB_NAME', 'my-blog-local');
    define('DB_USER', 'user');
    define('DB_PASSWORD', 'pass');
}

And further down the file if using WordPress multisite…

if ($_SERVER['DOCUMENT_ROOT'] == 'C:/xampp/htdocs/MyBlogDirectory') { // Localhost development
    $base = '/';
    define('DOMAIN_CURRENT_SITE', 'my-blog-local-url'); // Whatever this is it must be resolvable, more than likely you have something in your hosts file
    define('PATH_CURRENT_SITE', '/');
    define('SITE_ID_CURRENT_SITE', 1);
    define('BLOG_ID_CURRENT_SITE', 1);
}
else {  // Production
    $base = '/';
    define('DOMAIN_CURRENT_SITE', 'my.blog.com'); // This will be the url for the production site
    define('PATH_CURRENT_SITE', '/');
    define('SITE_ID_CURRENT_SITE', 1);
    define('BLOG_ID_CURRENT_SITE', 1);
}

There are two things to note about my solution.

• It uses the excellent SimplePie library
• It also is aware of the JF3 Maintenance Redirect status for the site. An excellent plugin. I check the site is not in maintenance mode in line 43 of index.php

Be sure to update the heading information for the RSS file (starting from line 57 of index.php) before putting this into production. Just drop it into your root folder and then you should be able to see all feeds via http://www.your-url-here.com/feed-combine/

Download: WordPress multisite feed combination script

diff -rq dirA dirB

It will give you some kind of output like the following…

[X]$ diff -rq dirA dirB
Files dirA/differs.txt and dirB/differs.txt differ
Only in dirA: only_in_a.txt
Only in dirB: only_in_b.txt
[X]$

To resolve the differences check out rsync for one way sync. Be sure to test out any command with the -n flag. This does a “dry run” of rsync showing possible changes without actually making them.

http://en.wikipedia.org/wiki/Rsync

For two way sync try unison.

http://en.wikipedia.org/wiki/Unison_(file_synchronizer)

First of all, you need to download the log4net binaries…

http://logging.apache.org/log4net/download.html

…unpack and then include the log4net.dll file (found in the bin\net\[framework]\release). Do a test build. If you get an error complaining about System.Web then is it more than likely that you are using one of the client profile frameworks. Switch to one of the complete frameworks.

Step 2

I prefer to have the configuration in the App.config file for my console apps (there are options to have it exist externally) and so we shall start with that…

<?xml version="1.0"?>
<configuration>
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
  </configSections>

  <connectionStrings>
    <add name="DB" providerName="System.Data.SqlClient" connectionString="CONNECTION"/>
  </connectionStrings>
  <appSettings>
    <add key="KEY" value="VALUE"/>
  </appSettings>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
  </startup>
  
  <log4net debug="true">
    <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
      </layout>
    </appender>
    <appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
      <file value="C:\temp\Log.txt" />
      <appendToFile value="true" />
      <rollingStyle value="Size" />
      <maxSizeRollBackups value="10" />
      <maximumFileSize value="100MB" />
      <staticLogFileName value="true" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
      </layout>
    </appender>
    <root>
      <appender-ref ref="ConsoleAppender" />
      <appender-ref ref="RollingLogFileAppender" />
    </root>
  </log4net>
</configuration>

For this example I have decided to include a Console log and also a RollingLogFile log, written to at the same time with the same information. My default setup.

Step 3

Add the namespace…

using log4net;

In your code (usually just inside the class declaration for maximum scope) include the following…

private static readonly log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

Then it is time to write something to the log…

if (log.IsDebugEnabled)
    log.Debug("TICK");

if (log.IsWarnEnabled)
    log.Warn("TOCK");

if (log.IsErrorEnabled)
    log.Error("POP");

Step 4

Having scratched my head for a bit wondering why everything was not working at this stage (everything I could find said that it should) I then realised that this…

[assembly: log4net.Config.XmlConfigurator()]

Should be added to the AssemblyInfo.cs file. Hey Presto! Everything working and logging nicely!

public class SortFiles
{
    public class CompareFileByDate : IComparer
    {
        int IComparer.Compare(Object a, Object b)
        {
            FileInfo fia = new FileInfo((string)a);
            FileInfo fib = new FileInfo((string)b);

            DateTime cta = fia.CreationTime;
            DateTime ctb = fib.CreationTime;

            return DateTime.Compare(cta, ctb);
        }
    }

    public class CompareFileByTimestampInternal : IComparer
    {
        int IComparer.Compare(Object a, Object b)
        {
            XElement EventA = XDocument.Load((string)a).Root.Element("timestamp");
            DateTime cta = DateTime.Parse(EventA.Value);

            XElement EventB = XDocument.Load((string)b).Root.Element("timestamp");
            DateTime ctb = DateTime.Parse(EventB.Value);

            return DateTime.Compare(cta, ctb);
        }
    }
}

Calling the code is no different than the usual methods…

string[] files = System.IO.Directory.GetFiles(Some.Path, "*.xml");
IComparer fileComparer = new SortFiles.CompareFileByTimestampInternal();
Array.Sort(files, fileComparer);

All you need to do is alter your web.config…

<configuration>
  <system.net>
    <defaultProxy>
      <proxy usesystemdefault="False"
        bypassonlocal="True" 
        proxyaddress="http://127.0.0.1:8888" />
    </defaultProxy>
  </system.net>
</configuration>

…and now you will see everything pass through Fiddler. Click on the specific call and start your investigations.

iftop

yum install iftop

As the name might suggest to you ‘top’ but for interfaces (network interfaces). The simplest layout of them all – a monochrome readout telling you all of the hosts connected to the current interface (or another interface, use the -i option on the command line), how fast they are pulling information from the machine and the overall transmit and receive. Use this one if you have a site with some traffic and you want to see who are the biggest bandwidth hogs connected. This can be a useful one for debugging a site if it gets slow periodically (can be cron / load based or it can also be bot / hits based).

http://www.ex-parrot.com/pdw/iftop/

iptraf

yum install iptraf

This one is a little more involved than iptop. A full blown network statistics utility. If you have been using tcpdump in the past then you might want to give this one a shot also. You can monitor (in glorious console colour) traffic to the machine, interface statistics, statistical breakdowns and more. Add to this a really nice filter (with IP as one of the options) and you are onto a winner. Use this one if you want to get some more detail or check out traffic from a specific IP.

http://iptraf.seul.org/

apachetop

yum install apachetop

Whilst Google Analytics and other packages have made collecting information about your sites visitors super simple it is nice sometime to check out the hits as you are getting them. This is particularly interesting and useful in the optimisation process for any site. You can see what’s being called and in near realtime since apachetop keeps a track of all your logfiles. Click on the ‘h’ key whilst in there and you will get to see some interesting options such as how to get some detailed information about any of the requests. One of the most useful options is ‘n’ to switch from the bytes statistic to the HTTP return code value. Great for seeing where those 2xx, 3xx, 4xx and 5xx requests are coming from. Use this one also to check out what’s hot in realtime on your site, what pages are transmitting most frequently and which assets might be worth optimising.

http://code.google.com/p/apachetop/

Personally, there are many things that I have had exposure to in university and also along my career path which would be fascinating to delve a bit deeper into. Neural networks / AI have always been one of these subjects for me. I for one can’t wait to see what they will offer!

Check out this link to Forbes for further detail…

M.I.T. Game Changer

http://wordpress.org/extend/plugins/jf3-maintenance-mode/

It also has some nice IP blocking and cookie features in there to allow access to the right people. I’m currently using this in an installation where the company in question wanted to ready sites with posts before pushing them out live. Not a bad plan but it can be tricky to implement.

As a bonus here’s a little code fragment to tell which sites are in and out of maintenance mode…

$options = $wpdb->get_results("
    SELECT option_value
    FROM {$wpdb->get_blog_prefix($blog->blog_id)}options
    WHERE option_name = 'wpjf3_mr'
");

// If you like you can decode the JSON returned here with json_decode
if (!strstr($options[0]->option_value, "YES")) {
//Do something if the site is not on the list of sites in maintenance mode.
}

$blogs = $wpdb->get_results("
    SELECT blog_id
    FROM {$wpdb->blogs}
    WHERE site_id = '{$wpdb->siteid}'
    AND spam = '0'
    AND deleted = '0'
    AND archived = '0'
    ORDER BY path
");

$sites = array();

foreach ($blogs as $blog) {
    // Get the sitenames
    $sites[$blog->blog_id] = get_blog_option($blog->blog_id, 'blogname');

    // Get the tagline out for the blog description
    $sql = "SELECT option_value AS tagline FROM {$wpdb->get_blog_prefix($blog->blog_id)}options WHERE option_name = 'blogdescription';";
    $description = $wpdb->get_results($sql);
    $tagline[$blog->blog_id] = $description[0]->tagline;
}

cursor: hand;

Well, that’s an IE only supported CSS word. Use this one instead…

cursor: pointer;

Job done!

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.something\.co\.uk [NC]
RewriteCond %{QUERY_STRING} ^(.*)$
RewriteRule ^(.*)$ http://www.something.com/$1?%1 [R=301,L]