April has been one hectic month! Between launching WhatGrowersUse.com, pre-launch marketing Optimise and fixing PlantConnection, I’ve barely had a moment to stop and think!

Add to this the fact that I’ve been swotting up on all things dev. I love coding in PHP and today I managed to hack out a super slender piece of functionality for an awesome *top secret* project I’m working on!

I basically need to pull a massive XML feed (think >10MB) and process the contents into a database. So this is grand-scale processing of huge wads of XML data – something I’ve never really had to do before.

This may seem a bit backwards, but moving away from the XML is imperative. This project needs to be fast and agile. I don’t want to have to allocate huge chunks of memory to my whole PHP layer just to deal with a few records in various, disparate XML files.

Starting with SimpleXML

SimpleXML is a fantastic XML parser. Being able to turn XML into native PHP objects, arrays and variables is supremely handy. However, “Simple” is definitely the operative word! When it comes to huge amounts of XML data, SimpleXML just doesn’t cut it!

So in my search for an alternative, I came across something I’ve never used before: a little-known – and it seems little-used – PHP extension called XMLReader. My hero.

XMLReader saves the day

XMLReader is what is known as a pull parser… you grab what you need as the parser races through the XML at lightning speed. It streams the file into context too: reading, caching, splurging… so you can deal with the XML before the whole file is loaded into memory.

This is a huge advantage for massive files. As long as the XML is valid, you can run away processing elements and attributes at a blistering pace.

Something to remember though is that doing it this way is no good if you then try to process all of this data in memory intensive variables or objects (which is the problem with SimpleXML). You either face increasing the memory allocation to PHP (which is limited by how much RAM you have) or you find a quick way to deal with the data and move on.

It’s grab-and-release… no time for processing here. Any kind of re-assignment of data where volumes exceed 30,000 reps of a while-loop will suck your memory dry. So no massive strings, no huge arrays and definitely no objects!

Execution Time

The only issue left is execution time. Between parsing the XML file  and saving it to a local MySQL db, your script could take a lot longer than your default execution time.

XMLReader is pretty darn snappy and there’s really no way to improve that with digging into some C code and rebuilding the extension. The code that actually does stuff with the data from the XML is very minimal. The biggest challenge is time writing to the database.

I’m using MySQL. The quickest method of writing to a database (save reformatting the data into some kind delimited text file and using load_data_infile) is mysqli prepared statements. This greatly reduces database load and running thousands (or hundreds of thousands) of queries can be done in mere seconds.

However, even this will be too slow if your max_execution time is 30 seconds. But the only real way to speed MySQL up is a faster processor and improved disk-write speed (think SSDs). Those are expensive options.

The simplest option? Increase script execution time. If this isn’t a user page, you can be a little more relaxed on timeouts. You should be able to use the set_time_limit() function to increase execution time. In fact this function has a handy habit of simply extending your current execution time by the limit you set.

<php
  while ( $xmlReader->read() ) {
    set_time_limit(2);
    // ... get data, save to DB etc ...
  }
?>

This will give each loop an extra two seconds to execute, which should be more than enough time to parse an extra few XML nodes and execute a prepared statement one more time.

So there it is. If you’d like to see the full code or have any questions, just ask, I’d be happy to share it. A huge thanks goes to Chad Fennell for his excellent post on XMLReader

[tweetmeme]I was recently invited to the NEC to give a 20-minute seminar to those gathered for the annual PROMOTA trade show.

It was an interesting day as I was also called in as a “Business Expert” to provide free, hands-on advice to attendees.

The only question everyone asks me at these kind of shows is: How can I get to the top of Google? My answer: stop trying!

It may sound a little simple and stupid, but in most cases, the people asking have neither the time or budgets to make it worthwhile.

If you’re a small business with the goal of being at the top spot (or close to it) in Google, in certain cases you will need more than a couple of hundred pounds every month to get there!

In other cases you might not need a thing: Google “professional connections” and look for www.xon.cc… that’s a client of mine who hasn’t paid a penny for SEO!

So what’s the solution? Well, there’s no easy way to say this… be content with the position you’re in.

I’m not saying that being number one won’t bring you more sales… statistics show that it does. But don’t let your desire for the podium allow you to become disappointed in your lacklustre performance.

Remember, enterprising small businesses have made a decent living way before Google and will continue to do so.

If you can find the time to work on improving your online presence yourself (or with resources within your organisation), do it. There’s plenty of advice online to guide you in the right direction and many small companies are succeeding with this.

No matter what position you have on Google, sales aren’t guaranteed – after all, people only buy what they need/want and being at the top of a search is no guarantee of quality or competitiveness.

Also, being at the top of a results page for a particular Google search is still no guarantee that you’ll be seen or that users will click through!

It’s like having your company name on a slither of paper at the top of a pile of other names on similar slithers, in a box with a label on it, stowed away in an old library basement archives section – sat there… waiting in earnest for someone to dig it up.

Even if you managed to get the archivist to put you on top, there’s still no guarantee that anyone will see your label or that they’ll do anything with it when they do!

Image: Andersen library archives © dancypants

—

UPDATE 11.03.2010:
Some interesting related articles were published today that you might find useful. I feel they back up my point quite well. Check out Nick Pettit’s SEO is a terrible idea and Paul Boag’s response to the question Does Google Personalised Listings Affect Your Ranking?

I met three ace guys yesterday: Sean Leigh, Jeremy Harding and Alan Mann. They are all techy/web entrepreneurs and are making big bucks out of big ideas. They know some crazy people in the domaining field.

It turns out that there a lot of wealthy people doing domaining. I found out yesterday that there are some people out there that own thousands of domains with no websites attached to them! Craziness! They obviously have a lot of money.

The reason I met these great guys though is because they found me and this brings me to a little tip: Get your name out there!

It turns out they found me through a service I signed up for and hardly use, Elance. The data from Elance was probably shared on a network of other services, which they stumbled across. My profile on this service is sketchy at best. But it was enough of a lead for them to do more digging.

Because I’m an active participant on a number of blogs, forums, Twitter etc. they managed to fill in the gaps pretty quick.

So if you want to be found, get active in your community! A word of warning though: Be prepared for a grilling (i.e. bring something amazing along).

It’s not often I have to worry too much about the minutia of database administration… well, I try not to. But this question on StackOverflow got me intrigued, so I put on my trilby.

Of course, if you write your programs to send MySQL the correct datatypes, changing this option shouldn’t cause any problems

In our office, we have a small custom, headless 32-bit PC running Ubuntu Server 8.10 (Intrepid Ibex). It’s ideally suited as our testing platform web server, file server, SVN server… well you get the picture.

I’ve just been on the phone with a potential client. From our brief chat it’s clear that they have experienced problems with “web” people before now. This has affected their view of our services at FlipStorm, even though they know nothing about us.

By G-Spot I mean the first page of search engine results in Google. A couple of months ago I sent out this email to a friend of mine who asked me to analyse his website (Damian Brown Photography).

The web developer and design community is growing super large and because of that it has been an endless source of advice and inspiration to me. As a large thank you gesture I want to offer my professional advice and support in any areas I can.