April has been one hectic month! Between launching WhatGrowersUse.com, pre-launch marketing Optimise and fixing PlantConnection, I’ve barely had a moment to stop and think!

Add to this the fact that I’ve been swotting up on all things dev. I love coding in PHP and today I managed to hack out a super slender piece of functionality for an awesome *top secret* project I’m working on!

I basically need to pull a massive XML feed (think >10MB) and process the contents into a database. So this is grand-scale processing of huge wads of XML data – something I’ve never really had to do before.

This may seem a bit backwards, but moving away from the XML is imperative. This project needs to be fast and agile. I don’t want to have to allocate huge chunks of memory to my whole PHP layer just to deal with a few records in various, disparate XML files.

Starting with SimpleXML

SimpleXML is a fantastic XML parser. Being able to turn XML into native PHP objects, arrays and variables is supremely handy. However, “Simple” is definitely the operative word! When it comes to huge amounts of XML data, SimpleXML just doesn’t cut it!

So in my search for an alternative, I came across something I’ve never used before: a little-known – and it seems little-used – PHP extension called XMLReader. My hero.

XMLReader saves the day

XMLReader is what is known as a pull parser… you grab what you need as the parser races through the XML at lightning speed. It streams the file into context too: reading, caching, splurging… so you can deal with the XML before the whole file is loaded into memory.

This is a huge advantage for massive files. As long as the XML is valid, you can run away processing elements and attributes at a blistering pace.

Something to remember though is that doing it this way is no good if you then try to process all of this data in memory intensive variables or objects (which is the problem with SimpleXML). You either face increasing the memory allocation to PHP (which is limited by how much RAM you have) or you find a quick way to deal with the data and move on.

It’s grab-and-release… no time for processing here. Any kind of re-assignment of data where volumes exceed 30,000 reps of a while-loop will suck your memory dry. So no massive strings, no huge arrays and definitely no objects!

Execution Time

The only issue left is execution time. Between parsing the XML file  and saving it to a local MySQL db, your script could take a lot longer than your default execution time.

XMLReader is pretty darn snappy and there’s really no way to improve that with digging into some C code and rebuilding the extension. The code that actually does stuff with the data from the XML is very minimal. The biggest challenge is time writing to the database.

I’m using MySQL. The quickest method of writing to a database (save reformatting the data into some kind delimited text file and using load_data_infile) is mysqli prepared statements. This greatly reduces database load and running thousands (or hundreds of thousands) of queries can be done in mere seconds.

However, even this will be too slow if your max_execution time is 30 seconds. But the only real way to speed MySQL up is a faster processor and improved disk-write speed (think SSDs). Those are expensive options.

The simplest option? Increase script execution time. If this isn’t a user page, you can be a little more relaxed on timeouts. You should be able to use the set_time_limit() function to increase execution time. In fact this function has a handy habit of simply extending your current execution time by the limit you set.

<php
  while ( $xmlReader->read() ) {
    set_time_limit(2);
    // ... get data, save to DB etc ...
  }
?>

This will give each loop an extra two seconds to execute, which should be more than enough time to parse an extra few XML nodes and execute a prepared statement one more time.

So there it is. If you’d like to see the full code or have any questions, just ask, I’d be happy to share it. A huge thanks goes to Chad Fennell for his excellent post on XMLReader

APIs and connecting web applications together is going to be the next challenge of the evolution of the web. The next decade should see easier-to-implement, yet more secure methods for connecting the various web applications that we use.

We’re already seeing this in simple ways: RSS, Atom Publishing, Web Hooks. But these are generally insecure. They have been built/conceived with a set of specific goals in mind and then adapted – shoe-horned – to fit a variety of other purposes.

We are making progress: APIs have flooded the web. Mashups threaten to implode the universe in their brilliance. But there are still hurdles. There are still no standards.

REST is just a set of principles (for many things, not just web app APIs). HTTP-REST for Web Service APIs is no definition of a standard method for implementation. It doesn’t even begin to address the need for documenting or versioning APIs.

SOAP is better, but this is so wrapped up in corporate boggling and debate that real developers are reluctant to use it (as evidenced by the fact that 68% of APIs are built on RESTful principles vs 19% on SOAP) or simply want a more open standard.

Even if all web service APIs were RESTful, we still need a better method of standardisation and documentation (and implementation could be a lot easier with some other common features set down in stone).

What we need isn’t another working group to start thinking about how to solve these problems, but rather, someone to stick their neck out and do something to tackle these challenges.

We are at a turning point. The languages and processes are in place. The standards that need to be ratified have been. The scalability challenges are fading away. The core building blocks of something brilliant are there.

This is why I am building reactor. It’s aim isn’t to create a formal standard that all others must obey, but rather to encapsulate existing and future systems, boil them down to their basic standards and provide a single, simple point of entry into every API out there.

My vision is to make it easier for API implementers (provider developers) to standardise and document their APIs and easier for integrators (consumer developers) to consume those APIs in a standard way.

“One API to rule them all”

Does this sound like something you’d be interested in getting involved in? If “HELL YEAH!!” is your answer, then sign up for the mailing list.

Ok, so it’s not a big announcement because the app hasn’t actually launched yet. But I’ve launched the holding site.

So what’s reactor?

Well it’s a really exciting idea for a web app that I’ve had for ages. Now I’m moving forward with it. If you want more information, sign up to the mailing list. If you really want a lot more information, email me or something. I’ll send you out an NDA and whatnot (yes it’s that official!).

w00t! My first proper big web app!

Hope you’ll join me (*cough* developers)

It’s not often I have to worry too much about the minutia of database administration… well, I try not to. But this question on StackOverflow got me intrigued, so I put on my trilby.

Of course, if you write your programs to send MySQL the correct datatypes, changing this option shouldn’t cause any problems

I’ve been doing a lot of reading up on HTML5 and XHTML2. I know neither of these technologies are anywhere near well-supported enough to start using in production. However, we should all be starting to get our heads around the changes – if not only to be ready for the shift, but also the benefits it will bring.

Well it’s coming to the end of 2008 and there’s nothing left to say, but:

Why Are CMSs Not Included In The “Web Apps” Balloon?

I realise this is my first blog post in a long while. Despite the fact that I want to make it interesting, I am terribly busy. However I needed to get this off my chest as it has been bugging me for ages (hopefully it will help some of you!).

While researching faster methods for data retrieval and manipulation for my *TOP SECRET PROJECT* I found that, as of a very recent version of PHP, the MySQL_Result class’ method ‘fetch_object’ allows you to specify a class to dump the object into.

My first thoughts were: fantastic! What a better method than what I was doing previously. I can instantiate an object and set all the properties all of the back of the query – dynamic properties while still inheriting all methods and hard-coded properties… excellent!

As I continued down this route I found some unusual behaviour though. First of all, it appears that a call to $result->fetch_object(‘Class’) makes use of the magic method __set() *EVEN WITHOUT __set() BEING DEFINED IN THE CLASS*. Now there’s no note of that in the docs!

So what happens when I do define a __set() method? Interestingly $result->fetch_object(‘Class’) then uses the __set() method I define. Good, it behaves. But then I found something else. A little irritating “bug” is that, upon calling $result->fetch_object(‘Class’), it constructs the object in an unusual order.

Perhaps it’s just me and my thought process, but I would’ve gone for:

• Check if __construct() is defined
• Instantiate object
• Check if __set() is defined
• Set properties

However the actual process order is:

• Check if __set() is defined
• Set properties
• Check if __construct() is defined
• Instantiate object

This threw me a little, especially as there’s little documentation detailing the behaviour of MySQL_Result::fetch_object() when overloading with a class name. It does make some sense, but it’s not the human way of doing it (if my way is human ). I tend to think like ‘build box, put toys in’, rather than ‘pile toys up, build box around toys,’ but that’s me.

But now I’ve sorted it and it’s out there! Let me know if that helps you.

We (and by we I mean the company I work for) launched one of the most interesting web sites we’ve produced to date.

Are we still supporting browsers that have had their day? It seems the simple answer is “hell yes!” I ask why…

I stumbled across a website today consisting of one page prompting users to upgrade their browser. It offers the hope of a better user experience, but also notes that it would make developers’ and designers’ lives easier.

I know there are many arguments that exist for supporting all possible browsers on all possible platforms. I know there are services around (such as Litmus) that provide cross-browser, cross-platform testing. I know there are still a bevy of users out running older browsers.

However, my argument is that there are few enough of those that we as designers/developers can actually start to make them feel outcast. Not that we are trying to bully anyone into upgrading their browser – heck we’ve been so subservient up till now, our bluff would be called and we’d probably cry and run off.

The thing is that most users who are still sporting legacy software are running a machine too old to cope with a newer browser. That, or they’re insistent on using old, comfy browsers. In either case, supporting them is a burden we need not bear and definitely not a necessity.

I put forward that web sites (and other modern software) should be built to work on the latest and greatest… to a point. I concede that there is an acceptable grace period while we wait for the majority of users to upgrade, but if we make it clear that they should, then this process wouldn’t be quite so long. If the content doesn’t render quite right, then that’s their fault.

I appreciate that’s a little harsh. I would therefore like to put forward a proposal: an introduction of META tags for various user agents. Once you are happy that a site works well in a minimum browser version, insert the tag for that browser.

Browser vendors could then check for their User Agent meta tag and compare the version data and use this to perhaps prompt the user that they need to upgrade. I guess this is a bit like Adobe AIR and Microsoft’s proposals in IE8 (except none of this conditional comments crap).

Again I find my blog post titles not really resembling any coherence… if you follow the TWiT podcast you’ll notice my titles are starting to follow the ethos: it’s more a summary of what’s covered than an introduction… Still it works for now.