April has been one hectic month! Between launching WhatGrowersUse.com, pre-launch marketing Optimise and fixing PlantConnection, I’ve barely had a moment to stop and think!

Add to this the fact that I’ve been swotting up on all things dev. I love coding in PHP and today I managed to hack out a super slender piece of functionality for an awesome *top secret* project I’m working on!

I basically need to pull a massive XML feed (think >10MB) and process the contents into a database. So this is grand-scale processing of huge wads of XML data – something I’ve never really had to do before.

This may seem a bit backwards, but moving away from the XML is imperative. This project needs to be fast and agile. I don’t want to have to allocate huge chunks of memory to my whole PHP layer just to deal with a few records in various, disparate XML files.

Starting with SimpleXML

SimpleXML is a fantastic XML parser. Being able to turn XML into native PHP objects, arrays and variables is supremely handy. However, “Simple” is definitely the operative word! When it comes to huge amounts of XML data, SimpleXML just doesn’t cut it!

So in my search for an alternative, I came across something I’ve never used before: a little-known – and it seems little-used – PHP extension called XMLReader. My hero.

XMLReader saves the day

XMLReader is what is known as a pull parser… you grab what you need as the parser races through the XML at lightning speed. It streams the file into context too: reading, caching, splurging… so you can deal with the XML before the whole file is loaded into memory.

This is a huge advantage for massive files. As long as the XML is valid, you can run away processing elements and attributes at a blistering pace.

Something to remember though is that doing it this way is no good if you then try to process all of this data in memory intensive variables or objects (which is the problem with SimpleXML). You either face increasing the memory allocation to PHP (which is limited by how much RAM you have) or you find a quick way to deal with the data and move on.

It’s grab-and-release… no time for processing here. Any kind of re-assignment of data where volumes exceed 30,000 reps of a while-loop will suck your memory dry. So no massive strings, no huge arrays and definitely no objects!

Execution Time

The only issue left is execution time. Between parsing the XML file  and saving it to a local MySQL db, your script could take a lot longer than your default execution time.

XMLReader is pretty darn snappy and there’s really no way to improve that with digging into some C code and rebuilding the extension. The code that actually does stuff with the data from the XML is very minimal. The biggest challenge is time writing to the database.

I’m using MySQL. The quickest method of writing to a database (save reformatting the data into some kind delimited text file and using load_data_infile) is mysqli prepared statements. This greatly reduces database load and running thousands (or hundreds of thousands) of queries can be done in mere seconds.

However, even this will be too slow if your max_execution time is 30 seconds. But the only real way to speed MySQL up is a faster processor and improved disk-write speed (think SSDs). Those are expensive options.

The simplest option? Increase script execution time. If this isn’t a user page, you can be a little more relaxed on timeouts. You should be able to use the set_time_limit() function to increase execution time. In fact this function has a handy habit of simply extending your current execution time by the limit you set.

<php
  while ( $xmlReader->read() ) {
    set_time_limit(2);
    // ... get data, save to DB etc ...
  }
?>

This will give each loop an extra two seconds to execute, which should be more than enough time to parse an extra few XML nodes and execute a prepared statement one more time.

So there it is. If you’d like to see the full code or have any questions, just ask, I’d be happy to share it. A huge thanks goes to Chad Fennell for his excellent post on XMLReader

I absolutely love coding in PHP. Sometimes I get distracted by the glitz and glamour of some of the more popular languages (and their associated frameworks) – and I agree, they have their place. But PHP is in a class of its own.

The truth is that for years PHP has had an active community – and this continues to thrive. Because it’s so easy to get your hands on PHP, and get it installed, hosting is cheap, and there are no major licensing issues, uptake was fast.

The PHP site fast became the home of many PHP-related projects and spun off into package managers and extension managers (PEAR and PECL respectively). Literally thousands of developers write setup and “Hello World” tutorials. Loads of scripts and apps are open-source and open-licensed. No wonder it is the most popular language in use on the web!

However, we’re moving into a new age. A very competitive era dawns. PHP needs to stand up to the impending onslaught of faster compilers and VMs. The competition will be tough. But I believe PHP can hold its own.

As we move into this era of greater web app interconnectivity, PHP seems poised to reign strong. With a flock of transport layer extensions (think JSON, SOAP, XML-RPC) and authentication modules (OAuth, OpenID etc) at the ready, it looks pretty solid.

One thing stands at the gates, pleading nervously and gently as we give out our warcry: documentation, documentation, documentation. And for good measure… DOCUMENTATION!

The PHP site is littered with it. The community add to it. But when it comes to third-party tools, especially PEAR libraries and other third-party code, we’re pretty weak.

The area of most concern has to be client libraries for web service APIs. Some are just totally appalling: outdated, incorrect, poorly formatted or purely non-existent.

This is a bad show and something we need to improve upon desperately if PHP is to be the language of choice for web development in the next decade.

You might argue that it’s not going to disappear off the map. You may be right. But there’s no harm in making our lives easier and ensuring PHP’s safety and continued popularity by writing better documentation!

Come on chaps… for GLORY!

APIs and connecting web applications together is going to be the next challenge of the evolution of the web. The next decade should see easier-to-implement, yet more secure methods for connecting the various web applications that we use.

We’re already seeing this in simple ways: RSS, Atom Publishing, Web Hooks. But these are generally insecure. They have been built/conceived with a set of specific goals in mind and then adapted – shoe-horned – to fit a variety of other purposes.

We are making progress: APIs have flooded the web. Mashups threaten to implode the universe in their brilliance. But there are still hurdles. There are still no standards.

REST is just a set of principles (for many things, not just web app APIs). HTTP-REST for Web Service APIs is no definition of a standard method for implementation. It doesn’t even begin to address the need for documenting or versioning APIs.

SOAP is better, but this is so wrapped up in corporate boggling and debate that real developers are reluctant to use it (as evidenced by the fact that 68% of APIs are built on RESTful principles vs 19% on SOAP) or simply want a more open standard.

Even if all web service APIs were RESTful, we still need a better method of standardisation and documentation (and implementation could be a lot easier with some other common features set down in stone).

What we need isn’t another working group to start thinking about how to solve these problems, but rather, someone to stick their neck out and do something to tackle these challenges.

We are at a turning point. The languages and processes are in place. The standards that need to be ratified have been. The scalability challenges are fading away. The core building blocks of something brilliant are there.

This is why I am building reactor. It’s aim isn’t to create a formal standard that all others must obey, but rather to encapsulate existing and future systems, boil them down to their basic standards and provide a single, simple point of entry into every API out there.

My vision is to make it easier for API implementers (provider developers) to standardise and document their APIs and easier for integrators (consumer developers) to consume those APIs in a standard way.

“One API to rule them all”

Does this sound like something you’d be interested in getting involved in? If “HELL YEAH!!” is your answer, then sign up for the mailing list.

It’s not often I have to worry too much about the minutia of database administration… well, I try not to. But this question on StackOverflow got me intrigued, so I put on my trilby.

Of course, if you write your programs to send MySQL the correct datatypes, changing this option shouldn’t cause any problems

So I hit my first attempt at using something not bundled in Apple’s ’safe’ PHP build for Mac OS X. I decided to implement memcached on my big project. I found out that memcached is pretty easy to set up and had that running in no time.

The problems start when trying to get PHP to talk ‘memcached’. Windows binaries come bundled with the latest stable release of the PECL Memcache library, but Apple has decided not to bundle it into the PHP Apache module.

Then I got thinking: “Well what about other extensions that I might need?”. I resolved to get this working so I knew what to do for future extensions!

I’d read so much about how Apple has disabled extensions and there’s no way to make it work… blah blah blah… Then I found this brilliant tutorial by Erik Giberti.

Stupidly though I didn’t follow the guidelines to the letter (I was probably distracted), but Erik provided continuing support. I Twitter’d him and he seemed more than happy to oblige. One of the good guys!

Upshot… I now have my first working self-compiled extension to PHP loaded and empowering my local development platform. That wasn’t so bad, was it?

For future reference:

1. Download latest stable extension from PECL
2. cd Downloads/{library_name-x.x.x}/
3. phpize
4. ./configure (may need extra compiler options!)*
5. make
6. sudo make install
7. Update PHP ini with extension=(whatever the installed file is)

You may need to find out what your extension_dir ini variable is (HINT: phpinfo!) as this is where the .so file will need to be. In point 7 just put the file name not the path.

I will no doubt be adding more libraries in the coming weeks. If I get any trouble, I will let you know. Sound off in the comments your issues with PHP extensions in Mac OS X

UPDATE 04/09:
*If you’re running 64-bit architecture (and more than likely have a 64-bit build of PHP) you should run the followingat point 4 instead:
MACOSX_DEPLOYMENT_TARGET=10.5 CFLAGS=”-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp” CCFLAGS=”-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe” CXXFLAGS=”-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe” LDFLAGS=”-arch ppc -arch ppc64 -arch i386 -arch x86_64 -bind_at_load” ./configure

I just wanted to write a quick post (as it’s late and my Mrs wants me to come to bed) about lingering jobs. We all have them. Washing up that’s starting to grow its own ecosystem, that lightswitch that’s still hanging off the wall (but it works!)…

For me it’s a plethora of oddjobs for old clients. In my case they come back for more, usually at the most inconvenient times. This time however it’s all my fault. In desperation I have offered to do work in the hopes that I will finally get these people off my back.

It’s not that they’re begging me to do things, but more for my own sanity. I guess I have some mild form of OCD. I just want things tidy. Is that so bad?

I find that eventually there are so many things on my mind that I can do none of them. So I suffer silently in a corner somewhere waiting for the light to come on. Ultimately it’s a crippling fear of failure that grips me most. Of course it is that very fear that breeds itself.

I’m sure you understand…

So follow my changing mental state on Twitter. You’ll catch some of my mini rants and you may even witness my entire nervous breakdown (reality TV eat your heart out!)

Dear Audrie,

Dear Judy,

Since the Jobs-note last Monday (9th), a lot of media attention has been focussed on the App Store and the coming features of iPhone 2.0.

Sadly, orders of magnitude more media attention has been focussed on the 3G iPhone – the highly anticipated, but hugely underwhelming revelation that it is.

The iPhone’s new little (big) brother does address some of the more than glaring ommissions that the original was left wanting. And in my view it does finally justify the hefty price tag. But even this has been reduced! To the shock of all!

I shall definitely be getting a 3G iPhone as soon as my existing contract expires (especially as my handset is slowly falling apart).

There are still some issues that I have with the iPhone. These are more correctable with firmware updates than hardware changes, and save for any major improvements in carrier networks’ hardware I don’t see that there’s much need for adding to the tech specs of the 3G iPhone anytime soon.

In my opinion, most gadget ‘freaks’ want the best of all their gizmos in one – the Buddhist (as i like to call them): one-with-everything types. This was (and still is) true of me… to a point. I’ve come to realise that it’s not always best. So I care little about the quality of the camera in the iPhone. It’s sufficient to know that it has a camera.

Little touches are more annoying: like the lack of a light, which I always find no end of uses for, besides for taking photographs. Or a non-removable battery – what the hell? These are about the only physical changes that would make the iPhone my perfect gadget (besides any changes to improve battery life and reduce cost of manufacture whilst improving end-user experience).

The real focus of this post though is the facility that we as end users are all being equipped with: a touch-screen interface. Here MultiTouch isn’t such a big deal. Arguably it does provide a few more natural ways to interact with the operating environment, but it still has its challenges.

However, just having an all-purpose piece of touch-screen kit on the open market (and at a good price) presents some potentially new and exciting ways for us to interact with our other gizmos.

Many are already exploring the possibilities of controlling other devices using the iPhone and iPod Touch. What I would really like to see are practical applications for the majority, not just experimental things that are restricted to specific enterprises. I’m talking about controlling household appliances from my iPhone/iPod Touch.

I realise that it would require that any device that you want to control with your super-phone would need to be connected to the IP network and configured accordingly. But if manufacturers of these devices can be persuaded to see the potential, there’s all sorts of wonderful possibilities.

Aside of my finger-marked status symbol becoming a universal remote control, it could provide interfaces and readouts for numerous appliances.

The real beauty of it is that a touch screen is not limited by what it can display. There’s no fixed set of commands. You don’t have to provide overlays to provide the right input. It’s all customisable and application-specific. It’s intuitive and easy to learn.

The iPhone is the start of many similar devices making their way into our homes. If we can come up with some useful applications it may be the only one we will need!

Dear Tanya,

Google Friend Connect was officially announced last week. It has since come to light that this is no ordinary social network. In fact in the truest sense of the term, it isn’t a social network at all!

Friend Connect is a social tool. It seems the lovely people at the big G have been working their socks off finding a way to help us all connect a lot more easily. This fantastic tool is still only in preview, but from the fairly sketchy detail proffered by Google, it seems that even now it has a very wide appeal.

In order to place a perspective on what Friend Connect allows you to do, imagine: you are the only software developer on the planet. You, and you alone, have developed each and every web site in the world and you had to build each site from scratch.

So that means no code sharing. No data sharing. This means for every user who visits a web site that requires authentication they will have to register and verify their details. Then they will have to maintain them. This is the kind of data duplication that would make a data analyst’s butt clench.

By now you would have won plenty of awards… at the very least you would be long overdue some much-needed shut-eye.

The solution would seem obvious: extract the common data structures (in this case the individual member’s details) into a central pool that each site can get access to. That is what Friend Connect does!

You can close your mouth now.

As well as being a social connector, it is also a social enabler. What I mean is that a site that is not a social network (DamnILoveChocoDip.net for example) and never intends to be can still benefit from the viral nature of social networking. The social aspect of visiting a web site and sharing your experience is made even easier.

Yes as the modern social surfer you no longer have to re-register the same details you saved with the previous web site into this one. Simply log in with the same credentials et voila you’re in. Your data isn’t copied or duplicated. You keep the one central piece up to date and Friend Connect does the rest (as they say).

Beautiful! Adding to this is the entire social aspect. In theory (because I haven’t had a chance to test it yet) you can see who of your friends are registered with this site. They will see when you do things on the site (that you explicitly allow).

So all of a sudden, for the majority of web sites we visit, we can now see who else is there. It’s like all of the other patrons visiting this virtual shop/bar/resource centre become visible.

You could actually “bump” into someone who you know from another web site. As you seem to share a passion for at least two things, you might pluck up the courage to strike up a conversation, perhaps opening with an oft-hounded ‘chat-up line’.

It would be the equivalent of walking around the town, bumping into that cute girl in two different shops, suggesting you go for coffee, then quickly cut to 3 years later and you’re story is being told in some horrible romantic comedy starring Tom Hanks.

There are some questions on security and integration, all of which I’m sure will be answered in time. If you’re a programmer like me, I’m sure you will see the benefits though. I’m drooling over some of the possibilities. Least of all not having to build a registration engine every time I build a site. A close second is the speed with which people will discover a new web site.

Also in this orgy of social debauchery is OpenSocial. Now application developers have an opportunity for their applications to make it onto millions of web sites, not just the few major and accessible social networks.

The issue at the moment is that not all networks are supported, and indeed not all will want to jump on the bandwagon. But when things become this easy for users, where do you think the majority will go?

So then the key lessons from all of this are… build a site that can make use of this centralised platform! If you’re building an application, build it for OpenSocial as the chances are it will lead the way in terms of mass integration.

Is this a sign that Facebook may slip into Microsoft’s online “I just don’t get it” pit? I wonder why that could happen.

Dear Geraldine,

Yes I am begging for a verbal beating from all the nay-sayers… but Apple have pushed the boat out with full featured web browsing on the mobile platform.

Many non-converts will say that it’s missing some vital features – and they are right. Without Java and Flash on either version of Apple’s uber status symbol, they is missing out on some serious functionality and usability.

Let us not forget though that this is merely the first incarnation of these oft-heckled devices. And many of these issues can be resolved by a “simple” software update.

Why am I so bold? You would be right in thinking I sold out. Yes I bought an iPod Touch. Happy with my current cellular device I couldn’t justify the expense of the iPhone without the features I now expect from a modern handset – especially one with so much (dare I say?) “PDA-like” potential! – What should the next iPhone have?

I’m no analyst, marketer or even a prophet, but Apple have started setting trends again. With the full browsing experience, that will now doubt get even more feature-complete as time goes on and public demand grows, we can now browse the existing web – not having to create a whole new layer of the web to cater for our mobile nature.

This will annoy some to say the least. To all those who have spent their resources on extracting their existing online presence into something that can be neatly accessible to mobile users I say under my breath: “Waste of time”. See Mobile Web, Ubiquitous Web – it’s going to happen if you don’t believe me.

Yes the .mobi domains and re-creation of applications sans Javascript are now not worth pursuing! Why should we when our existing online presence can be viewed almost perfectly on these modern devices?

Forget demographics, usage stats and the like… the iPhone has now set the base standard for how we want to browse the web on the go. Users will demand it. Manufacturers will play to them. Developers must go with the flow!

As a web developer, I have a keen interest in this shift in how users interact with the online community. With the dawn of these devices, our mobile-enabled web applications can now be part of our existing application development model. Requests made to our existing systems will automatically adjust the content to suit the platform.

The creators of Facebook have managed to churn out a modest working example of this for the iPhone and iPod Touch.

Google too are constantly pushing forward with their developments for mobile users. But they have development teams and funding. Although many (including me) are worried about the advertising model that Google has pioneered as to whether it will be maintainable in this new era of web browsing – it is already showing signs of cracking. But this is a topic for further discussion.

What about Joe Small Business Owner – how will he get his web site to be truly accessible in this new arena? You could argue that he doesn’t need to worry because of how these devices manage the current web.

It will become more evident in time though that the web applications that run natively on the platform the user prefers will be the ones that are more favoured and better used.

In order to deliver this in a manageable and cost-effective solution we really need a product that gives its administrators the power to decide what aspects of their online presence will be available to their mobile visitors.

This power doesn’t rest with them at present and thus is forcing a chasm in the mobile web where services either do not exist or are so simple they would be better off if they didn’t exist!

As we move forward into this almost-virgin territory are any of us truly ready?