I find myself using the client functionality of the gearman tool most often. If I am tasked with creating or updating a gearman worker I want to test that the worker actually works. The code that sends a job to the worker is normally part of the web application and I don't want to dig through the application trying to figure out what I need to do send the job out. I could just create a simple php script that creates a client and sends the job over, but the gearman tool already does this.

Example of using gearman as a client:

I use the gearman program as a worker less often. It is still useful for creating a simple worker to test my client code against. I can write my client code without a fully functional worker if the client code is not expecting a complex response.

Example of using gearman as a worker:

The gearman tool comes with the standard options for specifying a specific host and port. There are a number of other options that may be of use in specific circumstances. I encourage you to read them over by typing "gearman -H" on the command line.

Declaring a global functions in a gearman worker script may not seem like a big deal, but these things have a way of catching up to you. I personally ran into this when I suggested that HauteLook start using GearmanManager to manage the gearman workers. A side affect of this is that all gearman worker scripts now run in the same instance of PHP. There were a number of occasions when workers would fail to load on production because of the global naming conflicts.

I prefer to use register anonymous functions with my gearman workers. This keeps them out of the global scope and puts the logic right next to the worker registration call. This makes hard to test the logic inside the anonymous function, but I never test that logic. I treat the functions I register with gearman as controllers. I pass the workload off to a model class that is fully unit tested.

Here is a nice example:

I would update the Gearman examples on php.net, but I think a lot of people are still using PHP 5.2 (or even earlier). Providing multiple ways of registering the workers may confuse people too. Maybe next year.

The release of pecl/gearman 1.0 included a fix to the exception handling of gearman workers. An uncaught exception would go unnoticed by the gearmand server before pecl/gearman 1.0. The only way to work around this was to catch all exception and explicitly return a failure code to gearmand. I fixed this by making by detecting an exception with the pecl/gearman code and sending a special exception message to the gearmand server. Unfortunately, the API for sending the exception message was introduced in gearman 0.21.

I did not think much of the new gearmand requirement when fixing this bug. Since that time I have been introduced to the painful upgrade process of gearmand. I think this pain is mostly felt by those using Linux distributions with long term support. The first hurdle to get over is the fact that gearmand 0.21 requires the boost 1.41 libraries. A default Cent OS install only provides boost 1.39 libraries. There was a 3rd party that provided boost 1.41 libraries for Cent OS, but they used a non-standard installation directory. After some experimenting, I did find a fix: http://groups.google.com/group/gearman/msg/12bcc13691e132ae.

The good news is that upgrading to Cent OS 6 makes compiling gearman much easier with the inclusion of boost 1.41 libraries by default. The next problem comes when trying to submit a patch. Cent OS 5 comes with aclocal 1.9 and the gearman project requires aclocal 1.11. Again, upgrading to Cent OS 6 makes life easier as it comes with aclocal 1.11 by default. I learned the hard way that the gearman project also requires autoconf 2.64, but Cent OS 6 only provides autoconf 2.63. The 2.64 version of autoconf provides a macro named m4_ifnblank which is required to build the configure script used to compile gearman. You can see me answer my own question here: https://answers.launchpad.net/gearmand/+question/185592 regarding the autoconf issue.

These build issues have made it tough for users to upgrade to the pecl/gearman 1.x line. I think I will be forced to maintain the pecl/gearman 0.8.x line for some time with bug fixes as users struggle to get gearmand 0.21 on their Cent OS 5 servers. I have definitely learned to do more research on migration paths before requiring a newer dependency version.

This is a pretty common use case and I am sure most people would write the logic something like this:

There is nothing logically wrong with this code. It is a very readable and easy to understand way to write it. With the rise in popularity of JavaScript and other functional languages, some of us may take a different approach. Using the underscore framework (available for both PHP and JavaScript), you could also write it like this:

The use of a callback function should be familiar to anyone with at least a basic knowledge of JavaScript. The callback function is evaluating each item in the iterator to true or false. The major drawback to both of the code snippets is that the logic for determining whether or not an item is sold out is not able to be re-used. The use of procedural code in the first example and the use of an anonymous function in the second example also make it hard to test. We can improve the second example by not using an anonymous callback:

Now I can re-use this function and easily test it. We can use SPL FilterIterator in a very similar way to the functional example:

Now my logic for what constitutes a soldout item is isolated in a class. This coincides with the object oriented principle of "encapsulate what varies". I think the biggest stumbling block for using SPL is that it just seems too heavy. You might be wondering why you should go to the trouble of creating a new class to perform such a simple task. The procedural example above is faster to write. Some of us might be tempted to use the functional example with an anonymous function because it feels more "expressive". Now consider what happens when the ecommerce system introduces returns. The new formula for determining a soldout item is now:


availability = (purchased - sold) + returned


This isn't some hypothetical example, it actually happened. It was real easy to update the logic to handle returns and make sure the tests passed.

The last decision you have to make is whether or not to put the logic in a database query. If there is a huge performance boost by writing the logic into the query, it may be worthwhile. A query is still testable, but you have to setup some test data in the database in order to test it (which means you probably won't do write the test for it). It is harder to re-use queries, so the business logic for determining sold out items may be duplicated over a number of queries too. You also might want to consider what happens if you decide to alleviate the database load by putting the list of items for sale in a cache (like APC or memcache).

The SPL classes, especially FilterIterator, really start to shine when dealing with a dataset outside of our control. More and more platforms are becoming service based and we have no control over how the data comes back. Especially when you consider something like the Twitter API timeline response and trying to filter out any tweet that starts with "RT".

In order to make development of the gearman workers easier, I created a script to startup the gearman worker in a similar manner to the GearmanManager. The major benefit is that any sort of error or exception is shown right on the terminal screen. No digging through logs wasting time tying to figure out what happened. If you have logging already setup, it is now trivial to add a stdout writer to the logger when running php in cli mode.

One of the biggest complaints I heard from fellow developers was that they had a hard time trying to figure out if the worker actually received the job from the client. I added a logging to stdout that signals when a job is received and what a job is finished.

I setup a pear channel to make installation real easy. For installation instructions and some examples check out the github page page: https://github.com/hradtke/gearboot

Like Vic, I too have moved away from using ORM frameworks. One of my biggest problems with ORM frameworks is that they become a de-facto language within themselves. Instead of learning how to write proper SQL according to the SQL-99 standard, I now have to learn SQL and the syntax of the ORM. Things only get worse from there when trying to debug problems of performance and correctness.

There are some niche tasks where I still use an ORM despite the costs mentioned by Vic and myself. An ORM is great for creating dynamic queries. As we build more rich front-end web applications, it is very easy to build queries with user specified filters and sorting. Trying to do this manually is time-consuming and often bug prone. I find this setup particularly useful when building administrative pages that are looking at lists of things like users or orders. Performance is not a real big issue here and the queries themselves are usually only joining one or two tables. There is still the risk of a runaway query wreaking havoc on your master database. To combat this scare, I advocate a separate slave database to run queries like this.

Vic mentioned that he puts the SQL directory in the controllers, but I think this is a mistake. Controllers are tying the view to the model and should not contain any other logic. I prefer to keep business logic at the library level and data access logic in the model. The controller can consume library or model classes at will and pass them over to the view for output. This may seem like an extra step that is not needed, but it makes testing much easier. There is also the added benefit of being able to write a service layer or API with very little effort should the need arise.

All in all I think Vic makes a good argument. I think developers need to learn to embrace databases, SQL or NoSQL, instead of trying to abstract them away. Data is often the most important part of the business and it should be treated as such.

I think common interfaces would be bad for PHP. I find it ironic that a comment made by Stephen is advocating a Java-like dictatorship to move PHP forward. PHP is much more successful and widely used than Java. It is Java should be learning from PHP, not the other way around. PHP is an ugly language with a mish-mash of syntax and conventions that makes the code not very pretty (especially compared to Python). PHP's only saving grace is that it actually gets things done. And luckily, for PHP, that is all that really matters.

Caching, logging and http clients may seem like simple things to standardize, but the mere fact that there are so many differing implementations is proof that they are not so simple. This reminds me of the SimpleCloud framework that tries to standardize the different cloud APIs: there is nothing simple about it! Furthermore, trying to standardize things can kill innovation and creativity. I fear the community will reject a new way to do things on the basis that the code does not implement the standard PHP interfaces. The Lithium framework, which takes a very different approach to frameworks, might not even exist if these interfaces were around a few years ago.

The most progress comes about when people reject or ignore the so called "truths" that surround them. The recent PHP fork is evidence of this. Forking PHP was a huge "no no" (despite the ironic fact that PHP is open source). That fork sparked both change and debate and had a positive effect. The author of the fork had tried to play by the rules setup by the PHP internals team and got nowhere. It was only when he decided to publish a fork that people (particularly those in internals) actually started paying attention.

This notion of interoperability between frameworks is just like trying to achieve nirvana. It is good to try, but never truly attainable. Look at PDO, a (supposedly) "lightweight, consistent interface for accessing databases in PHP". Except that it doesn't work with any of the NoSQL databases. One may argue that PDO is for SQL databases only, except that no one even considered that fact when it was being written. Simply read http://us2.php.net/manual/en/intro.pdo.php and it makes no mention of being SQL only. It assumed that SQL was all there was at the time of writing. The SQL-99 standard was the gospel for databases. This is the danger in defining standard interfaces. The advent of NoSQL made PDO largely irrelevant. The same fate will happen to any standard interface that we come up with today.

I encourage freedom the create new things. I don't want all of my frameworks playing by the same rules. If they are all using common interfaces, then we might as well merge all of the frameworks because the competition is over. Instead, let the adoption rate of the framework be the measure of success. I believe this is why PHP has been so successful. Rasmus has the courage to not try and define or predict what PHP needs. He lets the people using PHP decide what they need. This is in stark contrast to other people in internals that try to exert control of the process and define rigid rules and policies. PHP is not a purely procedural, object oriented or functional language. It is a mashup of all kinds of concepts and ideas. The cool thing is that it works.

I use Gearman for most of my batch processing problems. The process is simple: create any number of tasks using GearmanClient::addTask() and then run the tasks in parallel with GearmanClient::runTasks(). All available workers that can handle that task will be used. In order to keep track of the batch process, I define callback functions in GearmanClient::setCompleteCallback() and GearmanClient::setFailCallback().

Every once in a while one of the workers would throw an exception that would not be caught. One would expect that this would trigger GearmanClient::setExceptionCallback(). Until recently, pecl/gearman was not properly handling exceptions from workers. This might not be so bad if an uncaught exception sent back a status of GEARMAN_WORK_FAIL and triggered the fail callback. The problem was that an uncaught worker exception actually sent back GEARMAN_SUCCESS. This made it appear to the client that the tasks was successfully completed. The workaround at the time was to wrap all workers in a generic try/catch block to prevent any uncaught exceptions. The catch block would then need to explicitly send back a status of GEARMAN_WORK_FAIL. This caused a lot of boilerplate code and each developer had to become intimately aware of how fickle pecl/gearman was with exceptions.

I decided to fix pecl/gearman to properly trigger the exception callback in the event of an uncaught exception. The worker would still die, but now the client would be properly informed. This has to be done in two parts. I had to first update the pecl extension to detect when an exception occurred. The second part was sending back the proper status to the Gearman daemon so the client could be informed. This was a little more tricky than it first appeared. The Gearman daemon does not automatically handle the exception status. Both the client and the worker have to tell the Gearman daemon that they want to enable exceptions when they connect. Once I figured out how to do this using libgearman I was able to trigger the exception callback.

This exposed another issue with the way libgearman works. Because the exception handling is optional, the worker will send back a status of GEARMAN_WORK_FAIL even a status of GEARMAN_WORK_EXCEPTION was already sent. This means that the fail callback can be triggered after the exception callback. You can have the exception callback return a status of GEARMAN_WORK_EXCEPTION, but that will cause the entire batch process to stop. I am working with Brian Aker, who maintains libgearman, to see if something can be changed. Until then, I am required to keep track of tasks that threw an exception and check that list in the fail callback.

A few days ago Hudson began emailing alerts that a freight calculation unit test was failing in production. I emailed the developers responsible for the code and asked them to investigate this right away. The funny thing about unit tests failing in production is that developers can fall into a mindset of "the code is working on production, so there must not be a problem". The freight calculations did appear to be working correctly on production. Because we are so busy, no further investigation was done until yesterday.

Yesterday afternoon two developers realized that this failing unit test was the result of a serious regression in our "premium" freight calculation. This freight calculation type is most commonly used for furniture items and can easily calculate a shipping cost well over $100. This particular bug was calculating the "premium" freight costs as free. Now HauteLook's business model is flash sales. This means new sales events start at 8:00 am PT each morning and most of the business is done within the first few hours of the event. If there had been a furniture sale event within the past couple of days, HauteLook would have given away thousands of dollars worth of free shipping (or even tens of thousands if the event was large enough) before we would have noticed and fixed the issue. Shipping costs were not the only thing we saved either. The amount of time, energy and stress that it would have taken for the various departments to address this issue would have been high.

I saw an immediate change in the way my colleagues view unit tests. This was a clear indication that our investment has paid off. In this case we had well written unit tests that quickly alerted us to a problem before it costs us anything. I know next time a unit tests fails on production, HauteLook developers will not assume everything is working.