Recent Posts by Michael Nygard

Topic: Release It! / Tim Ross released a C# circuit breaker

Tim Ross has published his implementation of the Circuit Breaker pattern, complete with unit tests.

I barely speak C#, so I’m not in any position to review his implementation, but I’m delighted to see it!

See Tim’s Blog

Topic: Release It! / 'Distributed' circuit breakers: co-ordinate or not?

Oliver,

The prospect of coordinating circuit breakers makes e very nervous, for a couple of reasons. Coordinating across the circuit breakers would either be done via point-to-point connections, which suggests a scalability problem, or via broadcast messages. (Broadcasts would not present a scalability problem.)

I see a second, slightly more subtle problem. Suppose a farm of 10 servers is calling a back end system. Due to an external event, maybe load-related, maybe human-error, whatever, disrupts the back end. If the first circuit breaker causes all 10 to stop making calls, then when the time limit passes, all 10 will start making calls at the same time. This can cause a surge in load on the newly restored back end system, exactly the same as when the power company restores service after a summertime blackout, only to have thousands of air conditioners turn on immediately. The surge in demand can destabilize the back end.

Finally, coordinating actions across the entire tier will tend to make the whole cluster react. If you assume that the circuit breaker trips only because of a problem with the back end, then communication between the circuit breakers would save a little bit of time. On the other hand, I have seen problems affect one host, disrupting its connection to the back end, thus causing circuit breakers to trip. If the problem is confined to that one host, you would not want the other circuit breakers to also flip open. (Imagine, for example, a failed NIC on that host. It still makes sense to protect the application by aborting the calls, even though it’s not strictly a problem with the back end system.)

So, in summary, I would choose not to have the circuit breakers communicate their status with each other.

Cheers,
-Mike Nygard

Topic: Release It! / Beginning to release some sample code

I’ve decided to start putting sample code up at http://www.michaelnygard.com/. The first piece is a quick utility class to format java.util.logging files in a one-line, column-aligned format.

More to come. I’m thinking about working on an evil Test Harness next.

Topic: Release It! / Assuring Availability of Administration Aspect

Sean,

One of the problems with writing a book that works for many different products and vendors is that you have to treat some topics generally. Thread pool configuration is an area that varies wildly from one application server to another.

For instance, with ATG, the administration aspect is automatically served by a reserved pool of threads. Not so for Tomcat. In Tomcat, request handling threads are associated with individual Connectors. So, if all you want to do is reserve capacity for admin work, then a new connector is all you require.

It’s a bit more work to make sure that the admin context is completely separated from the application contexts. (As a security enhancement, that is.) To do that, you need to bind the admin Connector to the IP address of your administrative NIC, while also binding the application Connector to the IP address of your production NIC (or teamed NIC pair).

This still has the drawback that /admin resolves correctly via either IP address & port. That is to say, both Connectors can reach exactly the same set of contexts because they reside in the same Service. To really achieve complete separation, you will need to create a second Service with its own Engine, Host, and application base.

Of course, it’s a pretty good idea to separate the admin function from the application function anyway, so this isn’t a bad approach… but it’s not an easy one, either.

Needless to say, WebSphere, JBoss, and WebLogic each have their own completely different ways to configure bulkheads.

Cheers,
Michael T. Nygard
michael@michaelnygard.com
http://www.michaelnygard.com/

Author of “Release It!”
http://pragmaticprogrammer.com/titles/mnee/index.html

Topic: Release It! / cicuit breakers

Dmitry,

The name “Circuit Breaker” is one that I coined, so I would not expect to find many references to it (yet.)

The existing implementations I’ve used are proprietary and are not something I can share. One of the common requests I’ve heard, however, is for some sample code. I am considering taking this on as an open source project.

What language would be most useful? I’m torn between doing an implementation in Ruby, where I might be able to use metaprogramming techniques to make using a circuit break a one-line declarative statement, versus doing it in Java, which has much more widespread applicability in the enterprise.

Cheers,
-Mike Nygard