'Distributed' circuit breakers: co-ordinate or not? posted by Oliver Kamps @ Thu, 14 Feb 2008 08:55:53 -0000

Oliver Kamps — Thu, 14 Feb 2008 08:55:53 -0000

Hi Mike,

Thanks for sharing your thoughts so quickly on this.

Your reasoning makes a lot of sense to me; I was leaning in the same direction but hadn’t thought things through that far.

Cheers,
Oliver

Tim Ross released a C# circuit breaker posted by Michael Nygard @ Thu, 14 Feb 2008 05:46:03 -0000

Michael Nygard — Thu, 14 Feb 2008 05:46:03 -0000

Tim Ross has published his implementation of the Circuit Breaker pattern, complete with unit tests.

I barely speak C#, so I’m not in any position to review his implementation, but I’m delighted to see it!

See Tim’s Blog

'Distributed' circuit breakers: co-ordinate or not? posted by Michael Nygard @ Thu, 14 Feb 2008 05:40:54 -0000

Michael Nygard — Thu, 14 Feb 2008 05:40:54 -0000

Oliver,

The prospect of coordinating circuit breakers makes e very nervous, for a couple of reasons. Coordinating across the circuit breakers would either be done via point-to-point connections, which suggests a scalability problem, or via broadcast messages. (Broadcasts would not present a scalability problem.)

I see a second, slightly more subtle problem. Suppose a farm of 10 servers is calling a back end system. Due to an external event, maybe load-related, maybe human-error, whatever, disrupts the back end. If the first circuit breaker causes all 10 to stop making calls, then when the time limit passes, all 10 will start making calls at the same time. This can cause a surge in load on the newly restored back end system, exactly the same as when the power company restores service after a summertime blackout, only to have thousands of air conditioners turn on immediately. The surge in demand can destabilize the back end.

Finally, coordinating actions across the entire tier will tend to make the whole cluster react. If you assume that the circuit breaker trips only because of a problem with the back end, then communication between the circuit breakers would save a little bit of time. On the other hand, I have seen problems affect one host, disrupting its connection to the back end, thus causing circuit breakers to trip. If the problem is confined to that one host, you would not want the other circuit breakers to also flip open. (Imagine, for example, a failed NIC on that host. It still makes sense to protect the application by aborting the calls, even though it’s not strictly a problem with the back end system.)

So, in summary, I would choose not to have the circuit breakers communicate their status with each other.

Cheers,
-Mike Nygard

'Distributed' circuit breakers: co-ordinate or not? posted by Oliver Kamps @ Wed, 13 Feb 2008 16:14:48 -0000

Oliver Kamps — Wed, 13 Feb 2008 16:14:48 -0000

Hello everybody,

Assume a redundantly deployed component (e.g. the same EJB deployed to a number of Java EE server instances) has an integration point to an external system and I’d want to make this more robust using a Circuit Breaker. Would you have the different instances of the Circuit Breaker communicate with each other (i.e. if one flips open, they all do)? Or would this be overkill as they’d probably all flip open within a short period of time anyway if there’s a systemic problem?

Thanks a lot,

Oliver

Beginning to release some sample code posted by Michael Nygard @ Mon, 10 Dec 2007 05:07:42 -0000

Michael Nygard — Mon, 10 Dec 2007 05:07:42 -0000

I’ve decided to start putting sample code up at http://www.michaelnygard.com/. The first piece is a quick utility class to format java.util.logging files in a one-line, column-aligned format.

More to come. I’m thinking about working on an evil Test Harness next.

Assuring Availability of Administration Aspect posted by Michael Nygard @ Tue, 27 Nov 2007 06:25:00 -0000

Michael Nygard — Tue, 27 Nov 2007 06:25:00 -0000

Sean,

One of the problems with writing a book that works for many different products and vendors is that you have to treat some topics generally. Thread pool configuration is an area that varies wildly from one application server to another.

For instance, with ATG, the administration aspect is automatically served by a reserved pool of threads. Not so for Tomcat. In Tomcat, request handling threads are associated with individual Connectors. So, if all you want to do is reserve capacity for admin work, then a new connector is all you require.

It’s a bit more work to make sure that the admin context is completely separated from the application contexts. (As a security enhancement, that is.) To do that, you need to bind the admin Connector to the IP address of your administrative NIC, while also binding the application Connector to the IP address of your production NIC (or teamed NIC pair).

This still has the drawback that /admin resolves correctly via either IP address & port. That is to say, both Connectors can reach exactly the same set of contexts because they reside in the same Service. To really achieve complete separation, you will need to create a second Service with its own Engine, Host, and application base.

Of course, it’s a pretty good idea to separate the admin function from the application function anyway, so this isn’t a bad approach… but it’s not an easy one, either.

Needless to say, WebSphere, JBoss, and WebLogic each have their own completely different ways to configure bulkheads.

Cheers,
Michael T. Nygard
michael@michaelnygard.com
http://www.michaelnygard.com/

Author of “Release It!”
http://pragmaticprogrammer.com/titles/mnee/index.html

Assuring Availability of Administration Aspect posted by Sean Dockery @ Fri, 23 Nov 2007 08:01:19 -0000

Sean Dockery — Fri, 23 Nov 2007 08:01:19 -0000

Hello Michael.

In your book, you discuss how applications should allocate various request handling thread pools to separate the customer request handling from other request handling, with the intent that the application can still be controlled or queried for its status despite being pummeled by customers. You didn’t suggest how this goal is achieved in real life, however.

For example, if I want to deploy my application to Tomcat (which I believe is configured to allow 100 concurrent requests out of the box), how do I go about ensuring that certain request threads run against one aspect (customers) versus another (administration)? Can this be done while allowing both the customer and administration aspects of an application to exist in the same web component, or do I need to deploy separate web components under two (or more) Services declared in the server.xml? Do you recommend multiple Services, or multiple instances of Tomcat (i.e.: JVM’s) to mitigate the risk?

Thanks.

cicuit breakers posted by Mohamed Mansour @ Mon, 05 Nov 2007 18:43:59 -0000

Mohamed Mansour — Mon, 05 Nov 2007 18:43:59 -0000

Hello, it would be better to see it in Java, since we can use that code to write in other languages as well, since a oneliner in ruby wouldn’t be usefull to extend in other languages.

cicuit breakers posted by Dmitry Beransky @ Fri, 26 Oct 2007 23:39:26 -0000

Dmitry Beransky — Fri, 26 Oct 2007 23:39:26 -0000

Thanks for the reply, Mike. I was just wondering if there was anythgin out there before starting coding on my own. An open source project would be cool, though, especially if it’s in Java, then I could even try to contribute. If it ends up being in Ruby, not a problem either, I can always rewrite it in Java myself.

thanks
d.

cicuit breakers posted by Michael Nygard @ Mon, 22 Oct 2007 20:54:10 -0000

Michael Nygard — Mon, 22 Oct 2007 20:54:10 -0000

Dmitry,

The name “Circuit Breaker” is one that I coined, so I would not expect to find many references to it (yet.)

The existing implementations I’ve used are proprietary and are not something I can share. One of the common requests I’ve heard, however, is for some sample code. I am considering taking this on as an open source project.

What language would be most useful? I’m torn between doing an implementation in Ruby, where I might be able to use metaprogramming techniques to make using a circuit break a one-line declarative statement, versus doing it in Java, which has much more widespread applicability in the enterprise.

Cheers,
-Mike Nygard

cicuit breakers posted by Dmitry Beransky @ Sun, 21 Oct 2007 05:47:03 -0000

Dmitry Beransky — Sun, 21 Oct 2007 05:47:03 -0000

Mike,

you talk a lot about circuit breakers and it sounds like a very useful pattern, indeed. Are there any existing implementations? A cursory google search didn’t return any results. Perhaps, the pattern is also known under a different name?

Thanks
Dmitry

Recent Posts in Release It! | Pragmatic Forums

More detail on the OpsDB posted by denis krizanovic @ Tue, 19 Feb 2008 02:38:59 -0000