Processing only one copy of a message
In integration, there often are situations when the same message is delivered more than once. Perhaps someone hit a re-send button, perhaps there was a failure in communication and all the messages were re-sent (even the ones that originally made it through) or perhaps some process decided to keep sending the same message. Whatever caused it, there are instances when processing this message multiple times could be dangerous. Think of a credit card payment, for instance. This post examines how to avoid this situation in Mule.
Whenever lecturing about this pattern, I sometimes refer to it as the Highlander principle expecting to get a laugh (or a smile!) but oddly, no one does. The point is that there can only one copy of the message should be allowed and any other copies should be rejected.
The grand name for this pattern is idempotency and Mule helpfully provides an idempotent inbound router for you. This is used as follows:
<service name="idempotent-router-service"> <inbound> <vm:inbound-endpoint path="for.idempotent"/> <idempotent-receiver-router/> </inbound> <outbound> <pass-through-router> <outbound-endpoint ref="ToTestCase"/> </pass-through-router> </outbound> </service>
This simple example will read information from the inbound VM endpoint and will bridge them to the outbound endpoint provided that they are unique. The router examines the message id on the MuleMessage and uses that as a unique identifier to prevent any other copies of the same message from being processed. If the same message is received twice, the idempotent router rejects it.
Consider, therefore, this test case:
public void testIdempotent () { MuleMessage aReply = null, aReply2 = null; MuleMessage myMessage = new DefaultMuleMessage("My Message"); try { MuleClient myClient = new MuleClient (); myClient.dispatch("vm://for.idempotent", myMessage); myClient.dispatch("vm://for.idempotent", myMessage); aReply = myClient.request("ToTestCase", 5000); aReply2 = myClient.request("ToTestCase", 4000); } catch (MuleException e) { fail (e.getDetailedMessage()); } assertNotNull (aReply); assertNotNull (aReply.getPayload()); assertTrue (aReply.getPayload() instanceof String); assertEquals ("My Message", (String) aReply.getPayload()); assertNull (aReply2); }
As you can see, here we send the same MuleMessage to the service twice and the second one is rejected. Clearly, the message ID is unique but since the ID is tied to the transport, if the transport itself does not support the concept of message IDs, then the idempotent router will not work as expected.
In these situations we need to be able to tell the idempotent router that something else should be used as an ID. Perhaps something in the payload can be used or maybe a property can flag uniqueness.
Using some sort of custom ID is quite useful since you can exert greater control over what constitutes idempotency. We can configure the idempotent router to look for something other than the Message ID like so:
<idempotent-receiver-router idExpression="#[headers:Custom-ID]"/>Here, the idempotent router will look for a MuleMessage property called Custom-ID and will use the value of that property as the unique identifier.
(Note that this expression is case sensitive!)
If we then change our previous test case like so:
myMessage.setStringProperty("Custom-ID", "Message-1"); myClient.dispatch("forIdempotent", myMessage); myMessage.setStringProperty("Custom-ID", "Message-2"); myClient.dispatch("forIdempotent", myMessage);
we have a test case which will send the same message to the endpoint twice. Here, we want both messages to be processed though, so we’re setting the Custom-ID property to different values before dispatching it.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Tags: expressions, idempotent-receiver-router, inbound router
March 8th, 2010 at 1:44 PM
Hi, I’ve just found your blog and it’s gonna be really helpful. I’m particularly interested in the idempotent-receiver-router as it perfectly fits one of our requirements. But I wonder about its performance. It seems that Mule stores in a file every MessageID it’s processed so it seems reasonable to think that after having processing several thousands messages, performance will be severely affected… Do you know how is this managed by Mule?
Thanks
March 9th, 2010 at 9:42 AM
Hello Pablo,
Glad to hear that the blog is going to helpful – that’s the point!
Mule’s Idempotent router uses an ObjectStore to handle those IDs and you can change this to be an in-memory one if you need to. Check out the source of the router and you’ll see what I mean.
Thanks for reading
Antoine