Enterprise Integration Zone is brought to you in partnership with:

Felipe Gaúcho works as senior software engineer at Netcetera AG in Switzerland. He is a well known Brazilian JUG leader and open-source evangelist. Felipe works with Java since its early versions and has plans to keep that Java tradition as it is. When he is not coding, he prefers to listen reggae and travel around with his lovely wife Alena and his son Rodrigo. Felipe is a DZone MVB and is not an employee of DZone and has posted 29 posts at DZone. View Full User Profile

Handling Poison Messages with Glassfish

09.24.2009
| 11997 views |
  • submit to reddit

Poison messages are basically delivery deadlocks caused by a continuous redelivery of a message to a JMS Queue or Topic. This usually happens due to a code bug or configuration problems in the project.

How to reproduce poison messages

The easiest way of reproducing the poison messages issue is to create a Message Driven Bean and then to throw an exception in its onMessage method, like the example below.

@MessageDriven(activationConfig = {@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")}, mappedName = "MyQueue")
public class RegistrationMessageBean implements MessageListener {
@Override
public void onMessage(Message registration) {
throw new RuntimeException("poison message");
}
}


Fixing the JMS deadlock

What happens is that JMS relies on transactions to guarantee that all messages in a Queue will be delivered despite any temporary problems in the message consumer. If a message consumer (MDB) throws an exception, the JMS server tries to redeliver the message (transaction rollback). Only if a method consuming the message finishes without error the transaction will be committed and then the message will be removed from the Queue (acknowledged). In the sample above, the message consumer will always throw an exception and due to that the server will always redeliver the message - a deadlock.

The workarounds to fix such a problem are:

  1. Fix all bugs from your code: that's the general best solution for poison messages, but as you know bugs are intrinsic to any software and it is not a surprise the time you will loose fighting against the poison messages :)
  2.  Try-Catch and digest the exceptions: for the paranoid, a good choice is to surround the whole onMessage code with a try{...} catch(Exception e){ ... }. Even if you strongly believe your code is sound, it is a recommended practice to do that. So, rewriting our sample code in a safe way, it looks like that:

 

      @Override
public void onMessage(Message registration) {
try {
throw new RuntimeException("poison message");
} catch(Exception error) {
logger.severe("I am ignoring the JMS exception: " + e.getMessage());
}
}


The second solution is a robust way of guaranteeing the consume of a message despite any problems. Not so elegant, but without exceptions in the onMessage method, the Message will be acknowledged and the JMS transaction will be committed.

A more complicated scenario with sub-transactions

The previous solution works for the general case, a simple Java code inside the onMessage method. Problem is, JMS uses the Java Transaction API (JTA) to control the message transactions and the JTA API supports sub-transactions. So imagine if the onMessage calls a JPA transactional method:

@PersistenceUnit(name = "arenapuj")
protected EntityManagerFactory emf;

@Override
public void onMessage(Message registration) {
try {
create(new MyJpaEntity());
} catch(Exception error) {
logger.severe("I am ignoring the JMS exception: " + e.getMessage());
}
}

public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}



The robust try-catch block is still there but guess what: if the method JPA Transaction of the method create is rolled back, the JMS transaction is also rolled back, causing the poison message problem despite the try-catch block on the onMessage code. So, the comfortable robustness provided by the try-catch block is actually a trap, a silent killer in the JMS sub-transactions scenario.

 

How to avoid poison messages caused by sub-transactions?

You should annotate the method with the first sub-transaction with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) to avoid the dependencies between the JMS Transaction and its sub-transactions. Notice that only the first sub-transaction need to be decoupled from the JMS one to avoid poison messages.

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}


Done, now your JMS method is decoupled from the new JPA transaction and if the JPA code rolls back, the JMS message will be acknowledged anyway.

Disclaimer: I am abusing the terms "JMS transaction" and "JPA transactions" for the sake of clarification here. Actually there is no such things, we have only one the transaction types defined in the Java Transaction API (JTA). For me it seems simpler to visualize the problem thinking about the methods and its transactions scope separately but after all it is all about JTA Transactions :)


Handling problems outside the code

Other common JMS scenario is to have resources problems, like connection failures, database down, etc. If we have a message consumer down and a message producer working, the producer will try to send a message, fail and try to send the message again. It would cause another type of deadlock, but the Java EE containers provide a set of configuration options to prevent such problems. The ActivationSpec for Message Driven Beans specifies two annotations to workaround activating problems:

  1. endpointExceptionRedeliveryAttempts: Number of times to redeliver a message when MDB throws an exception during message delivery
  2. sendUndeliverableMsgsToDMQ: Place message in dead message queue when MDB throws a runtime exception and number of redelivery attempts exceeds the value of endpointExceptionRedeliveryAttempts? If false, the Message Queue broker will attempt redelivery of the message to any valid consumer, including the same MDB.


You may check these configurations in your code review, but it is not so critical since its default values are reasonable for the common scenarios. And you need to know container specific attributes. The Glassfish V3 activation properties changed a bit, and I suppose for the other containers we will find different names. Fortunately we can expect that all defaults works fine and we are not forced to dig in product-specific details all the time.

 

Summary

JMS is one of the most powerful Java EE resources available for developers and architects, but it is very important that anyone designing such applications to know in deep the specification and also some implementation tricks. Poison messages can make your server and eventually the whole host machine to hang for hours - forcing a machine restart. It is a common problem, in my opinion weakly supported by the containers and a problem we should know about. The goal of this blog entry is just to give you a chance to identify the poison messages problem of your application. to know more about JMS and its details, you need to read more and the links below seems to be a good starting point:

  •     Creating Robust JMS Applications - The Java EE 5 Tutorial
  •     JMS Messaging Using GlassFish by Deepa Sobhana
  •     Bitter Messages: Java* Messaging Anti-Patterns by Bruce A. Tate
  •     Thread: how can i setup glassfish to deal with "poison message"


Aknowledgment: a special thanks for Marina Vatkina and Nigel Deakin for their friendly support through the Glassfish's mailing list. And a kudo to "The Professor" and his wise hints on this topic.

References
Published at DZone with permission of Felipe Gaúcho, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)