Felipe Gaúcho works as senior software engineer at Netcetera AG in Switzerland. He is a well known Brazilian JUG leader and open-source evangelist. Felipe works with Java since its early versions and has plans to keep that Java tradition as it is. When he is not coding, he prefers to listen reggae and travel around with his lovely wife Alena and his son Rodrigo. Felipe is a DZone MVB and is not an employee of DZone and has posted 29 posts at DZone. View Full User Profile

Handling Poison Messages With Glassfish

09.25.2009
| 3861 views |
  • submit to reddit

Poison messages are basically delivery deadlocks caused by a continuous redelivery of a message to a JMS Queue or Topic. That usually happens due to a code bug or configuration problems in the project.

How to reproduce poison messages

The easiest way of reproducing the poison messages issue is to create a Message Driven Bean and then to throw an exception in its onMessage method, like the example below.

@MessageDriven(activationConfig = {@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")}, mappedName = "MyQueue")
public class RegistrationMessageBean implements MessageListener {
@Override
public void onMessage(Message registration) {
throw new RuntimeException("poison message");
}
}

Fixing the JMS deadlock

What happens is that JMS relies on transactions to guarantee that all messages in a Queue will be delivered despite any temporary problems in the message consumer. If a message consumer (MDB) throws an exception, the JMS server tries to redeliver the message (transaction rollback). Only if a method consuming the message finishes without error the transaction will be committed and then the message will be removed from the Queue (acknowledged). In the sample above, the message consumer will always throw an exception and due to that the server will always redeliver the message - a deadlock.

The workarounds to fix such a problem are:

  1. Fix all bugs from your code: that's the general best solution for poison messages, but as you know bugs are intrinsic to any software and it is not a surprise the time you will loose fighting against the poison messages :)
  2. Try-Catch and digest the exceptions: for the paranoid, a good choice is to surround the whole onMessage code with a try{...} catch(Exception e){ ... }. Even if you strongly believe your code is sound, it is a recommended practice to do that. So, rewriting our sample code in a safe way, it looks like this:
    @Override
    public void onMessage(Message registration) {
    try {
    throw new RuntimeException("poison message");
    } catch(Exception error) {
    logger.severe("I am ignoring the JMS exception: " + e.getMessage());
    }
    }

The second solution is a robust way of guaranteeing the consume of a message despite any problems. Not so elegant, but without exceptions in the onMessage method, the Message will be acknowledged and the JMS transaction will be committed.

A more complicated scenario with sub-transactions

The previous solution works for the general case, a simple Java code inside the onMessage method. Problem is, JMS uses the Java Transaction API (JTA) to control the message transactions and the JTA API supports sub-transactions. So imagine if the onMessage calls a JPA transactional method:

@PersistenceUnit(name = "arenapuj")
protected EntityManagerFactory emf;

@Override
public void onMessage(Message registration) {
try {
create(new MyJpaEntity());
} catch(Exception error) {
logger.severe("I am ignoring the JMS exception: " + e.getMessage());
}
}

public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}

The robust try-catch block is still there but guess what: if the method JPA Transaction of the method create is rolled back, the JMS transaction is also rolled back, causing the poison message problem despite the try-catch block on the onMessage code. So, the comfortable robustness provided by the try-catch block is actually a trap, a silent killer in the JMS sub-transactions scenario.

How to avoid poison messages caused by sub-transactions?

You should annotate the method with the first sub-transaction with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) to avoid the dependencies between the JMS Transaction and its sub-transactions. Notice that only the first sub-transaction needs to be decoupled from the JMS one to avoid poison messages.

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public MyJpaEntity create(final MyJpaEntity entity) throws Exception {
EntityManager manager = emf.createEntityManager();
try {
manager.persist(entity);
manager.flush();
return entity;
} finally {
if (manager != null && manager.isOpen()) {
manager.close();
}
}
}

Done, now your JMS method is decoupled from the new JPA transaction and if the JPA code rolls back, the JMS message will be acknowledged anyway.

Disclaimer: I am abusing the terms "JMS transaction" and "JPA transactions" for the sake of clarification here. Actually there are no such things, we have only the transactions defined in the Java Transaction API (JTA). For me it seems simpler to visualize the problem thinking about the methods and their transactions scope separately but after all it is all about JTA Transactions :)

Handling problems outside the code

Other common JMS scenario is to have resources problems, like connection failures, database down, etc. If we have a message consumer down and a message producer working, the producer will try to send a message, fail and try to send the message again. It would cause another type of deadlock, but the Java EE containers provide a set of configuration options to prevent such problems. The ActivationSpec for Message Driven Beans specifies two annotations to workaround activating problems:

  1. endpointExceptionRedeliveryAttempts: Number of times to redeliver a message when MDB throws an exception during message delivery
  2. sendUndeliverableMsgsToDMQ: Place message in dead message queue when MDB throws a runtime exception and number of redelivery attempts exceeds the value of endpointExceptionRedeliveryAttempts? If false, the Message Queue broker will attempt redelivery of the message to any valid consumer, including the same MDB.

You may check those configurations in your code review, but it is not so critical since their default values are reasonable for the common scenarios. And you need to know container specific attributes. The Glassfish V3 activation properties changed a bit, and I suppose for the other containers we will find different names. Fortunately we can expect that all defaults work fine and we are not forced to dig in product-specific details all the time.

Summary

JMS is one of the most powerful java EE resources available for developers and architects, but it is very important that anyone designing such applications knows in deep the specification and also some implementation tricks. Poison messages can make your server and eventually the whole host machine to hang for hours - forcing a machine to restart. It is a common problem, in my opinion weakly supported by the containers and a problem we should know about. The goal of this blog entry is just to give you a chance to identify the poison messages problem of your application. to know more about JMS and its details, you need to read more and the links below seem to be a good starting point:

Aknowledgment: a special thanks for Marina Vatkina and Nigel Deakin for their friendly support through the Glassfish's mailing list. And a kudo to "The Professor" and his wise hints on this topic.

From http://weblogs.java.net/blog/felipegaucho

Published at DZone with permission of Felipe Gaúcho, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Naveen Pudari replied on Wed, 2011/04/27 - 9:14am

hi , we are using JMS Message Listner and some times my message listner is not calling OnMessage() --- we are doing some business login in OnMessage method and we are calling one of the WCTP URL in ON MESSAGE to send some business logic. here some times we are facing no response form WCTP site.. this casue message queue gets hang? method. we are facing lot of issues. could you please see below code here iam creating Queue connection and Session one time at GatewayServlet init() method flow is GatewayServlet init()--> calls -->GatewayMessageReceiver init() method when GatewayServlet loads into sun java applicaiton server or deployed into sun java app server. then init() method in GatewayMessageReceiver class creates jms session and queue connection. here GatewayMessageReceiver implements Message listner class... here problem is onMessage() is not calling for some times, when i do restart server its calling onMessage(). but it should call when ever message arrives in Queue, its not happning and no Error or Exception thrown. please find below code import java.util.; import java.io.; import java.sql.*; import javax.servlet.; import javax.servlet.http.; public class GatewayServlet extends HttpServlet { private GatewayMessageReceiver receiver = null; /** Initializes the servlet. */ public void init(ServletConfig config) throws ServletException { super.init(config); receiver = new GatewayMessageReceiver(); //here iam calling my GatewayMessageReceiver for JMS connection creations info(""+receiver); } /** Destroys the servlet */ public void destroy() { if (receiver != null) { receiver.destroy(); } } protected void processGatewayRequest(ServletRequest request, ServletResponse response) throws ServletException, java.io.IOException { //doing some business logic } protected void processRequest(ServletRequest request, ServletResponse response) throws ServletException, java.io.IOException { CCMLogger.getGatewayLogger(GeneralConfigurator.getInstance().getUtility()).debug("Host sending request is:"+request.getRemoteHost()); //check whether it's a push request processGatewayRequest(request, response); } /** Handles the HTTP POST method. * @param request servlet request * @param response servlet response */ public void service(ServletRequest request, ServletResponse response) throws ServletException, java.io.IOException { processRequest(request, response); } public void doGet(ServletRequest request, ServletResponse response) throws ServletException, java.io.IOException { service(request, response); } public void doPost(ServletRequest request, ServletResponse response) throws ServletException, java.io.IOException { service(request, response); } JMS MESSAGE LISTNER IS mport javax.jms.; import java.util.logging.; import com.carrier.ccm.business.; import com.carrier.ccm.gateway.service.; import com.carrier.ccm.gateway.config.; import com.carrier.ccm.service.; import com.carrier.ccm.logging.; import com.carrier.ccm.util.; import com.carrier.ccm.exception.; /* * * @author Administrator */ public class GatewayMessageReceiver implements MessageListener { private QueueConnection connection = null; /** Creates a new instance of GatewayMessageReceiver */ public GatewayMessageReceiver() { super(); init(); } private void init() { QueueSession session = null; QueueReceiver queueReceiver = null; try{ String queueName = "infoQueue";//its sun java app sever queue name String qcfName = "infoQueueCF";//connectionfactory created in sun java app sever Logger.log.log(Level.INFO, "Queue name: "+queueName); Logger.log.log(Level.INFO, "Queue CF name: "+qcfName); QueueConnectionFactory qcf = (QueueConnectionFactory)JndiUtilities.get(qcfName); Logger.log.log(Level.INFO, "Queue CF: "+qcf); Queue queue = (Queue)JndiUtilities.get(queueName); Logger.log.log(Level.INFO, "Queue: "+queue); // Creating a QueueConnection to the Message service"); connection = qcf.createQueueConnection(); // Creating a session within the connection session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE); // Creating a QueueReceiver queueReceiver = session.createReceiver(queue); // setting up a message listener queueReceiver.setMessageListener(this); //Starting the Connection connection.start(); } catch (Throwable t) { Logger.log(Level.SEVERE, "Failed to start queue listener for business messages", t); } } public void destroy() { try { if (connection != null) { connection.close(); } } catch (Throwable t) { Logger.log(Level.SEVERE, "Failed to close queue connection", t); } } public void onMessage(javax.jms.Message message) { String ut = null; try { String utm = message.getStringProperty(IConstants.UTILITY_TAG); int bcDelay = message.getIntProperty(IConstants.BC_DELAY); //it must be an ObjectMessage! ObjectMessage omsg = (ObjectMessage)message; //Here iam doing business logic } catch (Throwable t) { Logger.log(Level.SEVERE, "Failed to process business message", t); } } } THE JNDI UTILITIES CLASS mport javax.naming.; import javax.sql.; /** * * @author Administrator */ public class JndiUtilities { private static Context context = null; static { setJndiContext(); } /** Creates a new instance of JndiUtilities */ private JndiUtilities() { super(); } private static void setJndiContext() { try { context = new InitialContext(); } catch (Exception e) { System.err.println("ERROR getting JNDI context: "+e); } } public static Object get(String name) { if (context == null) { setJndiContext(); if (context == null) return null; } Object obj; try { obj = context.lookup(name); } catch (Exception e) { obj = null; System.err.println("ERROR getting JNDI resource named \""+name+"\": "+e); } return obj; } } could you please help me in this case Thanks Naveen

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.