Enterprise Integration Zone is brought to you in partnership with:

Christian is a Principal Middleware Specialist/Architect at Red Hat specializing in developing enterprise software applications with an emphasis on software integration and messaging. His strengths include helping clients build software using industry best practices, Test Driven Design, ActiveMQ,Apache Camel, ServiceMix, Spring Framework, and most importantly, modeling complex domains so that they can be realized in software. He works primarily using Java and its many frameworks, but his favorite programming language is Python. He's in the midst of learning Scala and hopes to contribute to the Apache Apollo project. Christian is a DZone MVB and is not an employee of DZone and has posted 50 posts at DZone. You can read more from them at their website. View Full User Profile

Top Posts of 2013: Apache Camel vs. Spring Integration

12.30.2013
| 16666 views |
  • submit to reddit

First of all, for full disclosure, for the last 1.5 years, I’ve been working as a consultant for FuseSource (now Red Hat) supporting SOA and integration projects for large and small companies in diverse verticals from retail, shipping, banking/finance, health, etc. My specialty has been architecting solutions with high scalability and throughput demands using some of the best open-source projects in this space: Apache ActiveMQ, Apache Camel, Apache ServiceMix, Apache CXF, et al.

I am the author of one of the DZone Refcardz for Apache Camel titled “Apache Camel Essential Components” as well as the speaker for the same titled webcast. I’m also a technical reviewer for an upcoming book on Apache Camel for Packt Publishing, which I hope to soon post about publicly.

Needless to say, I’ve done some work with Apache Camel, I contribute to it, and I encourage people to check it out.

But before I got started with FuseSource, I had experiences with other middleware integration projects — some open-source, some commercial — but my first foray with lightweight open-source integration projects was not with Apache Camel.

I stumbled upon Spring Integration back in 2009 and really liked it.

I wrote blogs about Spring Integration, contributed to the user forums, contributed bug fixes and examples, and stayed a little active in the community. I have tremendous respect for Mark, Oleg, Gary, Gunnar, et. al. for their amazing work on Spring Integration.

I liked how it extended Spring and how closely it implemented the Enterprise Integration Pattens as cataloged by Gregor Hohpe and Bobby Wolf. I used it on a few projects back then, and I firmly believe you don’t get a real feel for a project unless you’ve used it on a real project to solve real problems. But since I’ve had the opportunity to use both projects deeply enough, I would like to contribute to the blogosphere my observations and opinions :)

First, there are quite a few write ups comparing the two projects. Some focus on stats like community activity or #s of components. While relevant to some extent, I’d like to go a tiny bit deeper. So the following blog post is my thoughts about Apache Camel and Spring Integration, and (spoiler alert!) why I would choose Camel for integration projects.

Abstraction, Abstraction, Abstraction

Both projects aim to fill similar needs: light-weight integration library with full implementations of EIPs, for mediation and routing, as well as provide abstractions on top of commonly used technologies to interface with external systems. Some common technologies include Web Services (SOAP/REST), JMS or asynchronous messaging, File-based batch systems, TCP sockets, FTP/SFTP, RDBMS and an many others. Basically whenever two heterogeneous systems need to interact, either synchronously or asynchronously, and exchange data, a light-weight integration library could help you significantly. For those of you asking “isn’t that what an ESB does?”.. well… kinda… but I can write about that another time.

For those of us integrating systems, we find out two things really quickly about integration. #1 it’s not sexy and #2 it’s difficult.
Understanding the differences in the systems, both technical as well as semantic and how it’s used to implement business functions is crucial. The code that you do end up writing should be focused on those business concerns and you should leverage existing commodity components where possible. Writing code to interact with the file system, or send messages to queues is definitely considered commodity code and just adds to the code base that you’ll have to maintain. If you code it by hand/roll your own, you also accept the risk of introducing bugs into areas that are not crucial to the business logic or business semantics. In my humble opinion, that’s an unnecessary risk.

So that’s where Apache Camel or Spring Integration come in. They help make integration projects easier by providing access to commodity components that have been vetted by the community, as well as implement common EIPs like transformations, routing, filtering, splitting/aggregating, etc… again.. vetted by the community. If you code these things by hand, good luck :) Of course it can be done, people do it all the time, but it’s an unnecessary risk for most projects.

Where was I….

Yes, so you can think of either of these two projects as “abstractions” above specific technologies. This abstract notion allows you to focus on “what the integration is supposed to do”. And in an effort to solve this question, I believe Apache Camel shines.

Domain Specific Languages

Integrations are read FAR MORE often than they’re written .. just the same as code. So just like you aspire to write nice clean code with short functions, descriptive variables, proper levels of data hiding, etc, etc, the same is true with coding for integration projects but taken to the next level. You want to code it so you can read it and understand it. Though with integration code, the full meaning and intent of “what is being integrated” ends up lost in piles of code even if your code is written neatly. You can leverage an integration abstraction library, but as good as they are for the purposes described above, what you want is clarity to be able to answer “what is being integrated”.

Apache Camel brings to the table a Domain Specific Language and I believe this is a big key difference between the two projects. Using the DSL, you can very concisely and cleanly express “what is being integrated”, even in the face of moderate to complex integrations which is where other projects begin to lack clarity. Apache Camel offers DSLs in Java (my favorite), Spring XML, Scala, Blueprint-OSGI, Groovy.. and for what it’s worth, even Kotlin :)

Using the DSL, you build integration “Routes” with constructs like “from” “split” “choice” and “to”. These allow you to specify in common language “what the integration is doing.”

When I was using Spring Integration, they didn’t have a DSL. You dealt directly with the Channels and Components (pipes and filters) using Spring XML and Channels were the main abstraction. They were working on a Scala DSL back then, but as much as I love Scala, it’s still not very widely used, so for the purposes of using it on integration projects in common language or XML, there weren’t many choices. Basically you hooked together endpoints like jms-gateway or http-gateway with channels and connected the other end of the channel to EIPs like Splitter or Router. But for every component you wanted to connect up, you also had to pay attention to what Channel you used what its inputs were and what outputs it had.

And once your project starts to grow, you find an explosion in channel objects. And it ends up diluting the meaning of the integration.

For example, take a look at the Cafe Spring Integration Example. This example isn’t super complicated, but it illustrates what I ran into. Imagine splitting channels and components among many Spring context files and you can see how this can get quite messy.

<int:gateway id="cafe" service-interface="org.springframework.integration.samples.cafe.Cafe"/>
 
<!-- each order has a collection of order items that is split apart to be processed -->
<int:channel id="orders"/>
<int:splitter input-channel="orders" expression="payload.items" output-channel="drinks"/>
 
<!-- The router sends different drink orders on different paths -->
<int:channel id="drinks"/>
<int:router input-channel="drinks" expression="payload.iced ? 'coldDrinks' : 'hotDrinks'"/>
 
<!-- individual order items are processed by the barista -->
<int:channel id="coldDrinks">
<int:queue capacity="10"/>
</int:channel>
<int:service-activator input-channel="coldDrinks" ref="barista" method="prepareColdDrink" output-channel="preparedDrinks"/>
 
<!-- individual order items are processed by the barista -->
<int:channel id="hotDrinks">
<int:queue capacity="10"/>
</int:channel>
<int:service-activator input-channel="hotDrinks" ref="barista" method="prepareHotDrink" output-channel="preparedDrinks"/>
 
<!-- drink order items are aggregated in a call to the waiter -->
<int:channel id="preparedDrinks"/>
<int:aggregator input-channel="preparedDrinks" method="prepareDelivery" output-channel="deliveries">
<beans:bean class="org.springframework.integration.samples.cafe.xml.Waiter"/>
</int:aggregator>
 
<int-stream:stdout-channel-adapter id="deliveries"/>
 
<beans:bean id="barista" class="org.springframework.integration.samples.cafe.xml.Barista"/>
 
<int:poller id="poller" default="true" fixed-delay="1000"/>

Here is how the exact same example is implemented with Apache Camel:

public void configure() {

  from("direct:cafe")
    .split().method("orderSplitter")
    .to("direct:drink");
  from("direct:drink").recipientList().method("drinkRouter");
  from("seda:coldDrinks?concurrentConsumers=2")
    .to("bean:barista?method=prepareColdDrink")
    .to("direct:deliveries");
  from("seda:hotDrinks?concurrentConsumers=3")
    .to("bean:barista?method=prepareHotDrink")
    .to("direct:deliveries");
  from("direct:deliveries")
    .aggregate(new CafeAggregationStrategy()).method("waiter", "checkOrder").completionTimeout(5 * 1000L)
    .to("bean:waiter?method=prepareDelivery")
    .to("bean:waiter?method=deliverCafes");
 
}

Of course, I don’t expect one to be able to pick up exactly what each example is doing without knowing or have used each of the respective libraries, but without a doubt I find reading the expressive DSL represented in the Camel DSL to be easier vs trying to interpret the SI route/flow with all of its channels. This gets me closer to “what is this integration doing” without being bogged down by details.

Testing

Another very important part to writing integration flows is testing. I won’t go on my usual rant about how you’d be a fool not to test your code, but even doubly so for not testing your integrations. As complex as they can get, and the mediation that’s involved, you want to verify you’ve got it working!!

At the time, Spring Integration didn’t have any good practices for testing routes specifically. Presumably because they already have a generic test framework for Spring itself. You can get by with this for sure, because we’re all used to writing unit test or integration tests with Spring and mocking out collaborators with EasyMock or Mockito, right? And if you wanted any extra capabilities you’d have to build them out yourself.

But with Apache Camel, rich testing support comes out of the box, and it’s easy enough to use as to encourage testing your routes. Camel’s testing framework is built on top of some of the Spring Test features, so it ends up being the best of both worlds. For example, you can mock out parts of a route that hit live systems and focus on the routing and mediation capabilities. You can set expectations and assert behaviors just like you can with Mockito, but it’s built right into the library and adds additional capabilities. A colleague of mine, David Valeri has an excellent Testing and Debugging blog as well as his talk at CamelOne

For example, in the following snippet, we can assert that we’ve received two messages at the mock endpoint within a period of 5000ms:

MockEndpoint resultEndpoint = context.resolveEndpoint("mock:foo", MockEndpoint.class);
resultEndpoint.setAssertPeriod(5000);
resultEndpoint.expectedMessageCount(2);
// send some messages
...
// now lets assert that the mock:foo endpoint received 2 messages
resultEndpoint.assertIsSatisfied();


You can even mock out live endpoints by applying AOP advice simply like this:

 context.getRouteDefinitions().get(0).adviceWith(context, new AdviceWithRouteBuilder() {
  @Override
    public void configure() throws Exception {
      // mock sending to direct:foo and direct:bar and skip send to it
      mockEndpointsAndSkip("direct:foo", "direct:bar");
    }
});


Along with the mock components, you also get functionality to assist in testing Blueprint XML if you’re doing OSGI without having to deploy to a container, you can do soak tests or stress tests with the DataSet component, and many other complex testing scenarios.

Check out these links for more:

Community

Lastly, but certainly not any less importantly, I would like to point out that Apache Camel has a vibrant, diverse community made up of contributors from all around the world at different companies working on different projects. Upon writing to the mailing list or submitting a JIRA, it’s not uncommon to get responses within minutes. This same ecosystem, which you would expect to find at a non-profit, core opensource foundation like The Apache Software Foundation isn’t exactly the same thing you’d find at other projects.

I certainly am not saying the Spring Integration forums and JIRAs aren’t lively, but I perceive there to be more activity from non-project or Company (VMWare/SpringSource) personnel in the Apache Camel project. In my opinion that tends to fosters more creativity, more outsiders to contribute and make the project better, etc, etc.. the core of open-source and open community development. Of course that comes with more arguments :) but the net result is very positive.

In some of the other posts about Spring Integration vs Camel, the authors point to how many more components Camel has than Spring Integration. There is a point to be made there in terms of larger options for connectivity, but I think the more important point is that a lot of those were contributed by non-core Camel developers. Additionally, you can try doing a quick search on GitHub and you’ll find people who write Camel components that haven’t even been brought to the Apache community, but that you can still leverage.

Which one should YOU use?

So I have spewed my opinions in almost never-ending prose here, but I encourage you to check out BOTH projects and decide for yourself. Others will have different experiences and opinions, and to some degree that will play a large role in deciding which you use for your next integration projects.

A lot of the above was my opinion. But if I’ve misstated something, or misrepresented something, please comment and correct me!



Published at DZone with permission of Christian Posta, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Sachin Badoni replied on Fri, 2013/10/04 - 8:29am

Nice post.... 

I've used both of these frameworks. They both can do similar job and very hard to choose one. But as far as Unit Testing goes, it is much easier to test with Camel than SI. And of course DSL keeps the code cleaner and easy to read.

On the other hand SI provides you better control over message queue, their concurrent consumption.





Grzegorz Borkowski replied on Wed, 2013/10/09 - 4:30pm

We used Camel "test framework" intensively to test our application, but found out it has one serious limitation: to make tests repeatable, you have to add some sleep time at the end of every test. Basically, our test approach was: feed some input data into the Camel-based system, and scan outputs to assert the produced results are correct. The problem is, if you use any async endpoints/components, you will never know when the processing really ended. So to make sure you don't miss any delayed message, you have to wait relatively long time at the end of each test. The net effect is that running a suit of tests starts taking longer and longer and longer the more tests you have, and what it really does is sitting idle in a Thread.sleep most of the time! Quite frustrating, and it is against all good test practises (which say tests should be as quick as possible). What I expected to have was ability to be able to immediately know when all processing inside Camel ended, but as far as I can tell, there is no way to do it (we tried using InflightRepositorym, but it didn't work reliably  with async components). And, BTW, this article presents exactly the same problematic test pattern - it waits 5000ms at the end of the test - so it means, the actual processing could take less then 1ms, but then you're wasting 5 seconds for idle waiting! Imagine a suite of 100 tests, which could execute in say 10 seconds, would instead take 10 minutes using this approach. I'd be glad to find a solution for this.

Another issue with camel testing was related to mock endpoints usage: whenever such test failed, the error message it produced was frustratingly unhelpful: all you got was "mock endpoint expected 1 message, got 0", go figure out why.

Claus Ibsen replied on Thu, 2013/10/10 - 2:07am in response to: Grzegorz Borkowski

Disclaimer

I am an Apache Camel committer for many years. 

Ad 1)

Camel testing has a lot of power. A good staring point is the documentation at: http://camel.apache.org/testing

For testing with delays, then take a look at notify builder, which allows you like a "black box" testing scenario, to setup expectations that X number of messages must be processed. And then you can use the notifier to wait for this to happen. You can then use that as a substitute for any manual thread sleeps.

The testing capabities is really powerful and its used to test Camel itself. There is hardly any thread.sleep in our own testing - if there are they are usually there from back in the days when we did not have notify builder or the test kit wasn't so powerful as today.

Ad 2)

About the error messages from the mock endpoints. What do you suggest it should say? You only tell it that 1 message should arrive, and it fails because 0 message arrived.

Would you like to be able to specify your own error message? For example as an optional argument to the mock API. For example:

MockEndpoint mock = getMockEndpoint("mock:foo").expectedMessageCount(1, "We should have one message being sent to foo to complete the business transaction. If this does not happen then the problem is likely because of the backend did not process in time.");



Claus Ibsen replied on Thu, 2013/10/10 - 2:09am in response to: Grzegorz Borkowski

And btw the test example in the article will wait at most 5000 millis. If the mock is satisfied after 1 milli seconds, then the test complete. So the value is an upper limit to wait.



Grzegorz Borkowski replied on Thu, 2013/10/10 - 4:03pm

Thanks for your reply Claus, I will need to take a look at NotifyBuilder, it looks promising. Regarding testing with mocks, and what message would I expect - actually, this is a difficult question, and I don't think I have a simple answer. I just wanted to point out that depending heavily on tests which use mock endpoints with assertions based on received messages, can be frustrating if something goes wrong. Our scenario was that we had a dead-latter channel configured to catch all unhandled exceptions thrown during the processing. In normal unit test, the test would just blow up and print the exception. On contrary, when using camel routes with deadletter channel, the exception was handled by it, and the exchange did not reach the expected standard endpoint, which resulted in the test failing with the unhelpful message "expected one message, received 0".

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.