Performance Zone is brought to you in partnership with:

I am a software architect passionate about software integration, high scalability and concurrency challenges. Vlad is a DZone MVB and is not an employee of DZone and has posted 43 posts at DZone. You can read more from them at their website. View Full User Profile

21st Century Logging

11.04.2013
| 7864 views |
  • submit to reddit

I think logging should get more attention than we are currently giving it. When designing an application a great deal of effort goes into modelling the customer business logic, making sure all use cases are covered and handled properly. The business model is mapped to a persistence storage (be at a RDBMS or a NoSQL solution), frameworks are being chosen: web, middle-ware, batch jobs, and probably SLF4J with log4j or logback.

This has been the case of almost all applications I've been involved with, and logging was always a second class citizen, relying on good old string logging frameworks for that.

But recently I come to realize there is much more to logging than the current string based logging systems. Especially if  my system gets deployed in the cloud and takes advantage of auto-scaling, then gathering text files and aggregating them to a common place smells like hacking.

In my latest application we implemented a notification mechanism that holds more complex information since the String based log wasn't sufficient. I have to thank one of my colleagues I work with who opened my eyes when saying "Notifications are at the heart of our application". I haven't ever thought of logging as the heart of any application. Business Logic is the heart of the application, not logging. But there is a lot of truth in his words, since you can't deploy something without a good mechanism of knowing if your system is actually doing what it was meant for.

So my notifications are complex objects (debug ones having less data than error ones) and a NoSQL document database is a perfect store for our logs. A notification contains all sorts of data:
- the current executing job,
- the source of data,
- the component where the log originated,
- exceptions being thrown,
- input arguments,
- the message history of the Spring Integration Message carrying our request.

Therefore, since I am able to store complex objects in a schema-less fashion, I am also able to query logs, and it doesn't matter the order they arrive since I can order them by source and creation time. I can have a scheduled job generating alerts and reports when too many error entries are detected.

This is a custom-built logging implementation as we haven't been using a dedicated framework for our notifications, but I get more value out of it than from the classic String based log files.

I still think log4j and logback are very good implementations and we haven't replaced them, we've only added an extra logging feature to overcome their limitations, but even with the new logback appenders, I still think the current String based logs are way too simple for production systems requirements. And if you use them more for debugging purposes, while having additional monitoring solutions for production environments, then maybe it's time to use a smart logging solution that works for both development and production environments too.

If that was difficult to implement 10 years ago, when RDBMS ruled the storage world, and file based logging was a good trade-off, I think we have means for implementing better logging frameworks now. The current "string-based file logging" model might have been sufficient especially when our server was scaling vertically on a single machine, but in a world of many horizontally distributed servers, this requires extra processing.

Big players are already employing such new generation logging systems Facebook Scribe, and LinkedIn Kafka log processing.

I really liked the LinkedIn solution, and it inspires me to reason about a new logging system working in a CQRS fashion, where log entries are events stored into a log database, and each event passes through a chain of handlers updating the current system state. This combines both logging and monitoring, and the monitoring commands go directly to a cached latest system state representation, which holds:

 - alerts,
 - status reports
 - monitoring views of the current system status

How does it sound to you, is it worth implementing such solution, should we start a new open source new generation logging project?

Published at DZone with permission of Vlad Mihalcea, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Steve Schols replied on Tue, 2013/11/05 - 2:11am

I never thought of logging with complex objects. This subject intriges me, and I would like to read more about it.

I don't have any NoSQL experience yet, but indeed the document-based storage, and the fact that you can just query your "log files" so to speak, would make root cause analysis a lot easier or at least more clear.

Vlad Mihalcea replied on Tue, 2013/11/05 - 2:45am in response to: Steve Schols

Hi Steven,

My vision is that file-based string logging is like using text-files instead of a database. Usually logging is not taking to seriously until you move into production, when you realize logging/monitoring are equally important as any other aspects of your application.

Having so many NoSql solutions nowadays simplifies implementing a system of smart logging, and if more people get interested in such idea, I plan on starting a new open-source project to address this need.

The project goals should be quite straight forward:

- simple API to submit log objects

- asynchronous batch job to save the log objects into a NoSql storage

- support for handling a log object and update the "current system state"

- support for exposing the "current system state" as JMX

It see it as a library on top of which you start implementing your own smart-logging solution based on your current project requirements, rather than a full-featured logging application which cannot foresee the complex requirements of any project you'd want to integrate with.

Vlad

Michael Shields replied on Thu, 2013/11/07 - 4:16pm

Useful log output can save you lotsa effort in Dev, QA & Prod; or you can just throw it over the wall to Ops...

I cooked up a simple notification solution that allows devs, who want to log something, to add objects to a Map wrapper (that knows when it was created) and log the wrapper as JSON (with some business context); we use Splunk queries to investigate errors, build dashboards, etc. 

Recently, we re-used the values collected in the Map to identify generic Exceptions, for improved user-facing messaging. 

I'm now thinking of providing (in non-Prod environments) hyperlinks to Splunk queries embedded in user-facing error messages - so QA can investigate more deeply, saving Dev time.

Vlad Mihalcea replied on Fri, 2013/11/08 - 2:05am

Hi,

Splunk seems like a very handy tool, I'll have to investigate it.

Vlad

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.