Agile Zone is brought to you in partnership with:

Mark is a graph advocate and field engineer for Neo Technology, the company behind the Neo4j graph database. As a field engineer, Mark helps customers embrace graph data and Neo4j building sophisticated solutions to challenging data problems. When he's not with customers Mark is a developer on Neo4j and writes his experiences of being a graphista on a popular blog at http://markhneedham.com/blog. He tweets at @markhneedham. Mark is a DZone MVB and is not an employee of DZone and has posted 513 posts at DZone. You can read more from them at their website. View Full User Profile

Clojure: Getting caught out by lazy collections

08.01.2011
| 2836 views |
  • submit to reddit

Most of the work that I’ve done with Clojure has involved running a bunch of functions directly in the REPL or through Leiningen’s run target which led to me getting caught out when I created a JAR and tried to run that.

As I mentioned a few weeks ago I’ve been rewriting part of our system in Clojure to see how the design would differ and a couple of levels down the Clojure version comprises of applying a map function over a collection of documents.

The code in question originally looked like this:

(ns aim.main (:gen-class))
 
(defn import-zip-file [zipFile working-dir]
  (let [xml-files (filter xml-file? (unzip zipFile working-dir))]
    (map import-document xml-files)))
 
(defn -main [& args]
  (import-zip-file "our/file.zip", "/tmp/unzip/to/here"))

 

Which led to absolutely nothing happening when run like this!

$ lein uberjar && java -jar my-project-0.1.0-standalone.jar

I originally assumed that I had something wrong in the code but my colleague Uday reminded me that collections in Clojure are lazily evaluated and there was nothing in the code that would force the evaluation of ours.

In this situation we had to wrap the map with a doall in order to force evaluation of the collection:

(ns aim.main (:gen-class))
 
(defn import-zip-file [zip-file working-dir]
  (let [xml-files (filter xml-file? (unzip zip-file working-dir))]
    (doall (map import-document xml-files))))
 
(defn -main [& args]
  (import-zip-file "our/file.zip", "/tmp/unzip/to/here"))

 

When we run the code in the REPL or through ‘lein run’ the code is being eagerly evaluated as far as I understand it which is why we see a different behaviour than when we run it on its own.

I also got caught out on another occasion where I tried to pass around a collection of input streams which I’d retrieved from a zip file only to realise that when the code which used the input stream got evaluated the ZIP file was no longer around!

 

 

References
Published at DZone with permission of Mark Needham, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Damien Lepage replied on Tue, 2011/08/02 - 5:29pm

I believe a more idiomatic solution would be:

(doseq [xml-file xml-files] (import-document xml-file))

The difference is that doall will load your entire sequence into memory while it seems you need some sort of "for each" for side-effect.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.