Don’t Let Your DataMapper Streaming be Out of Control
Originally authored by Mariano Simone
As Nicolas pointed out in “7 things you didn’t know about DataMapper“, it’s not a trivial task to map a big file to some other data structure without eating up a lot of memory.
If the input is large enough, you might run out of memory: either while mapping or while processing the data in your flow-ref
just enable "streaming"
But just doing this doesn’t let you decide how many records at a time you want to get passed on to the next processor: in the worst-case-scenario, you might end up with just one line at a type processing. If your next processor is a database, you will have as many queries as lines in your file.
There is, however, a little trick to gain fine-grained control on how
many lines are being processed: setting the batchSize property of the
<foreach batchSize="100" doc:name="For Each"> <logger message="Got a bunch! (of #[payload.size()])" level="INFO" doc:name="Logger"/> </foreach>
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)