Geertjan is a DZone Zone Leader and has posted 453 posts at DZone. You can read more from them at their website. View Full User Profile

In-Memory Virtual Filesystems

11.12.2008
| 16306 views |
  • submit to reddit
Alex Miller recently wrote an interesting piece about the need for Java to have an in-memory virtual filesystem (here), in the context of JSR-203. Alex notes that athough "[a] virtual filesystem will not be included as part of JSR 203", "the API is designed to be extensible and all of the hooks exist to provide such an implementation".

I really like his use case for having such a filesystem:

Having a virtual memory system to plug into would be fantastic for unit testing. You could create files to your heart’s content and the file access would be fast while also saving you from all the annoying issues with deleting temporary files, Windows file locking, etc.

Makes sense to me. In the comments to his blog entry, a few existing solutions are mentioned, in particular, Commons VFS and the JBoss Microcontainer project. Also, another use case for a filesystem of this kind can be found there:

It may also be useful when running on operating systems that don’t have /tmp or equivalent on swap.

Another solution, much less known in this context, is provided by two standalone JARs that are used within the NetBeans Platform (full disclosure: I work there). These two JARs, org-openide-filesystems.jar and org-openide-util.jar, can simply be dropped in your classpath and then... you have access to an in-memory virtual filesystem. Those two JARs are all that it takes. In other words, there are no dependencies of any kind on any part of the NetBeans Platform. These two JARs can simply be copied/pasted from the NetBeans Platform (i.e., which is a folder within NetBeans IDE) to your own classpath.

The filesystem that you then have is hierarchical. Basically, imagine an XML file in memory and you've got the idea. You can write folders and files into this filesystem and then you can ascribe attributes to those files. These can then be used as part of your unit tests, exactly as described above by Alex. Here's a small example:

package demo;

import java.io.IOException;
import junit.framework.TestCase;
import org.junit.Before;
import org.junit.Test;
import org.openide.filesystems.FileObject;
import org.openide.filesystems.FileSystem;
import org.openide.filesystems.FileUtil;
import org.openide.util.Exceptions;

public class FsJFrameTest extends TestCase {

FileSystem fs = FileUtil.createMemoryFileSystem();
FileObject root = fs.getRoot();

@Before
@Override
public void setUp() {
try {

//Create a virtual folder:
FileObject testDataFolder = root.createFolder("TestData");

//Create a virtual file:
FileObject testData1 = testDataFolder.createData("testData1");
//Create three virtual attributes for the file:
testData1.setAttribute("name", "John");
testData1.setAttribute("age", 27);
testData1.setAttribute("employed", true);

//Create a second virtual file:
FileObject testData2 = testDataFolder.createData("testData2");
//Create three virtual attributes for the file:
testData2.setAttribute("name", "Jane");
testData2.setAttribute("age", 34);
testData2.setAttribute("employed", false);

} catch (IOException ex) {
Exceptions.printStackTrace(ex);
}
}

//This test will pass because all attributes match the test data.
@Test
public void testData1() {
FileObject testData1 = root.getFileObject("TestData/testData1");
assertEquals(testData1.getAttribute("name"), "John");
assertEquals(testData1.getAttribute("age"), 27);
assertEquals(testData1.getAttribute("employed"), true);
}

//This test will fail because age = 34 in the test data.
@Test
public void testData2() {
FileObject testData2 = root.getFileObject("TestData/testData2");
assertEquals(testData2.getAttribute("name"), "Jane");
assertEquals(testData2.getAttribute("age"), 33);
assertEquals(testData2.getAttribute("employed"), false);
}

}

 

Here you can see that even though you haven't written any actual physical files anywhere on disk and even though you haven't retrieved anything from actual physical files on disk, you do have the important thing: the test data that you need for your unit tests. Plus, that data can be organized as easily as if one were working with a physical hierarchical filesystem. The two JARs provide a lot more besides, such as access to ZIP/JAR archives, via filesystems provided by the selfsame JARs. Just get them from your NetBeans IDE distro and then remove the distro and use another IDE, if that's your preference. (By the way, a screencast of the NetBeans Platform's filesystem [together with a transcript] can be found here.) Looking forward to seeing an in-memory virtual filesystem of this kind available in the JDK, though.

 

Published at DZone with permission of its author, Geertjan Wielenga.
Tags:

Comments

Fuqiang Zhao replied on Wed, 2008/11/12 - 8:00pm

interesting and helpful:)

Radek Jun replied on Thu, 2008/11/13 - 2:53am

Interesting, it would be nice to have it as part of JDK :).

Roman Pichlik replied on Thu, 2008/11/13 - 8:59am

+1 for in-memory file system. Maybe i missed something, but i don't see any addition value in using of Open IDE API. Usually you test some API depending on JDK abstraction as java.io.File, so classes from org.openide.filesystems do not help at all ;-).

Geertjan Wielenga replied on Thu, 2008/11/13 - 10:44am in response to: Roman Pichlik

[quote=rp117107]+1 for in-memory file system. Maybe i missed something, but i don't see any addition value in using of Open IDE API. Usually you test some API depending on JDK abstraction as java.io.File, so classes from org.openide.filesystems do not help at all ;-). [/quote]

Classes from  org.openide.filesystems give you an in-memory virtual filesystem.

kackbratze1 kac... replied on Sat, 2008/11/15 - 1:28pm in response to: Geertjan Wielenga

I think rp117107 is raising the same point that I would raise: that having an in-memory "file system" is all very well, but if it doesn't have the same interface (or a bridging interface available) to make it look like regular Java file access (and/or JSR203), it's not much help for code already written that writes/reads files to the file system in the usual Java way!

 

 

Aaron Digulla replied on Tue, 2008/11/25 - 8:32am

Am I the only one who is missing the "file" part here? How do I use this code to actually write data to these "files"? Do they operate nicely with java.io.File? I/O streams? Can I use them directly in existing code that wants to read, say, config files?

michael james replied on Wed, 2009/06/10 - 6:23am

Virtual File System is an interface providing a clearly defined link ... kernel's memory is the same for all File System implementations.

 

thank you,

Cheap Flights

michael james replied on Sun, 2009/06/14 - 6:59am

The results indicate that variable-size cache mechanisms work well when virtualmemory - and file-intensive programs are run in sequence; the cache is able to change in size in order to provide overall performance no worse than that provided by a small fixed-size cache.

 

cheers,

College Degrees

eugene backs replied on Mon, 2009/06/15 - 4:40am

When I started designing my new 3D Engine, I realized that I needed some kind of file system. Not only some file classes or so, but something I call a Virtual File System (VFS), my own file system that supports nice features like compression, encryption, fast access times, and so on.

 

thank you,

commercial mailboxes

eugene backs replied on Mon, 2009/06/15 - 3:34pm

A computer system having a kernel interface that provides a technique for creating memory descriptors that provides a single way of representing memory objects and provides a common interface to operations upon those objects.

 

great,

FL health insurance

cool jin replied on Wed, 2009/06/17 - 3:58am

Very helpful , glad to know that we have some very helpful peoples thanks. Regards, loan

eugene backs replied on Wed, 2009/06/17 - 5:40am

What's all that junk for? I would presume it's some sort of encrypted version of your file system, but I've no means of handling it so can't do anything with it...

 

cheers,

Cheesecake Recipes

eugene base replied on Wed, 2009/06/17 - 9:24am

You can take a disk file, format it as an ext2, ext3, or reiser filesystem, and then mount it, just like a physical drive. It's then possible to read and write files to this newly-mounted device.

 

regards,

knowledge base

rahul roy replied on Thu, 2009/06/18 - 3:37am

A memory-based file system that uses resources and structures of the SunOS virtual memory subsystem. Rather than using dedicated physical memory such as a ‘‘RAM disk’’, tmpfs uses the operating system page cache for file data. regards, golf course management

eugene base replied on Fri, 2009/06/19 - 2:39pm

inmemvfs.tcl implements an in-memory virtual file system. This is
useful for creating a directory hierarchy in which to store small files
and mount other file systems.

 

regards,

Model Railroads

eugene base replied on Thu, 2009/06/25 - 12:14pm

I have a command-line executable which I need to run from Java on Windows XP. It uses files as input and output. But I want to avoid the overhead of file IO, so I thought of an in-memory RAM file system.

 

health insurance leads 

dany rich replied on Fri, 2009/06/26 - 1:48pm

Like many technologies in the history of computing, virtual memory was not accepted without challenge. Before it could be implemented in mainstream operating systems, many models, experiments, and theories had to be developed to overcome the numerous problems.

 

cheap concert tickets

eugene base replied on Mon, 2009/06/29 - 4:02am

This module makes extensive use of the functions in File::Spec to be portable, so it might trip you up if you are developing on a linux box and trying to play with '/foo' on a win32 box :)

gratis inserate

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.