I'm an author and a developer focused on build tools. I'm currently focusing on Gradle, but I have an interest in all build tools and most development infrastructure. I focus on Enterprise Java, Ruby, and the interface between Systems Administration and Software Development. The focus of my work is to make it easier for individuals to adopt open source software. Tim is a DZone MVB and is not an employee of DZone and has posted 41 posts at DZone. You can read more from them at their website. View Full User Profile

Distributing Binaries: Why not just use a Shared Filesystem?

02.16.2012
| 2554 views |
  • submit to reddit

I’ve seen it. I’ve seen the builds you never want to have to reckon with. The kinds of build scripts that make you stop, take a deep breath, turn to your client and say: “This is going to take a while to untangle.” Ruinous hulks of procedural build logic relying on a lot of token replacement (both pre and post compilation). These builds, especially the more difficult builds, are in love with the idea of everyone having access to the same shared filesystem, or worse, they use source code management as a way to distribute binaries.

While I’ve seen this pattern in Unix-based environments, the “Everything is on the Shared Filesystem” build happens most often in environments that have standardized on Windows, and I’m pretty sure it happens because it’s so easy to see the shared filesystem as a cure-all for collaboration. If you share Word documents and internal files through the shared filesystem, why not build output. If everyone’s already connected to it, why not just dump all your libraries on it? Isn’t Central just a glorified filesystem any way?

What a “Shared Filesystem” approach turns into…

It must be easy to fall into this trap, because I’ve untangled so many builds that assume the presence of an E: drive or G: drive instead of integrating something like Nexus. This is so common, I want to take some time to tell people what I see at the tail-end of these projects. What happens several years after someone makes the decision to depend on a shared network filesystem instead of using Nexus.

These builds are characterized by the following anti-patterns:

  1. Need a Dependency? Just put it on the G: Drive. At the beginning of a project, it’s easy. You have five external dependencies, just drop all the JARs in G:\Project\Build\lib and point your Ant build script to this directory. Now fast forward seven years and you have 40 dependencies, each with transitive dependencies. Every time you try to upgrade to a new version of a library you end up having to wade through incompatibilities and inconsistencies. Long story short, no one touches this directory after a while, it’s just too difficult to change because the build is so massive, no one understand the ramifications of removing a file from this directory. You end up with 10 versions of commons-lang, seven version of Spring.
  2. We deploy on Linux, but it only builds on Windows. When your build depends on the G: drive, it’s usually tough to translate all of these paths to a non-Windows environment. People have tried, but they had to hack at the build script. In general, if operations calls you up and asks for a build, tell them to just grab the latest copy from the G: drive. It’s impossible to transition these builds to anyone other than a developer.
  3. Don’t delete that! (and don’t change it either) Remember six months ago when someone from the infrastructure group decided to delete our files to free up space? That put us back a couple of weeks. Sure, it would make sense to lock down permissions on this thing, but the organization uses it to collaborate on Word documents and budget spreadsheets. Shared filesystems rarely have strong security, if they were really secure they’d be less useful for collaboration.
  4. Can’t run the build, the network’s down. When everything in your build relies on a shared drive, you are dead in the water when that resource becomes unavailable. Now, if you had Nexus installed and you were using an intelligent build tool like Maven, Gradle, or Ivy, your builds would still work because these tools are designed for disconnected operation.
  5. Oh those files? I have no idea, that guy quit last week. The “Fog of War” surrounding awful builds is what amazes me. By the time a client engages me to fix a build, it’s usually because the build has been fumbled by a build engineer who decided to move on. Quite often this shared filesystem contains a random collection of files that once supported one version of the build. When you rely on a shared filesystem to share binaries, it often ends up looking like someone’s messy junk drawer.
  6. Right, don’t run the production build at the same time as someone else…. When your builds store bits on a shared network filesystem, it is often the case that two people can clobber each other’s build. Concurrency is an issue when you use shared network filesystems.

Nexus to the Rescue

Instead of using a shared filesystem to share binaries and dependencies. Use Nexus. Nexus provides a series of advantages over shared filesystems, here’s an enumeration of the benefits:

  1. Structure: Nexus repositories follow a standard structure. Whether you are using a Maven 2 repository, an OSGi repository, or the upcoming support for NuGet, you’ll be sharing binaries within a set structure. If you need to share a library, your tools will be able to address it using a coordinate system and you won’t open yourself up to the “messy junk drawer” syndrome I’ve outlined above.
  2. Cross Platform Compatibility: Any tool that publishes or reads from Nexus is simply interacting with a set of REST endpoints. You don’t have to standardize your build on a single operating system, and it will be easier to deconstruct massive, problematic builds. Your developers can work on Windows, OSX, or Linux, and your build scripts can run on a Hudson instance running Linux.
  3. Concurrency/Scalability: When you depend on a shared network filesystem just the right simultaneous build being run by just the right developers can wreak havoc. With Nexus, you have a web-based system that understands how to serve thousands of concurrent requests. If your team is scaling up, you should scale up to Nexus.
  4. Stability: Don’t create build systems that can be held hostage by network interruptions. Tools like Maven, Gradle, and Ivy will keep a local cache of dependencies they download from Nexus. This will lead to more stable builds when compared to scripts that depend on the presence of a shared network drive.

If you recognize some of the anti-patterns in this post, it’s probably time to check out Nexus OSS. To learn more about Nexus, go to http://nexus.sonatype.org

 

From http://www.sonatype.com/people/2012/02/distributing-binaries-why-not-just-use-a-shared-filesystem/

Published at DZone with permission of Tim O'brien, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Goel Yatendra replied on Thu, 2012/03/15 - 4:00pm

nterestingly, I'm getting rid of fileshares from our build right now.  The only issue I'm having is how to provide easy access to artifacts for the less technical folks who might touch them.  These artifacts are binary installers, not jars.  Previously, our build put the latest version in a folder called latest on the file share . . . how can I provide the simplest, and I do mean simplest, access to "latest" for the non-technical folks.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.