Wille Faler is an experienced software developer, architect and agile coach with experience across a number of different industries as an independent consultant. Wille specializes in backend, integration, and Java technologies, but has more recently found a passion for Scala and text mining/analysis. Wille is a DZone MVB and is not an employee of DZone and has posted 40 posts at DZone. You can read more from them at their website. View Full User Profile

Will a Two Tier Market For Developers Emerge As a Result of Scala & Clojure?

12.20.2010
| 11292 views |
  • submit to reddit

Two years after my initial encounter with Scala, I’m just now starting to delve deeper into the functional aspects of Scala and the type system of the language. As I’ve done this, I’ve had multiple “aha!” moments, where I realise that the combination of OO and FP allow for massive leaps forward in developer productivity if you apply it intelligently. I’ve also started glancing at Clojure, which certainly looks interesting, though perhaps more of a niche language to Scala’s mainstream proposition.

However, in discovering FP I also realise that some of the concepts require a fair amount of brain power and sophistication from the developer to fully utilize and understand: It’s like a Formula One car: in the hands of Michael Schumacher, incredible things can be achieved, but in the hands of Joe Bloggs, tragedy might happen.

This has made me consider a possibility: will this help create a two-tier market for developers?
We have always had the “blue collar” developers, the 9-5:ers who see development as a job, not a passion. The type of person who would never dream of visiting a blog like this or a site like DZone. These developers are fundamentally uninterested in learning anything that challenges their minds, they want to do the job and go home.
At the other side we have the guys who live, breathe and dream their profession, the craftsmen who want to create. The type who has been woefully undervalued up to this point, having to compete on price with the blue collar developer even though his work is of much higher quality and his productivity is at least 10x that of the “cheap guy”.

With the advent of languages like Scala and Clojure, this 10x productivity and quality factor is likely to increase to 20x or even 30x. These are starting to be economics that not even the bean counters at large enterprises can ignore. So will this mean bigger rewards for the smart guys who “get” the features of the newer, more modern languages?
Will Functional Programming be the big dividing point where craftsmanship becomes properly valued by the market? Is it going to be the proverbial dividing point where the wheat is divided from the chaff?

I don’t have the answer, but I know a two-tier market has been developing ever since the dotcom bust in 2000-2001, but maybe this is the real dividing point, the time when the economics of craftsmanship combined with the progress of software technology together combine to create a market where highly skilled software engineers/architects finally get the recognition they rightfully deserve.

References
Published at DZone with permission of Wille Faler, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Alec Lee replied on Mon, 2010/12/20 - 9:06am

Don't get your hopes up. Decades ago, Tom DeMarco documented that there was a 10x difference (productivity, bug rates, you name it) between the top 10% of developers and the bottom 10%. Where did that get us? To bean counters, a "Scala resource" isn't going to be significantly different from a "Java resource". In fact, bean counters and PHBs probably just lump them together as "development resources" with perhaps a minor bump for Scala because "the market rate is higher".

I'll also wish you good luck convincing PHBs to adopt a technology that their "mainstream" (read: "average") staff can't grok. They (understandably) don't want to be dependent on non-commodity "development resources". They're hard to find and harder to retain.

You're not going to see meaningful correlation between developer productivity and compensation so long as the industry views knowledge workers as "resources". When "years of experience" with a stack is the primary gauge of a developer's facility, how are things going to change? Frankly, you're making the same mistake many PHBs do: you're hoping that technology adoption will solve a cultual problem.

Armin Ehrenreich replied on Mon, 2010/12/20 - 9:26am

Maybe you should realize that Scala dropped out from the top 50 languages of the TIOBE index recently:

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Damien Lepage replied on Mon, 2010/12/20 - 1:01pm in response to: Armin Ehrenreich

So what? Assembly just entered the index and is ranked 17th ...

It's tough to guess what will be the mainstream language in a few years or decades. But beating the current mainstream is something that anybody can do with a bit of geekitude.

 

 

Mark Haniford replied on Mon, 2010/12/20 - 2:31pm

Scala hasn't really gotten any closer to "a mainstream proposition".    And I'm not buying the argument that your 10% top developers are going to be 20x or 30x more productive by using Scala or Clojure than if they were using a mainstream language.

Proper tooling accounts for a lot, even though geeks never want to admit that - instead acting like they're hardcore with Emacs or Vi.

Cloves Almeida replied on Mon, 2010/12/20 - 6:22pm

Forget the bean counters, really.

Band up with a few others top tier developers and add a few top notch senior salesmen and go get consulting pay rates to finish those projects the lower end developers can't.

But don't be so focused on Scala or Clojure - sell a mainstream language a non-genius developer will be able to mantain later. Even though it's a bit more verbose, you can implement most of Scala's features in Java using proper libraries.

 

Karl Peterbauer replied on Mon, 2010/12/20 - 7:27pm

From time to time, I ask Scala proponents what they think about the fact that Scala does not provide it's own - say - I/O or networking library, and does not even provide fundamental stuff like a string or date class. Scala in it's current form is ultimately tied to old-school Java. Most of the time I get side-stepping mumbling such as "don't wan't to reinvent the wheel", "waiting for NIO2 to make it into Java 7" or  "waiting for Joda-Time to become a JSR". Not exactly the answer I expect for a "real dividing point". Does the claimed 2x productivity boost not apply to low-level library development?

Jesper Nordenberg replied on Tue, 2010/12/21 - 3:18am in response to: Karl Peterbauer

@Karl:

Being able to use and extend existing Java libraries is a major advantage of Scala compared to non-JVM languages. We can use the libraries as is (just as easily as you can in Java code), or provide a nice Scala API wrapper to them (as for scala.swing for example). Instead the Scala community is focusing energy on providing new libraries where Java has none, or where they are severely lacking, for example the Scala collection library, Actors, STM, scala.react, Scalaz. These are areas where Scala's support for functional programming and powerful type system really shines.

IMHO, this strategy is optimal, utilize existing solutions where they are of sufficient quality and focus energy on creating new solutions for problems which haven't been solved yet in the Java ecosystem.

Nicolas Bousquet replied on Tue, 2010/12/21 - 3:56am

It is difficult to state that the productivity will increase by using scala.

 We all know that top 10% developper perform really better than the bottom 10%. We also know that they provide better return for the bulk and can do things that the average or bad developper simply can't do.

 But will theses developper gain an additionnal 2x or 3x boost by using scala ?

I'am not sure. Scala support for days to days programming is not on part with java. IDEs support Scala, but it is not of the quality of java support. Tools, metrics and all is not so good for scala and because your projects will be likely to use java anyway it is another language to learn, master and keep on top of your mind. We already have HTML, CSS, javascript, java, SQL, HQL not counting frameworks. Framework that are like to not support scala as much as plain java.

 I am not sure either that developper spend so much time writting code. You know, verbosity, expressivity is important if you spend all your day coding.

 But that not what a typical developper do. The developper think, design, configure, compile, test, deploy, and commit too. The developper even talk with other developpers, or with the client or with it's boss.The go sometime to meetings too.
 

Where i work typically, code is read much much more often than written. You can spend one day just to see where to modify the code out of the 65000 class and dozens million line of codes. Then you modify maybe 1 to 10 lines of code, and test another full day to ensure nothing is broken. The fact that thoses 10 lines could be broken to 3 or 5 seems not that important.

The question is more does scala code is easier and faster to debug ? Does a huge scala codebase faster to understand and master ? I am not so sure.

Karl Peterbauer replied on Tue, 2010/12/21 - 4:55am in response to: Jesper Nordenberg

@Jesper: Attempting to wrap Swing (a really huge API) but not wrapping or replacing I/O or the pathetic old java.util.Date/Calendar etc. indicates that current Scala is still a playground for enthusiasts, tightly coupled to it's mothership, without clear focus and without a vision for becoming a full-blown competitor for "Java the Language" (that is, providing a programming environment which can be mastered without deep Java knowledge).

Nicolas Bousquet replied on Tue, 2010/12/21 - 8:18am in response to: Karl Peterbauer

@Karl

the java API is not the language, it is an API. We could have the nearly the same API behaviour in many languages be it .NET, C++, python... Slight differences would appear due to differents feature set but that's it.

It is a little like if you choose F#, Visual Basic or C# in .NET. And i would not say Microsoft languare are just a playground for enthousiats.

 

Erik Post replied on Tue, 2010/12/21 - 8:32am in response to: Karl Peterbauer

@Karl: Exactly! It's a shame Odersky and that visionless bunch of noobs of his didn't have your phone number, it sounds like you could have set their priorities straight.

Erik Post replied on Tue, 2010/12/21 - 8:39am in response to: Nicolas Bousquet

@Nicolas: I think you raise some valid points, but it's not just about LOC reduction. Scala offers a lot of ways to organize your code differently, so that its intent is reflected more clearly. To name a few: mixin composition, the very powerful type system, implicits, closures, named and default function arguments, etc. For what it's worth, I find my own code much easier (and more fun) to maintain thanks to Scala. It's a bit of a learning experience of course, especially when you're starting out, but for me it's worth it.

Eric Giese replied on Tue, 2010/12/21 - 4:56pm

I am a big fan of scala - it's still my favorite language. And I don't care about productivity rates.. What's true is that both scala and clojure make you better programmers - as learning good languages always does. But thats the long term. In the long term, learning too much the java way is worse, because you learn less.

However, short term, not having to fiddle with an unstable IDE, a slow memory hungry compiler, bleeding edge new libraries and a dozen new programming concepts may not be the worst idea as well. Especially if you want to earn cash for a living.

Anyway, I guess the combination of functional and imperative programming in a statical typed(inferred) language should be the leading ideal for the future. For enterprise applications.

However, I have got one concern with scala: It still cares too much about syntax. Ok, its syntax is much simpler than both java's and c#, when taking the corner cases of these languages into account. But it could be much simpler by sacrificing a small bit of conciseness for consistency. There are too many syntactical variants to express something which do not really result in better or more reusable code.

What we need is a langauge with simple syntax and a great and expressive typesystem. Smalltalk could be an inspiration for the former (message-passing), Scala for the later.

Mark Haniford replied on Tue, 2010/12/21 - 9:53pm in response to: Eric Giese

What's true is that both scala and clojure make you better programmers - as learning good languages always does. But thats the long term. In the long term, learning too much the java way is worse, because you learn less.

Yes, the classic myth that if you hack on Haskell and Common Lisp you'll be a great programmer, but with a twist that "learning too much the java way is worse, because you learn less"...whatever the hell that last part really means.

 Solving a wide variety of problems and writing lots of code is how you become a better programmer.  People are always looking for this productivity x30 silver bullet.   Scala and Clojure aren't it.

 

Eric Giese replied on Wed, 2010/12/22 - 3:08am in response to: Mark Haniford

"learning too much the java way is worse, because you learn less"

Each language has a kind of limit up to which it can bring you up. Learning better programming is usually learning abstraction. Java is better than Cobol or Basic in that regard, but worse than a lot of other languages. It takes you to the OO-level, but not any further. It does even have some very simple elements of abstract programming: closures and higher order functions, the Uniform Access Principle, multiple (mixin) inheritance, "Everything is an Object" paradigm, ... the list goes further.

I have coded in java for more than seven years now in countless problems. And a lot of the time I spent thinking my head around problems which could easily be solved in a simpler way if the language was more expressive.

Nicolas Bousquet replied on Wed, 2010/12/22 - 4:12am in response to: Mark Haniford

@Eric Giese

I would not say that you learn less with JAVA than scala or closure.

For learning, you learn the language yes, but you also learn the API, the patterns, the frameworks.

Java is bundled with complex and advenced frameworks for everything. In a way, thoses framework and java are more verbose than many other languages or typical frameworks.

We can say it is an accidental complexity, but it is also by design. The Java ecosystem embrasses this way of programming where basically you control anything, code is very generic and where you need to master everything to become an effective developper.

So maybe we can say it is OO, but not only, because you can have OO and stay simple.

Scala/Closure are just syntaxic sugar and embrasses a different way of programming by extending the base API. Nobody prevent you to use the immutable type in JAVA or to make new one. Nobody prevent you to use a transactionnal memory model in java

But I admit it is true that you can't and you shouldn't try by learning scala/closure. They are so dependent of the underlying API and java ecosystem that you need to know it anyway. So we can say the give you something more rather than something different just because they are on top of the stack.

 Anyway i don't know for you but learning some sort of lisp (including lambda calculus) and functionnal programming languages what part of the standard courses at university.

Nicolas Bousquet replied on Wed, 2010/12/22 - 4:31am in response to: Mark Haniford

"Solving a wide variety of problems and writing lots of code is how you become a better programmer.  People are always looking for this productivity x30 silver bullet.   Scala and Clojure aren't it."

 These is no silver bullet for that king of boost anyway. It is easy to have a 10x or 30x productivity boost just by choosing the right design for the job. But then it depends of the job.

 I think for exemple that if you need a plain simple application with basic crud operation, in a sence, just use Microsoft access and you ll be done in no time.

Then if you want a great design with an ORM, OOP, IOC, AJAX and anything, you'll need maybe more than 30X the time to be done.

And you know great programmer DO offer a 10x productivity boost. That come from their better understanding of the platform and code of course. But it is just one part. The other part is their abilities to think out of the box and choose the best solutions (or at least good one) for a good range of problems. That include not using scala sometime.

Mark Haniford replied on Wed, 2010/12/22 - 9:43am in response to: Eric Giese

Eric, I understand your argument, but for the most part I don't agree with it.    Your argument seems to be that it's a Java or Scala/Clojure proposition.  That it's a zero-sum game.  I don't really know how you learn "too much" Java.  For the most part you can pick up the syntax and semantics of Java in fairly short order.  Learning how to apply frameworks and libraries to your domain is what tends to be lengthy. 

 If you look at great programmers like John Carmack (Doom/Quake) and Tim Sweeny (Unreal), they didn't get to be great programmers by learning a bunch of languages.  They got to be great programmers by applying languages like asm/C/C++ to their chosen domain - graphics and games.

Learning Clojure and/or Scala or whatever "better" language isn't a waste, and CAN lead to better designs and more productivity with the right team.  But in the real world, as I'm sure you know, we have to deal with the hand we're dealt. And typically that has to do with high turnover, non-technical management, management that isn't prepared to invest the resources we need, co-workers that aren't at the same level as others.  

To me it's really about learning how to be a better programmer is really about learning how to work within the constraints that are you're facing.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.