There are certainly some cool big-data tools in Scala (personally I find Spark much more compelling than Scalding, but the fact that I'm even comparing two reasonable choices is more than you'd get with many languages), but I don't think that's what's driving adoption. My case for Scala would be that it's a great language to write and an even better one to read: even more expressive than Python but much safer to refactor, a type system powerful enough to encode most of the business-level constraints and easy enough that you'll actually use it, and it's easy to understand which parts of the code do what because the type system makes it possible to isolate and track effects in a reliable way. In all the companies I've seen adopting Scala it's been a bottom-up thing, less because it offers some particular killer feature than because developers want it and it makes them more productive.
Hiring experience with a specific language is of course easier with more popular languages. But unless you're hiring emergency consultants or something, it's more important to hire smart people who'll be able to learn whatever language comes up. And I suspect that while it'll be very easy to find someone who "knows Python", the talent level is... variable. Whereas people who know Scala, though harder to find, will be the kind of people who learn multiple languages, who enjoy learning new skills even when there's no obvious market for them - i.e. the kind of people who you want to hire.
Thanks for the response. I agree with your thoughts on hiring. I'm one of those guys who came in and learned Scala. The initial reason for my team using Scala was, "Why not?" Since that time, my colleague who hired me has left the company and I have been the lone engineer on the team.
We're currently looking to hire a second engineer and have also been rethinking the direction of the data engineering team. Changing the language to Python was a suggestion for the hiring factor and because it is the popular language for data engineering. Scala has been a fun language over the last year and a half and I wouldn't mind sticking with it because it has been fun. For that reason, I have spent time comparing the two languages for data engineering and haven't come to a consensus on which one is ideal for my team.
It depends on the level of the developer you can find and are willing to pay for. Not every dev shop is able to hire top talent, simply because that talent is limited (by definition) and not always necessary.
Most shops that are adopting Scala are doing so because Java has been so stagnant and they need to stay on the JVM. The argument of Python vs. Scala is less clear.
To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.
I've also had to write code like this:
client.post(args).mapTo[Response].map(r => (r.success, r.serverException, r.unhandledException) match {
case (Some(response), None, None) => response
case (None, Some(serverEx), None) => throw serverEx
case (None, None, Some(unhandledEx)) => throw unhandledEx
})
simply because of the underlying api that other co-workers have built. When it takes you longer than 30 seconds to explain how a piece of code works to others, something is wrong.
All that being said, I do like the language; just not the compile times. It's also pretty close to ruby (with which I'm most familiar):
> To a Scala veteran, I'm sure that's easy to understand; to someone who's been learning the language (like me) it looks like gibberish.
Compare what the same code would look like in Ruby; something like (hope I get the syntax right):
def toVector = fold(new Vector()) {|x, y| x :+ y} {|x, y| x ++ y}
Are the extra |x, y|s actually clarifying anything? Or are they just syntactic ceremony? Maybe it's just my scala experience talking, but I think the scala example is clearer; there's very little performance to get in the way, just the meat of what the function's actually doing
(You can think that's a good or bad function to have, but that's a library question, not a language question).
> simply because of the underlying api that other co-workers have built.
You can write bad APIs in any language; with a more sensible one that would look like:
client.post(args).map{
case Success(response) => response
case ServerError(ex) => throw ex
case UnhandledError(ex) => throw ex
}
Explicitly handling the different cases is exactly the kind of debugging advantage I was talking about; it takes up-front effort to distinguish between ServerError and UnhandledError, but the result is code where you can see all the possibilities and know exactly where any given failure might be happening. (And again, the language doesn't force you to do this; you can just write the happy path and allow any kind of exception to happen at any point. But you'll pay the price in debugging, as I have in Python, and as I presume you have in Ruby).
Hiring experience with a specific language is of course easier with more popular languages. But unless you're hiring emergency consultants or something, it's more important to hire smart people who'll be able to learn whatever language comes up. And I suspect that while it'll be very easy to find someone who "knows Python", the talent level is... variable. Whereas people who know Scala, though harder to find, will be the kind of people who learn multiple languages, who enjoy learning new skills even when there's no obvious market for them - i.e. the kind of people who you want to hire.