All the latest news and views from the world of Java

The Podcast

Podcast 14: Java 11 is here

02-Oct-2018 in Podcasts Core Java

It's been a while but the latest podcast is now here... just after the official releaes of Java 11. Find out whether Java 11 is worth paying for, and whether you should be rushing to upgrade. We'll also mention what we're working on (SparkSQL and Spark Machine Learning)...

Transcript

Matt:We're recording.

Richard:We are recording. And, I assume you're going to do the fancy ...

Matt:Hello! And, welcome. This is the fancy introduction that you wanted, Richard. Hello, and welcome to podcast number 13. And, can I start ... No, number 14.

Richard:14.

Matt:Oh my goodness. This is going to be a disaster. I also want to apologise to the poor people who've got to subtitle this podcast, 'cause it's always difficult when people are laughing and talking over laughing. So, to our great subtitlers out there, apologies for that, or transcribers rather, anyway. Oh, this is a disaster.

Richard:We pay a fortune for the subtitles, so ... We do. It costs a fortune. So, I'm not sorry. Let's make it contrafibularities to our subtitles.

Matt:Oh, dear.

Richard:I'm anaspeptic

Matt:What?

Richard:... to cause such discombobulation.

Matt:Can I ... It's been along gap since the last podcast, and I suspect it's going to be a long gap to the next one. Because, I think that would've just bankrupt us, the cost of transcribing difficult words like that.

Richard:Oh, dear. That's our best introduction yet, well done.

Matt:So, this is ... Let's try that again. This is podcast number 14 of All Things Java. My name's Matt Greencroft.

Richard:I'm Richard Chesterwood.

Matt:And, I suppose we should apologise for quite a long gap.

Richard:It has. Yeah.

Matt:May, that we did the last edition.

Richard:May.

Matt:And, I guess we've just been completely slowed under with projects...

Richard:Yeah. We had a guy, didn't we, send us a Facebook comment this week saying, "Your blogs are silent. You've no podcasts." He mentioned something else.

Matt:I think very little on Facebook was the third thing he said. Yeah.

Richard:Yeah. And, to be fair, I think we had scheduled in this podcast already. We're not doing the podcast just for that one commenter.

Matt:No. Absolutely.

Richard:But, he was absolutely right. And, I certainly, me in particular, I go completely dark when I'm writing. I just can't think of other things. You're much better at multitasking, but-

Matt:Well, I don't know if I'm great at multitasking, but I think we've both been occupied on writing, which we'll talk about in the podcast. And-

Richard:Apache Spark we're continuing the developments of.

Matt:Yes.

Richard:So, we've got lots of machine learning coming up on this podcast, I hope. Yep.

Matt:Indeed. Indeed. Personally, I've been working on a project doing, actually, some live lecturing with a university, that's one of our customers. But also, we were just talking about this before we pressed record, is that we're based in the UK. Brexit is happening, and we're trying to work out how it impacts us and our customers. And, it's taking up an awful lot of time and a lot of mental energy. And, I guess that's taken ... It made us take our minds off the ball a bit from this, getting out all that regular communication that we really should be doing. So, apologies.

Richard:Yeah.

Matt:And, we will do better. Or, we'll try and do better.

Richard:No we won't.

Matt:No, we won't - you're absolutely right.

Richard:We will carry on exactly the same way. But, it is good to be podcasting, I think.

Matt:It is. It is. And, we'd scheduled in today, today is Friday the 28th of September, knowing that this week was the week that a momentous occasion was scheduled to happen, which is the release of Java 11.

Richard:Yes.

Matt:And so, we assumed when we scheduled in this podcast, that we would want to talk about that. Richard, do you want to talk about that?

Richard:No. Well, I've said it before, I find these version things really boring. From a technical point of view, it's not a big deal. But, the Java 11 is a big massive deal for two reasons. I suppose they're connected reasons, aren't they? So, this is the first of these LTS releases.

Matt:Yes.

Richard:So, I have a cold at the minute. So, you have to do the bulk of the talking on this podcast. So, I'm just going to cough my way through this. So, tell us all about what LTS ... what-

Matt:So, LTS stands for long term support. It's interesting, actually, 'cause I think ... Because I've been working on doing a couple of lectures, I've been reviewing some of this stuff recently. So, it's taken us something like 23, 24 years to get eight versions of Java, followed by 18 months to get the next three. And, there'll be another 18 months 'till the next three.

Matt:And so, nobody that I know has moved from Java 8 to 9 or 10. Because, everyone knows that Java 11 is the first version that-

Richard:Oh. That's an interesting ... I've got to pick you up on that.

Matt:Okay.

Richard:You think that's the reason why people have not moved to Java 9, for example?

Matt:No.

Richard:Okay.

Matt:It's one of the reasons. But, certainly stability and confidence is others.

Richard:Yeah.

Matt:And, the fact, obviously, there are some big changes between 8 and 9, which break things for the first time.

Richard:So, for example, as far as I know, I haven't looked recently, but Apache Spark is not compatible with Java 9. It certainly wasn't when I last looked.

Matt:I'm pretty sure it isn't. Android certainly isn't.

Richard:Yeah.

Matt:And, I don't know ... Most things that you look at say, in my experience, "You need to use Java 8 for a guarantee that it's going to work." Because, a lot of the framework manufacturers do need to make significant changes to make things work with Java 9.

Matt:The other thing I think that might alter the workflow, if you are a IntelliJ user, then you know that in the Java 8 world, it is one project per IDE instance. Whereas, if you're an Eclipse user, you can have two or three projects open and be working on them all within on IDE. In the Java 9+ world, that becomes, as I understand it, one IDE instance per module.

Richard:Right.

Matt:Now, because of this modularization, I believe, as a programmer, and I'm riffing a bit here, that you could be working on two or three modules simultaneously that are going to be interacting with each other, and you're going to need to do that with multiple IDEs open. And, that is a different workflow and experience. And, that will take some getting used to.

Richard:Okay.

Matt:So, that's certainly a couple of comments from people I've heard. So, there's definitely some big changes.

Matt:But, going back to this LTS idea-

Richard:Yes.

Matt:... if you're a company, and you are using Java in earnest, and one of the things that's important to you is knowing that there is support out there, and by support I mean somebody you can sue when things go wrong, the LTS bit becomes important, right?

Richard:Sure.

Matt:So, Oracle are saying, or ... I'm not sure if it's Oracle or the Java committee, but certainly, they're saying they're going to commit to updates for Java 11, in terms of security updates and patches, for a significant number of years. And, I can't remember off of the top of my head how many it is.

Richard:I think the next LTS will be Java 17. So, that's six versions. So, it must be three years.

Matt:Well, I think they're saying they're going to give patches longer than that.

Richard:Longer than that. Right.

Matt:Because, in the same way, they're still giving patches to Java 8, right?

Richard:We'll come on to that, won't we?

Matt:We will.

Richard:That's a big, big deal.

Matt:So, I guess people are now, now that Java 11 is out, they're starting to say, "Well, actually, what's in Java 11? When should I upgrade? When should I start looking at it?" And, I don't know about you, my instinct at the moment is, I certainly want to go back through our Java fundamentals course and just test everything with Java 11 and see. Because, for the fundamentals of learning a language, I imagine there should be a couple of extra bits that you might now want to know about. But, the fundamentals, how to do a loop, how to do conditions, interfaces, all that kind of stuff, is probably still there and should, hopefully, not be broken. But, that might be famous last words.

Matt:So, certainly that's on my agenda, is, as a good way for me to explore Java 11, is let's go back through the fundamentals and work out what we've done, if anything, that's broken. And, if there's nothing, great. Then we just start talking about what's new and what's extra. And, the important bit is whether it's a good idea to use it just 'cause it's there or not.

Richard:Okay.

Matt:The other aspect of Java 11, which I'm guessing you were alluding to, is around cost and paying.

Richard:Yeah. There's a lot of fear, uncertainty, and doubt going around the community at the moment.

Matt:So, shall we try and understand? My understanding, which may not be correct, I've literally spent half an hour reading some of this stuff, is that if you wish to use the Oracle JRE, sorry Oracle Java virtual machine, to run a Java application in a production environment, and I'm using that term quite loosely, or quite vaguely I should say, then you are required under their licence agreement to pay them a fee.

Richard:Yes.

Matt:So, as I understand it, for the purposes of development, and for the purpose of demonstrating an app that you've built, there is no fee required.

Richard:Okay.

Matt:It's for running it in production. And, the fee ... There is one fee for desktops, one fee for servers. And, it's something like, and I'm making this up, this might not be exactly right, about $2.50 a month, US dollars, for desktops, and something like $25 per CPU in a server per month.

Richard:I think that's bang on. Yes.

Matt:I think there's a bit of variation based on sizes and things, but you can use the OpenJDK for free. So, you're not, of course, obliged to use Oracle's virtual machine. But, if you are, then you are meant to pay for it. That's my understanding.

Richard:Okay.

Matt:So, I guess the question is, who's going to pay and when should you ... Or, perhaps the question actually is, is there any reason not to just use the Oracle ... sorry, the OpenJDK virtual machine and not pay Oracle for a licence? I can see if you're a big company, and you actually want support from Oracle, that you might well be willing. And, it's not big money for a big company, clearly. But, if you're a start-up running a single server...

Richard:Oh. There's no problem in that case - the OpenJDK has been around for a long time.

Matt:Is well-

Richard:It's pretty much the same code base as the Oracle.

Matt:And, is it not now the reference implementation for virtual machines?

Richard:Oh, I can't remember ... It may be. I've lost track of that. And, it is exactly the same code base. The Oracle came with some extra proprietary tools and so on-

Matt:Yes.

Richard:... but, for many years, now, there hasn't been a risk in using OpenJDK in production.

Matt:No.

Richard:So, the start-up that you're alluding to, that will be ... I mean, we're one of them.

Matt:Well, I was going to say, we're using-

Richard:Absolutely.

Matt:Are we not using OpenJDK? We are. Yeah.

Richard:We absolutely run OpenJDKs on our servers. So, no problem there. So, yeah. As you were mentioning before, though, those companies who need support, etc., I guess that would be a problem for them. But, it's caused a massive stink in the community. Whatever the end result of all this is, it's causing a lot of fear and doubt.

Matt:Yes.

Richard:And, as you can tell from us riffing here, we don't know yet. This is going to take some time to settle, to get some clarity.

Matt:Yes. And, it's interesting that from the average programmer's point of view, if you are a developer, that's not your job to be purchasing licences for production environments, you know?

Richard:Uh-huh (affirmative.)

Matt:And, certainly in big organisations, I imagine most of them will have a purchasing department who will be responsible for this sort of thing. But, I guess as a developer, what you do need to know is, is there any reason, technically or performance type thing, why one JVM would be better than another, at least in terms of, if you've always used the Oracle JVM or production servers, is there a good reason to stay with that rather than switch it to the OpenJDK? And, I'm not aware of any at this stage.

Richard:I'm not aware of any. I wouldn't ... I'm not insured, so I can't say, "There's no problem. Everybody just move accross." I can't do that.

Matt:No.

Richard:But, it's the same running base.

Matt:Yes.

Richard:There is big variations in the performance of other virtual machines... Remember, there were other implementations of virtual machines.

Matt:Yes.

Richard:We don't use them. We don't have any experience with them. But, there are others.

Matt:No.

Richard:I thought, I might have ... Yeah, that there's no longer any free updates to Java 8 either. So-

Matt:When you say free updates, though, what does that mean? Would security patches not still be being released for free?

Richard:As far as I know, not, unless you're paying the licence fee.

Matt:Right.

Richard:What I'm not sure about is if Java 8, after January when this will kick in, January 2019, I took it as ... You'd been talking specifically about running virtual machines, run time environments. My reading of this was this all applies to the full JDK. I'm not clear if, after January, you will be able to obtain a Java 8 Oracle JDK.

Matt:Right.

Richard:So, this would affect our courses, for example, where if you have to use Java 8 to do Java fundamentals-

Matt:Yes.

Richard:I'm fairly certain the current video directs you to the Oracle site and you download from there.

Matt:Sure does. Yeah.

Richard:It would now be a case of having to go to the adoptopenjdk.net site to download from there.

Matt:Yes.

Richard:Which is going to be the ... I suspect it's going to be the community repository for pre-built OpenJDK binaries. And, they will have an archive of old JDKs as well.

Matt:Yes.

Richard:And, presumably, the will keep patching them and so on.

Matt:We need to establish that. I mean, certainly we're in this interim period where Java 11 won't have been widely adopted, clearly because it's only just been released. Even if crowds rush to use it, it would still take a year or two before it's in wide use.

Matt:So, I think we need to do ... I'm picturing that in terms of our catalogue, we might be recording one or two of the introductory videos to explain, "Most of this course is on Java 8. If you're using Java 8, that's great. There's an alternative for video three where we install it, because if you're using Java 11 ..." Or, "If you're using Java 8, you need to install it." I can see a couple of where we'll just have alternatives. And, people can choose the right video for what they're using. And then, in due course, in a year or two's time, as Java 11 becomes the defacto, we will make greater changes then with, as I say, a few extras.

Matt:I've had an inquiry about JUnit 5, so our test driven development course, which is written in JUnit 4, as I understand it, I haven't fully checked this yet, but it's absolutely fine with JUnit 5. But, there's a couple of alternative features in JUnit 5 that we probably should mention. So, there's going to be an extra chapter added on.

Richard:Yeah.

Matt:And, that's the only way, realistically, we can do this. But, of course, if you're learning JUnit, or if you're learning testing, you need to know everything in that course. It's not ... The fact that we've got a new version out is not really the most critical thing. So-

Richard:The worry I have, though, is that for a period, it might be the case that you won't be able to obtain any usable version of Java that works in our courses. So, we've got to ... This January review it.

Matt:And, that will certainly be a challenge if that happens.

Richard:Well, we would use adoptopenjdk.net as a ... We'll just put in a-

Matt:Yes.

Richard:"Go there. Just get the last Java 1.8 JDK, and you're fine."

Matt:Yes. So, we'll keep an eye on that in the next... work out what we do in the next six to eight weeks I guess.

Richard:Yeah. I'm slightly embarrassed that I don't have a ... As we said, we've been writing, we've been working on real projects, basically. And, we are stuck in this fear, uncertainty, and doubt as well, just as everybody else is, really.

Matt:Absolutely.

Richard:I do wonder if this is, we're beginning to see the beginning of the end of Java, possibly. I mean, it has to happen at some point. It's not edifying, this whole process.

Matt:No. We'll see. I'd be surprised if this is the beginning of the end, and it doesn't have to happen because C, which is way older than Java, is still out there.

Richard:It's different, though. C's simple. It's very ... This is ... I'm not going to mention specifics, 'cause it always starts wars, and comments, and all that. But, I can think of many languages who have done similar things to what we're seeing in Java, and it has resulted in loss of community, loss of support, and then that language becomes a technical dead end.

Matt:Right.

Richard:They might still exist as languages today, and you might still be able to get jobs as programmer in those languages. But, there's no interest in that community anymore.

Matt:When you say...-

Richard:I'm desperate to name what I'm thinking. But no I'm not going to do that.

Matt:Don't do it. But, if you're talking about the death of Java, are you talking about death of Java the programming language but not Java the ecosystem? So, bearing in mind there are other programming languages that can run on the JVM, and some of those are absolutely in the ascendent at the moment-

Richard:Yeah. I'm talking about the language, really.

Matt:The language. Yeah.

Richard:None of this effects scala or closure at all.

Matt:No.

Richard:Oh, you're a Kotlin fanboy. Yeah. I don't think. I'm sure they're laughing at this.

Matt:Yes.

Richard:In fact, this is a ... Yeah. It is complicated, because of course, combined with all of this, if you just look at the new features in Java 11, for example, a lot of it is virtual machine improvements. So, yeah, okay.

Richard:It is impacted, it is sort of wrapped up together. But, I'm not going to say ... Should we look at the features that are in Java 11, the technical features?

Matt:Okay.

Richard:I've had a skim down - 17 notable features and changes. I haven't explored any of them in any great detail, but they are largely ... I can't see any major new programming upgrades, such as the var keyword that came in on Java 10. So, that's a big deal.

Matt:Oh, so that came with Java 10, didn't it? Yeah.

Richard:Yeah. There's nothing like that. The only one that looks like ... nest based access control, which is JEP 181, could be interesting. But, I haven't explored it. It's related to nested classes, and that there will be some level of access control on nested classes. Well, okay. Could be big. But, just to give a flavour, a lot of the other things are things like there's a new no operation garbage collector. One of the new features is just removing some Java EE and CORBA modules out of the standard JDK.

Matt:But, they've added in this new HTTP client.

Richard:Client.

Matt:Which I think was created for JDK, for Java 9.

Richard:Yes.

Matt:But, they've now sort of ... Well, they're calling it standardised. So, I think that means they've given it an official package. Yes. That's right. It was something like jdk.incubator.http. It's now java.net.http.

Richard:Excellent. And, they've also expanded it. Previously it was blocking, so now it's non-blocking.

Matt:Oh, wow. Okay.

Richard:So, you can do all your web sockets and that kind of thing in there.

Matt:So, we will have to have a look at the Java Web Development Course again. Although, we'll see.

Richard:Indeed. You will.

Matt:We'll see how quickly we do that. It's not on the top of our to do list, if I'm honest.

Richard:And, there's things like new cryptographic algorithms. Dead important. Stuff that really is, but probably not something that effects many people day to day. A few deprecations in there. One thing I'm really interested in is JEP 328, and I have literally just now googled it. So, please don't, you're going to start grilling me with questions, but it's a low level feature, this. It's not language level. It's specifically for the JVM, the hotspot JVM. And, it's going to be a data collection framework within the JVM.

Richard:So, you're going to be able to do more enhanced monitoring of, actually, not monitoring, not really time monitoring. But, when you have a crashed virtual machine, you're going to be able to find out what went wrong and why.

Matt:Oh. Okay.

Richard:So, could be a fantastic upgrade there to our Java memory management modules.

Matt:Yep.

Richard:It just looks ... It's just triggering my little ... My nerdy radar is triggering just looking at it. I think there's going to be some good fun in there. So, that's JEP 328, the Flight Recorder.

Richard: Just the fact, can I do this rant ... It's not a rant, this, this is one of my real things I keep going on about. The things that are successful in our industry tend to be the things that have been given a good name.

Matt:Right.

Richard:And, just the fact you've called it Flight Recorder, I get it already. I've not even looked into it. I get it. I can see where it's going to be useful. Beautiful. Beautiful. Even Jigsaw, beautiful. Nice name. Doesn't mean that's one's going to be successful, but ... Well, it has to be, doesn't it? We're forced for it to be successful. But, look, beautiful name. But then, when you call things like the ... I'm trying to think of some examples.

Matt:I'm looking down the list now of Java. Yeah.

Richard:The Java Persistence API. Boring, isn't it? Call something Hibernate, and you kind of get that, "Oh, yeah. This'll be nice fluffy."

Matt:Yeah.

Richard:Anyway, I've done that about a million times on this podcast. I won't do that any more.

Matt:Okay.

Richard:Flight Recorder looks really interesting. But-

Matt:Good. Okay. So, expect over the next month, few months, that there might be some changes. And, we'll notify you through the news page, through the usual channels, as we change things to let you know what we've updated.

Matt:But, what we are both really working on right now is Spark.

Richard:Spark. Yes. We released module one of Spark back in February of this year. So, it seems like a long time ago, but it really wasn't. And, the intention was always ... So, the module one focuses on the RDDs, were sort of the core part of Spark. The older part of Spark, really, is ... I think it is a beautiful framework to work in. But, it's still ... The RDD part of Spark is still a little bit Hadoopey, a little bit map reducey. I mean, it isn't. It's much richer and nice to work with. But, the RDDs are still quite low level. Basically, you're working with very raw big data sets, typically just text files, often ...

Richard:Anyway, the intention was always that we'd have a follow-up module two, module three, maybe even module four. Not sure about that one yet. But, we would go on to do Spark SQL, and then a module on Spark ML. So, that's dealing with machine learning. So, that was always the intention. And, I get a sense no one was kind of screaming at us. No one's said, "Where's module two?" Which is a bit disappointing, really. But-

Matt:We have had a few people, though, asking that.

Richard:Yeah.

Matt:And, I think the ... Yeah, we have had a few people. It's not been screaming, but the machine ... So, where are we up to date with that then? So, just for the benefit is, we are working together on these courses. I'm doing some of the writing, Rich is doing the recording and further research, and so on.

Matt:So, module two, let's start with that one, which is the Spark SQL part. So, that's fully written now and-

Richard:Yeah. It's all good to go. So, just to ... Anyone interested in learning Spark SQL, let's give them an overview of what it's about. So, go back to ... traditional Spark RDDs, as I say, you're typically working with very raw, sort of non-tab... I'm trying to explain a typical input into RDD would be a long list of log messages or-

Matt:Yes.

Richard:... transactions, or very unstructured data, typically.

Matt:Yes. Yeah. Because, although you could work ... When you write code with Spark RDDs, it's a bit like writing code with a list or a map.

Richard:Yes.

Matt:So, you're thinking in terms of a list of strings. Or, if you're working with a list of customers, and you can absolutely do that, but it doesn't really work that well. Because, as soon as you start manipulating the data, is the output still a customer?

Richard:Yeah. So, your input there would be ... You might have a massive text value, key value pairs, maybe. The key could be a customer ID, and the value might be a great big long string of comma separated values or something.

Matt:Yes.

Richard:With the traditional Java RDD framework, you've then got to take that raw string and parse it, and split it up, and probably discard what you don't ... It's tedious.

Matt:It is. But, the one nice thing about RDDs, of course, though, is that you can write a lot of that code using the lambda expressions.

Richard:Oh, yes. Beautiful. Beautiful.

Matt:So, you are using functional programming techniques to work with it.

Richard:And, it ... I'm just saying this just to dissuade anyone. We have had people say, "No one uses Spark RDD anymore." And, it's true that the place where the money is, is Spark SQL now. That's really hot, much hotter than Spark RDD. But, don't ... You have to learn, but you can't just go straight to Spark SQL in my opinion. You want to learn RDD as well. It's got some beautiful use cases. So, you need it.

Matt:Absolutely. And, there are some thing ... Spark SQL is using RDDs under the hood. So, it's an abstraction on top of RDDs. And, there are some things where you will get better performance using RDDs.

Richard:Definitely.

Matt:And, in a production environment ... And, of course, the other thing is you can switch between the two. So, because it is an abstraction, it's not too difficult to go from one to the other, is that, actually, in a production environment you might well be using RDDs to get the job done in the most optimal way. So-

Richard:And, it's also a heck of a lot easier to understand your performance characteristics when you work with RDDs.

Matt:Absolutely. Absolutely.

Richard:'cause, you've got all that abstraction on top of Spark SQL. I'm still terrified. In fact, you just don't look at it, do you, at Spark SQL?

Matt:No.

Richard:Just ... It works well enough. But, we understand what Spark SQL is. So, I admit, when I started working with it, I assumed it was going to be, "Oh, this is how we use Spark with databases." And, that's not the case.

Matt:Absolutely. So, it is ... I would described Spark SQL as being a way of writing code that is more like how you think about writing database statements. So, there are various ways to interact with Spark SQL. One is to write SQL like statements as though you were using a relational database.

Richard:Yes. Which, gets us back more to ... I mean, we're developers. We think like developers. But, of course, data scientists are heavy uses of this technology as well.

Matt:Yes.

Richard:And, they don't want to think in codes. And, I imagine map reduce was a real nightmare for them. But, now we've got back to third generation language thinking, that you can write what looks like SQL style query language, not ... You can use this against databases, but this query could well be exploiting a text file, a big data text file-

Matt:Yes.

Richard:... that you've got some logs or something. And, you can interrogate it using an SQL like syntax. And, it's all tabular. That's the important thing. You're not dealing with ... Your input might well be a comma separated value, comma separated string. You don't need to worry about that. You just think in columns.

Matt:Exactly. And, actually, that's the key, is that even if you don't work with that "select * from" type structure, if you're writing it more as code, you are using ... Everything is an object. So, a column is an object, or a row is an object, and you are interacting with them in a very different programmatic style to when you're working with the RDDs. And, in fact, it's not functional programming, is it, because of that?

Richard:Yeah. The functional programme. I was disappointed at first, 'cause my big thing about the Spark RDD is it is a great, for me, it's a gateway drug into ... If you've looked at lambdas in Java, and you're kind of put off by them and you don't get them, or whatever, learn Spark RDDs. Even you're not interested in Spark RDDs, learn them. Because, you're going to find yourself using lambda expressions really fluently. And, it's simple and straightforward. It's beautiful.

Richard:When you get to Spark SQL, just because the model is different really, I thought at first it was, "Oh, they've not bothered implementing lambda for Spark SQL." It's just ... It's hard to explain really, but they're not needed. They're just not needed in that model.

Matt:Exactly. Because, it's a different ... The underlying data structure in RDDs is something we think of as being like a list, right, isn't it? So, an RDD is a bit like a Java list, right?

Richard:Yep.

Matt:Whereas, the data structure for Spark SQL, which is this dataset, is different. It's not like a list. And you don't-

Richard:It's a table.

Matt:It's a table.

Richard:Not a map. It's not key value pairs.

Matt:No.

Richard:It is a table with columns.

Matt:Yes. So, it's a very different way of thinking. And, I think the transition from one to the other is about changing your way of thinking about how you're trying to achieve the same thing.

Matt:So, that is Spark SQL.

Richard:So, that's going to be module two. And, very sorry that this ... It should've been released today, was actually going to be the release date. We've not announced it on the website, have we?

Matt:No, we haven't.

Richard:Probably should have a "Coming Soon" on the website.

Matt:We should.

Richard:It would've been today. Or at least, principle recording was due to be completed. But, I don't know if our millions of listeners can tell, but I have a cold at the minute. I can't record. So, it would just be impossible to do it. So, we're a bit delayed on it. But, it's ready to ... It is ready to pump out

Matt:So, hopefully within the next couple of weeks we'll have that one live.

Richard:And, I hate to ... This may well break things. But, I think if you go through module one, it's quite hard work, quite a lot to learn, we go quite deep, I think we go deeper than any other course, not that I look at too many other courses, but I think we go pretty deep into the performance, the underlying model. It's hard work. When you've got all that under your belt, the module two, there's no deep concepts. It's just another API. We don't labour it on module two, we just get straight in, start doing some examples.

Matt:Yeah.

Richard:It's conceptually easy. You're saying it's a big change. I didn't really get that-

Matt:It's more of ... No. It's sort of a mindset change. And, I think you get that as ... It comes naturally as you work through and you realise how this is working.

Richard:Yes.

Matt:But, certainly, it's useful to be able to know, "I can do this operation in either paradigm," right?

Richard:Yeah. Yeah. Yeah.

Matt:So, definitely.

Richard:Brilliant. Two sets of tools. So, that is going to be due out with Virtual Pair Programmers very soon. We'll give it a date soon. It has been delayed. But, we're there with it, looking forward to releasing it.

Richard:But, the one I'm really looking forward to will be module three, which you have been beavering away at.

Matt:I have. And so, module three is machine learning.

Richard:Yeah.

Matt:And, I have ... This has been a really hard module to write. So, first of all, what is machine learning? And, this is my ... Oh, I think we've talked about this before, Richard. I think this is ... My best summary of what I can describe machine learning is, is a new way of describing a technique that's been ... Sorry. No. A new way of automating a technique that's been around for a very long time, which is using statistical models to create models of the real world.

Matt:So, I used to work in a bank in IT. In that bank, they used statistical modelling to decide which customers they're going to lend money to and which customers they're not going to lend their money to. And, machine learning is a way of automating the process of coming up with the best model given the data, right? And so, they would have done a lot of this manually. Machine learning allows you to automate some of it. That's what this is about.

Richard:Yeah.

Matt:And, the challenge for this module is we can't sit here and teach the mathematics behind each of the different statistical models.

Richard:Okay.

Matt:Partly because, I think it's way outside the scope of what we can do. And, I don't understand all of the models that we are going to look at.

Richard:Sure.

Matt:I'll be honest with you.

Richard:Yeah. I don't think there's a problem with that, is there?

Matt:There isn't, but I think that part of the difficulty here is that ... Well, let me go ... I'll come onto that. The way I've structured this course is that we're going to take one model type, which actually's going to be a linear regression, and we're going to learn how you build a linear regression in Spark. And, actually, the way that works is we're going to do a really simple one, and then we're going to add elements to it to make the process complete. That is a big job, and at the moment that is eight chapters.

Richard:Wow.

Matt:But, on one model type. However, once we've done that, and we understand everything about the process of Spark modelling, what we'll see is it's two lines of code to swap that out for a different model type.

Richard:Yep.

Matt:Because, and this'll probably be no surprise, every aspect of modelling in Spark ML is implementing one of two types of interface. So, if you want to change one model type to another model type, because they implement the same interface-

Richard:Beautiful.

Matt:... it's a simple swap. Now, there's a bit more to it.

Richard:Oh, that is nice.

Matt:But, I have looked at a lot of books and courses and so on about how to do Spark ML. And, everyone focuses on, "There are eight different models, and this is model one. This is model two." I think a better way to do it is, "Let's understand the complete process and then complexities of that process, and then it's just a simple case of swapping in the right model at the end."

Richard:Gorgeous.

Matt:So, what we will be covering is something ... Well, first of all, model accuracy, which is a key topic. How do you know, once you've created your model, that it's actually any good? And, actually, that is one of those things that changes based on the model type. Because, your assessment of a linear regression model versus, let's pick one, at random a K means model, they'll have completely different metrics.

Richard:Okay.

Matt:But, the concept at least of that. And, this is the bit that I was going to say is the difficult bit, is when you create a model, and this is really hard to try and talk about when we are not halfway through this course already. Let's say you're building a linear regression. So, for the sake of argument, a linear regression model is something like, and let's use that banking example. A customer will be good if they score above 50%. And, you calculate your 50% as being 5x the number of years they lived at a particular address, and 3x the number of years they've been employed, and ... So, you're building a formula.

Richard:Yep.

Matt:Now, there is a statistical algorithm that generates that formula, but that algorithm needs input parameters. And, you've got to decide, "What are the input parameters?" Now, without a degree in statistics and a full understanding of the algorithm, as well as the statistics, it's pretty impossible to come up with sensible looking parameters.

Richard:Okay.

Matt:So, this is the bit that is a bit woolly and a bit vague, is we're sort of saying, "We're using some input parameters at random." And, that just doesn't feel quite right. If you were a mathematician, you would probably, I'd hope, have a better idea about the right starting parameters. And, I am waiving my hands around here as I say this. However, there is one redeeming feature, which is, and this is quite advanced, and we cover this as part of this process, is that Spark allows you to build what's called a parameter grid, which means you can say, "I don't know whether, for this particular model, the right input parameters are (one, one, one), (two, two, two), 100, 100, 100, or whatever." So, you can build a grid of these with lots of combinations, and let Spark run that model thousands of times and work out which produce the best model.

Richard:Yeah.

Matt:So, even not understanding it, you can still hopefully get a good outcome. So, we go into all of that.

Richard:Excellent.

Matt:And, the other big, I think, difficult topic, it's not that difficult once you understand it, is, if you're trying to build a mathematical model, and like I just said before, you've got features like the number of years someone's lived somewhere, the number of years they worked somewhere, you might also have their job type. And, their job type could be anything from teacher, to manager, to cleaner. So, there's a way of giving those a numerical value.

Richard:Yep.

Matt:And, it's called indexing. But, there's a clever technique to do it so that you don't assume a cleaner would be better or worse than a chief executive. Because, actually, in a particular example, you don't know a cleaner would be better worse.

Richard:Yeah. Of course. Yeah.

Matt:So, there's a whole way of doing this, which is quite complex to understand. It's called one hot encoding.

Richard:Right.

Matt:So, we cover all of that.

Richard:Oh, you love all this, don't you? This is really-

Matt:You know, it's been hard work understanding all of this-

Richard:I knew it ... I know.

Matt:... in a way that we could then try and explain this well.

Richard:But, I knew it would be right up your alley.

Matt:It has been.

Richard:So, can I be devil's advocaty here?

Matt:Yes.

Richard:And, I mean I know the answer to this, but I'm a developer. Why should I be doing a course on machine learning? What's the interest and relevance to me as a developer?

Matt:Well, I'll be interested to hear what your answer is to that. But, just going back to my career, when I worked for the bank, which was in IT, one of my jobs was to implement a mathematical model that was actually calculating something slightly different. It was calculating the amount of money the bank needed to hold in reserve to make sure it never went under if lots of people didn't pay them back. Let's not talk about the success of that model. But, my job was just to implement it, not to do the maths.

Matt:And actually, I think there's a blurring. Certainly math ... that bank employed, as I'm sure all banks do, teams of statisticians. They called them Development ... Sorry. They called them ... It's gone now. Something scientist. Data Scientists.

Richard:Oh, yeah. Of course. Yeah.

Matt:Yep. And, they were writing code for a statistical system which was called SAS. The code's a bit like R, I think. But, they are basically writing code-

Richard:I think R sisopen source version, is it?

Matt:Yes. I think you're right.

Richard:Yeah.

Matt:So, they were writing code of a form that was really about allowing them to do maths. So, there is a crossover now, certainly. But, actually, if your job is to productionize and implement these systems, using machine learning allows you, certainly, to integrate it into other systems much more easily. But, it does take away a lot of the drudgery of doing heavy statistical analysis.

Matt:What's your answer?

Richard:What I was really getting at with that question was, just to rephrase the question in a business way, and this is what we will have gone through when we decided to do this course, is who are we targeting this course at? Now, there are millions of courses out there on machine learning, and Spark, and all that kind of thing. You'll probably find, this is another angle to my question, that they're either written in Python, or R, or possibly Scala. Scala is kind of the natural language of Python anyway. But, sorry, of Spark.

Matt:Yes.

Richard:So, let me say that again. I'm listening to what I'm saying. Yeah, I just ramble. I don't listen. Scala is the natural language of Spark. That's right.

Matt:Yes.

Richard:So, why are we doing a course on how to use Spark and machine learning with Java, which is not its kind of natural forte? And, who are we targeting this at? And, I think we are being asked by a lot of Java developers, "When are you going to do a course on machine learning?"

Matt:Yes.

Richard:I don't think many have asked specifically for Spark ML. They've just asked for ML.

Matt:Yes.

Richard:So, we're using Spark as our gateway into machine learning. We think this is quite a nice, easy way into it. There's lots of other frameworks we could be using, scikit-learn, and all that sort of thing, and there's a whole forest of them.

Richard:So, I'm rambling around. I have in my head, and I think this is me, basically, I'm not a data scientist. I've never worked in, that I can think of ... I'm just wracking my brains. I haven't worked in a bank like you have. I think my early career I did do some mathematical modelling, but that was in a different context. But, it's not what I would think of as my ... I'm the middle tier. I do databases and that kind of thing.

Richard:But, it's obviously huge right now. All of this stuff has already existed, as you were saying before. It's just repackaged. But, there's a-

Matt:It's making something that was inaccessible to the non-mathematician-

Richard:Yes.

Matt:... accessible, at least, to the programmer.

Richard:Yeah.

Matt:And, for our own business, for Virtual Pair Programmers, we are constantly asking question of ourselves like, for example, "How do we detect a fraudulent customer?" Right?

Richard:Yes.

Matt:Which, machine learning is a technique for doing that. So-

Richard:And it's perfect for that, isn't it?

Matt:It is.

Richard:Now, traditionally, we would've implemented looking for fraudulent customers by sitting down and coming up with some brilliant algorithm. We'd be so pleased with that. After a days work, we'd go out to the pub and celebrate. "Aren't we clever?" And then, two days later someone will have, one of our enemies, will have worked out what algorithm we're using. And now, they're busy defeating it.

Matt:Yeah.

Richard:So then we go round and round.

Matt:Whilst, in the meantime, we've blocked our best customers by mistake.

Richard:Yeah. Absolutely. And, apologies to anybody we have done that to. It's a terrible thing. It's like anti-piracy measures on DVDs. The only people you hurt are the good customers. You don't hurt the pirates, 'cause they've already broken it

Richard:So, this is a much ... And so, if we hadn't known about, if there was this complete blind spot to us, we wouldn't know there's a whole set of tools here where we can let the machine work with the data that it has and let it come up with what ... let it come up with the algorithms.

Matt:Yes.

Richard:We wouldn't have known that was a possible line of inquiry for us. So, it's about broadening your tool set, I think.

Matt:Yes.

Richard:But, I think, just in general, it's that right now, machine learning and the whole field of data science is so vibrant that as a working developer, even if you're only interested in developing, you're missing out if you're not at least dipping your toe in the water of that area.

Matt:Yeah. And-

Richard:And-

Matt:And, the process of machine learning is about exploring and understanding your data. It's not about productionizing a system to calculate things, right?

Richard:Okay.

Matt:So, that's because the output of machine learning is an algorithm you then productionize if you want to, right? So, if you'd done a linear regression, and that means you've got a formula for determining your good customers, you wouldn't implement that in your production system in Spark because that's a massive overhead for what would actually be a simple calculation. So, what Spark does, is it gives you the formula for you to actually go and implement. So, it's an investigatory tool. It's tool to be used by developers.

Richard:Okay. That is very interesting. But, yeah, I still think your language of choice wouldn't be Java for this. It'd be Python, surely.

Matt:Well, your language of choice would be the language you are more familiar or most competent with. And, if you are a Java developer, and there's an awful lot of them out there, but you don't yet know any Python, you don't need to learn Python. You don't need to learn Scala. Anything you can do with ... And, Spark is a fantastic machine learning environment, right? So, anything you can do in any of those languages with Spark, you can do with Java. And, it's no ... Okay. It might be a fraction more vernbosethan Scala-

Richard:Yes.

Matt:... but, it's a fraction and not worth the overhead of learning Scala if that's your reason for learning Scala.

Richard:Yes. So, it lowers the cognitive load if you want to dip your toe in the water of this.

Matt:Absolutely.

Richard:And, then maybe you could go and learn a bit of Python or ...

Matt:Yeah. I'm not saying don't do it. But, I guess the other thing is, it's also about this interacting with other systems you might have. So, if you've already got a data warehouse with code that reads from it using Java, well, plugging Spark on the back of it is a no effort-

Richard:Okay. Interesting.

Matt:So, again it's ... There's no compelling reason. If Spark is your goal, well if machine learnign with Spark is your goal, there is no reason if you don't know another language to go off and lean it, I would say.

Richard:Yep. Okay.

Matt:There's no benefit of it.

Richard:Okay. So, I'm a little alarmed by what you're telling me though, 'cause we do have a hard deadline on module three. This has to be out ... So, we know module two's coming quite soon.

Matt:Yes.

Richard:We don't have a date on it. But, module three, and therefore module two as well, must be completed by the end of October. Now, what you're telling me is this is a mammoth, roller coaster of 84 chapters just to get you-

Matt:So, it's 12 chapters at the moment, of which eight are the process of model building with one model. Okay?

Richard:Yep.

Matt:And then, the remaining chapters look at four different models. And, just as an example, chapter nine, which is the first of the new models, is going from a linear regression to a logistic regression.

Richard:Okay.

Matt:I suspect that would be a 15 minute chapter. So, the hard work is definitely in the beginning. There are two types of models called supervised, and unsupervised learning. So, the first section is all supervised. Moving from that to unsupervised, which is ... Supervised learning is where you have data with an answer, if you like. So, this idea of good or bad customer. If you know who a good customer is and who a bad customer is, you're using that data to come up with a formula. Unsupervised is when you don't know an outcome, so you're looking for connections.

Richard:Yeah.

Matt:So, that is a slightly different ... It's not different code, but it's a different mindset. So-

Richard:You brilliantly just described there what I was sort of angling around a bit earlier, is that, although this is a course for developers and all that, you're just proving that these are good data science concepts. This is not a course that is going to get you into a data science job, I don't think.

Matt:No.

Richard:We are not trying to teach people to be data scientists. But, exactly what you're describing there is exactly what would be on a good data science course. So, you're going to pick up a lot of really good-

Matt:Yes. And, the bit that you're missing, that would get you the top data science job, is the in-depth maths and statistical knowledge, I think.

Richard:Yes. And experience, as well. Yeah.

Matt:And experience, of course. But, certainly as a programmer supporting a team of statisticians, you'd be well in. And-

Richard:I am so excited... I think this is the most exciting release we've had for years, ever probably.

Matt:Yeah. I just want to mention one final thing, which is there are two versions of Spark machine learning. One to use with RDDs, one to use with Spark SQL.

Richard:Okay.

Matt:This course is the Spark SQL version because they have actually announced that they will not support in the future ... There is an end of life planned for the RDD version. So, we....

Richard:That's just for ML?

Matt:That's just for ML. So, the Spark ML bit you run after you have done all the data manipulation you need to do in Spark generally to get your data in the right format. And, that, of course, you can do with RDDs. But, you then convert to a data set before you run Spark ML.

Richard:Sure. No problem with that.

Matt:That's just the message there. It just wraps to where we were talking about before.

Richard:Just to be clear, they are not deprecating the RDD API.

Matt:No, certainly not. No.

Richard:Good.

Matt:Just for the machine learning part.

Richard:I'm very excited by this thing. It has to be out by the end of October. We have a hard business deadline on that.

Matt:So, it is pretty much written, I've actually got one chapter left to write, which is one of the ... It'll be on matrix factorization. That's the final model we're going to look at, which is what's used for recommender systems. So, that's going to be the last one. But, I will mention that in order to write this course, we have at the same time been building up models for ourselves. So, we're actually going to use a couple of our own case studies in this course.

Richard:Fantastic. That is amazing.

Matt:Which, I know I've shared with you already. But, for the purposes of everybody who's out there, our listeners out there, one was all about free trials. So, when a customer takes a free trial, are they likely to be a good customer or not? And, my thinking behind that was, if we can ... Sorry, are they likely to convert to a paid customer. And, if we could find, by doing some analysis, that customers who watch a minimum of, say, two hours within the first day always go on to pay, then we want to be doing more to encourage customers to watch more earlier. So, it's that kind of idea. So, it's more of an investigation into, "How do we identify who are going to be great customers so that we can make more of our customers great customers?"

Matt:And, the other one was around actually the opposite. So, customers tend to leave when they've stopped watching, as a general rule. So, can we predict who might be about to stop watching our videos? So, who's tailing off, that kind of thing. So, they're the two case studies we've got. We do have our own internal one about fraud analysis, but, again, I didn't feel, unfortunately, we could make that public in a ... Because the metrics we're using on the course, it would obviously have to be a cut down version so we're not giving away customer information. So, unfortunately, for the fraud one, it does use a lot more around attributes to do with the customer that we wouldn't be able to release publicly.

Richard:Okay.

Matt:Whereas, chapter views is more around things like, "How long have they been a customer? How much of our catalogue have the watched already? How much did they watch last month?" A bit more generic things-

Richard:Sure.

Matt:... that we can do in an anonymous way. So, yeah.

Richard:This was the thing I very much liked about module one of Spark, was that we used two or three, was it, examples which were real.

Matt:Yeah.

Richard:You built the ... So, there's an example in module one, which is, we take raw data of views of a course, our real life case study as you say, and you have to ... and it is all you do it yourself, although we have a guided walkthrough of it as well. But, you're expected to try to do this for yourself. You have to come up with a league table of the best courses. And, I criticised you saying, "This is too hard. It's too gruelling for a training course."

Richard:And so, it caused some tension there. But, I absolutely ... Fine. The end result of that is you've done some real work that we had to do.

Matt:Yes.

Richard:And, it is not a trivial, "Let's count a few words," like you often get on training courses. It's gruelling. But, when you get there, it's very satisfying.

Matt:And, I want to be really clear that in order to do the case studies for this course, all of that has been done in advance. So, what you get, for example, is one data file in the right format with the data cleaned up for ...

Richard:Yeah.

Matt:Because, I have effectively replicated that process we did in module one, and done all of that off camera.

Richard:Yeah. There's no point.

Matt:Because, we don't want to waste time doing that.

Richard:No. Absolutely.

Matt:But, I want to be clear. In order to be able to do the case study on chapter views, I spent a good few hours writing the code to do that data manipulation and get it in there. So-

Richard:Yeah. Which was the same process, similar process, to our case study. Yeah.

Matt:It's very similar. So, it's about linking data together, about identify outliers, all this kind of stuff. And, if you're going to be doing machine learning in a productive environment, you will still need to do that part of the job.

Richard:Yeah. Yeah.

Matt:So, this is not going to be a stand-alone module.

Richard:Yeah. Right. Okay. I'm trying to brag about our courses really. We care about this. We're professional trainers, so therefore, we don't ... We don't just do a knowledge dump and then expect you to do really hard academic examples. But, we don't do "hello worlds" either. And, probably the reason why we take so long in producing them is we are constantly trying to find that balance to make you work hard enough that you get a lot of value out of the course without it being too trivial. And, we've guided you through.

Matt:Yes.

Richard:And, it sounds like you've done a brilliant job on the writing for this one.

Matt:I hope you think, though, when you look at it-

Richard:Can't wait.

Matt:We start simple. The first example is a competition in a gym. So, it's something like, you've got a list of 20 people in the gym, and you know their gender, their age, their weight, their height, I think. And, it tells you how many of a particular exercise they were able to do within five minutes.

Richard:Right.

Matt:And, the idea, is, can we predict from that data ... Well, actually, we use it twice. One is, can we predict from that data when the next competitor comes along, how they're going to do? We also use it for trying to group these competitors into classes. So, you know if you do a running race, you get veterans, and you get under 30s. You get groups, don't you?

Richard:Yeah.

Matt:Well, can we group these people so you could identify somebody who's performing particularly well for their group-

Richard:Right.

Matt:... given their age, and height, and weight, and all this kind of stuff.

Richard:Okay.

Matt:So, I wanted to pick a simple example that would still have some value. And, of course, that one's fabricated data, just so we can start.

Richard:Yeah. So, that's getting you up and running with it-

Matt:Exactly.

Richard:... and then you go and apply it.

Matt:And, we also do, because I think you have to if you're doing machine learning, we do a house price predictor. So, you have to do that. But-

Richard:Well, one of the problems is, of course, there's not that many big data sets available publicly. So, house prices is one that is publicly available from a long time ago.

Matt:Yes. And I picked ... Yes. I've picked a house price index or Kaggle for that.

Richard:Right.

Matt:I'm not sure if people are aware of Kagle. It's a site for getting large data sets. But, of course, the problem -

Richard:That's where you do the competitions, isn't it? You compete -

Matt:You can do competitions. I've not tried to do one of those yet, I confess.

Richard:Well, we'll definitely make that part of the course introducing people to Kaggle. If you're going to get into this thing, Kaggle is the place where you hang out, data scientists.

Matt:Yes. But, of course, one of the annoyance factors for me is that a lot of the work you have to do, if you pick one of these data sets, is that tidying up. And-

Richard:Yeah. It is. It's 90%+ of the work in data science, is cleaning.

Matt:It is. And so, I found a house price index in Kaggle that didn't need much tidying up. And, that's the one we're using, purely because it ... It allows us to focus on the learning of building a model, right?

Richard:Mm-hmm (affirmative).

Matt:And, it's an interesting one. I think it gives ... You can build some reasonable models with it.

Richard:Yeah.

Matt:So, anyway, that's Spark machine learning. We're both quite excited about this course.

Richard:Yeah. But, it has to be out by the end of October. Hard business deadline.

Matt:Well, I am officially handing over to you everything I've done so far today. And, I will be writing the last chapter in the next week. And then, it's over to you to the next stage of editing, and recording, and so on.

Richard:Right.

Matt:So, watch this space.

Richard:It's going to be a very pressured month, so there probably won't be much in the way of blog posts or podcasts. So-

Matt:I'm sure there won't be another podcast until this course is out. So-

Richard:Absolutely.

Matt:But, we will be working on that. So, apologies if you think we have been silent. But, we've not disappeared.

Richard:Yep.

Matt:Honestly, still here working very, very hard, hopefully producing some good material for you.

Richard:And, yeah.

Matt:So, shall we leave it there for today?

Richard:Yep. So, we'll do another podcast ... We'll do one when module three is released.

Matt:Yes. So, let's say beginning of November, 'cause this is going to be out early or end of October. But, I hope you found this in some way interesting. I don't know how interesting it's been.

Richard:Well, we also don't talk a lot about Java, do we, which is interesting that ... The Java 11 thing, we were trying to appear interested and excited in that, and we're not.

Matt:No.

Richard:Because, all the work has been done in frameworks, and libraries, and so on. So, there's a good chance a lot of our listeners are not interested in Spark, which is a shame.

Matt:Yes.

Richard:But, you know, it's a-

Matt:We did have ... And, actually, just 'cause you mentioned there might be a module four, I had an inquiry-

Richard:Yes.

Matt:... I can't remember if it was this week or last, a couple of weeks ago, asking about streaming.

Richard:Yeah, I forgot all about that. Yeah.

Matt:And, if we do a module four, it will be streaming with Spark.

Richard:Yes.

Matt:Or, at least it will be streaming, possibly with Spark, and possibly not with Spark. We need to look into that. But, that's what the topic would be.

Richard:Yeah. And, I think next year our focus for new releases has to be that kind of thing, plus Kafka, reactive.

Matt:Yes.

Richard:I've done lot of work on microservices and dev ops in the last year or two. And, I think the Kubernetes release was a good one. Really enjoyed it. Fantastic course there. But, I think I'm kind of done with that now for a bit.

Matt:Yes.

Richard:It's still an evolving area and all that, but I think we've tied off a lot of loose ends there. I think that needs to be the 2019 focus, personally-

Richard:... but I might be wrong.

Matt:The latest and greatest frameworks, and projects out there.

Richard:Yeah. And we are ... There's no doubt about it. We have to admit, we're behind in that area.

Matt:Yes.

Richard:And, we've got to catch up. So, that is a pledge for-

Matt:And, what will slow us down a bit is that we do need to go, which, I say we, it will probably be me, go through the back catalogue a bit and work out-

Richard:It's holding us back, isn't it? The catalogue.

Matt:Yeah. What needs to be either changed and updated, or retired. And, I think we're absolutely not now, I hope, going to sit and re-record things. So, when we went from Spring three to Spring four, it was a re-record. I would hope now it would be an extra few chapters explaining, "The fundamentals are still valid. If you're using it ..." So, actually, Android. I've done an update for Android, which is exactly that. It's one extra chapter in module one, and it is an errata for module two. Android changed the default layout, which means that every screen that you build is slightly different.

Matt:So, we've updated all the code. We've explained it upfront. I believe that's sufficient. And, it's not practical. And, actually, particularly for that course, which doesn't get huge viewing numbers, it really is impractical to a re-record.

Richard:Yeah. No.

Matt:Because, of course, they might change it again next month.

Richard:Sure.

Matt:And, this is the problem. So-

Richard:But something like Spring, which is very much core business to us, we've neglected it, frankly.

Matt:Yes.

Richard:All these Kubernetes Docker, and Spring Cloud, as well, which we covered very well. All these ... We've got a lot of that catalogue there. And, I think we tend to assume that ... Our customers are, obviously, top developers. Otherwise, they wouldn't be spending money with a company like us. So-

Matt:Well, they end up as top developers. They might not start that way. They're people who aspire to be.

Richard:They are top, motivated ....

Richard: And so, I think we kind of do assume that they're not going to go, "Wow, this is a disgrace. This Spring 4. I want Spring 5." I think they get that the differences are not massive, and they can easily pick up the differences. We are working on the core anyway.

Matt:Yes.

Richard:However, it does ... Spring Security is a good example. We've got some fantastic courses in our library on Spring. I'm so proud of them. Three big modules, though, and there's probably stuff in that third module that we didn't really need to do. I was just throwing everything I could at it. In fact, the third module we left a note saying, "This course is still under development. We've got a couple more chapters to add. We'll let you know when it's done." Never got around to it.

Matt:Right.

Richard:And, nobody ever mentioned it. So, it became obvious that no one wanted those extra topics. But, there's a good example of where that needs a re-record. It all needs turning into another module. Because, things like ... I believe we've got the best training course in the world on OAuth, OAuth 2.

Matt:Right.

Richard:We just ... it is. It really is. And, I've compared it with a lot. But, it's buried in a module. It's not quite obvious it's on OAuth course. And, now, in Spring, OAuth is a core part of Spring Security. You don't need this extra library. So, it's dated. And, it needs a re-record because it's one of the best ... It's the best course in the world at OAuth, so we should be selling it and making a big deal about it.

Richard:But, just as I was ... I was just looking into re-recording it, and I was having a conversation with somebody who's very into our microservice courses. And, I said, "Yeah, I won't be doing an microservices for a while because I'm thinking of re-recording Spring Security." And they said, "What? Don't do it. No one uses that now. You're going to use the security of your cloud provider. And, you don't need it for microservices." Well, of course, for micro ... The architecture that we have on the Kubernetes course, for example, the front end is in Angular. So, you're not going to be having customers ever logging in or using Spring Security, 'cause it's JavaScript.

Richard:And I realised, "Yeah. It's just not compatible with our latest releases."

Matt:Right.

Richard:It doesn't mean we can't go back and redevelop.

Matt:Yeah. It's tricky.

Richard:It's a lot of legacy to deal with.

Matt:Yes.

Richard:Just to close then, we've got courses on WildFly in Java EE at the minute. And, of course, that's now moving to Jakarta EE version one. It's going to be version one, I think-

Matt:Yes.

Richard:... is due out soon or something. Don't ask me any details, 'cause I don't care.

Matt:No. And, that is ... yes. We have four modules for Java EE. There is a core of customers who do look at those, definitely. And, we'll need to take a view as to whether we update that or whether we say, "Do you know what? It's not our core."

Richard:I think we should ... Yeah. We should fight. Because, who's going to ... I'm not having no problems with WildFly, and I wrote and recorded the original Java EE course and was very pleased with it.

Matt:Yes.

Richard:And, at the time it was, "A good developer needs to know both major frameworks so they can compare and contrast."

Matt:Yes.

Richard:I had to work in both of them, and I much preferred Spring. But, I had no problems working in Java EE. I think time has moved on now. And, as a business, now that we're a subscription business, that's the key difference-

Matt:Yes.

Richard:... if you need to learn WildFly, I would say our courses are brilliant for that. So, you come in, you subscribe, you do WildFly one, two, three, four. Where do you go next? You're at the dead end. So, you're going to unsubscribe. So, as a business to us, it's not worthwhile.

Matt:No, exactly. And, I think in terms of our platform, we have built our platform to be a way for people to learn over time a vast range of techniques and skills that are relevant to them and their jobs. And, WildFly-

Richard:Well, maybe even not relevant, though. I'm going to argue.

Matt:Well-

Richard:Like, Android, for example. I would say, yeah. Brilliant. If I'm a Java developer and I'm only interested in databases, it is not going to do me any harm to learn how Android works and how ... the challenges an Android developer faces. And, I've said before on the podcast, it sounds insulting, and it really isn't, when I did the Android course, I recorded module one, and watched, and edited module two, I know at the end of that I never want to apply for a job as an Android developer. Because, it doesn't suit my development style that's all.

Matt:Absolutely.

Richard:Nothing wrong with Android. Nothing wrong with Android development. So, it's a valuable thing for our subscribers to be watching

Richard:But, some are why ... Java EE is-

Matt:Well, I think the difference is that if I was applying for a job today, I know that there's a lot of job adverts that say, "Full stack."

Richard:Yeah.

Matt:And, having knowledge of Android, even though you're not going to do it, you don't want to do it, at least gives you the confidence to apply for that job. Because, I have ... I get Android. Well, I get enough to know how it works.

Richard:Yeah.

Matt:In reality, very few jobs are you are building everything from the front end to the back end. Whereas-

Richard:No. Full stack generally you know some Java scripts.

Matt:Yeah.

Richard:Usually.

Matt:Well, yeah. But, again, it adds to your sort of range of knowledge that is generally useful.

Richard:Yeah.

Matt:I understand, most systems are going to be either Java EE or Spring. And, it's interesting that when you look at ... Well, in terms of back end development, when you look at our customers who are doing Java EE, they are not looking, generally, at the vast majority of the rest of our platform.

Richard:Yeah.

Matt:So, it does sit on its own a bit. And, to me, we want customers who are going to stay with us for years, right? Clearly, that's a good model for any business.

Richard:Yeah.

Matt:So, we want to build the things that our core customers want, and are going to want in the future. Java EE just doesn't sit in that space. So, it might be the kind of course that works great on other platforms, but for us it's just on the right place.

Richard:Absolutely. Yeah. You could sell it as a stand-alone course. It'll be a good course. But, the problem is, as everybody can tell, we're not invested in Jakarta, Java EE.

Matt:No.

Richard:We're not interested in it.

Matt:We don't use it ourselves.

Richard:Yeah. We don't use it. To me, it doesn't ... I mean, it is good that it's now moved to ... It's Eclipse, isn't it, who are managing it?

Matt:Yes.

Richard:That's good, and I think it will ... It can't do that area any harm. But, I very, very much doubt that's going to revive it sufficiently that it becomes a hot mainline topic for regular Java developers. I suspect it's at a dead end.

Matt:It's got a long way to go to be up to date and modern. And, of course ... The one positive with Java EE, of course, is that there are companies out there who are sellers of application servers, which means if you are a bank who wants to buy something from somebody, that you can sue if it all goes wrong, you can do.

Richard:Yeah, yeah.

Matt:And so, that's why I think it's surviving.

Richard:That's always been the raison d'etre for it. But-

Matt:Yes.

Richard:Yeah.

Matt:I think the world's moving on. You know?

Richard:Yeah. And, Jakarta are going down the, "We're making things smaller, we're making things more agile."

Matt:Microservices. Yeah.

Richard:Etc., etc. Yeah. But, not interested.

Matt:It's got a long way to go.

Richard:So, we probably could do with retiring some courses-

Matt:Some areas.

Richard:... just to show that we're not ...

Matt:Yeah. Well, also, we don't want stuff that looks very old and very old fashioned. So, we will be spending a bit of time next year going through the back stack and updating, culling a bit, hopefully adding a bit where things are new features, like with Java 11. And-

Richard:Yeah. And, of course we need to build out ... I mentioned JavaScript. Full stack development generally implies you know some JavaScript.

Matt:Yes.

Richard:We need some JavaScript on this.

Matt:Yes.

Richard:I've been going on about, "Well, I'm going to do an Angular," but even Angular now is dated. It's ... React now, and all these sorts of things. A very fast moving area. And so, a bit like the machine. Yeah. I won't go on about that.

Matt:Okay.

Richard:Right. So, we were wrapping up about 15 minutes ago, and then we started on again. So-

Matt:Yes. Well, at least, having been a big gap before this one, and there will be a reasonable gap before the next one, at least it's an extra long podcast, then this time.

Richard:Yeah.

Matt:So, I hope you found it entertaining, if nothing else.

Richard:Yeah. So, we'll see you next time. And, thanks for listening.

Matt:Indeed. Thank you.

Richard:Is that it?

Matt:Well, that's it.

Richard:What a load of-

Matt:We'll leave that bit. We'll say, "Is that it?"

Richard:We'll leave that. Yeah. What a load of rubbish.

Listen via:


Be notified of new posts