Data Trends & Predictions 2024 with DataCamp's CEO & COO, Jo Cornelissen & Martijn Theuwissen

And everyone started to use data. Like what you see now is that data stepped into mainstream, AI stepped into mainstream. And that for me was definitely the biggest round of 2023. And then what's nice is that it's really going offer these customer experiences. Like it's really breaking out of that.

technical part and moving into like, okay, how are we like if you think about data camp, like, and our students, like, okay, how can they use AI to write better code to do better data analysis? And it doesn't matter if you're a beginner or an expert. So, AI entering into. the broader public for me was definitely the trend of 2023.

Richie Cotton: Absolutely. I have to say usually when I tell people that I work in data and AI, like historically, people just try and change the subject as quickly as possible. In this last year, there's been a sort of glimmer of recognition. So I think we are making progress with these things coming into the mainstream.

Jo, do you want to tell me like what you think the most popular use cases have been of generative AI so far?

Jo Cornelissen: Totally. So I think what's super exciting is the explosion of use cases we've seen across the board. You've seen use cases in almost every industry and almost every company within every industry. And then you've seen adoption of generative AI in every department of every organization. But to make that a little bit more specific, let's maybe talk about Datacamp and how have we adopted AI internally.

And then let's, pull it broader and talk about it. what the general trends are. So if you think about education and what we do, our vision always was to bring the learning experience as close as possible to what people do in their actual job and make it really user friendly, really interactive and ultimately For education, the best and most effective experience is having your personal tutor.

And I think what generative AI enables us is to get much closer to that much faster. Not just data camp, but really any education company, I would say, is building better educational experiences. So to get more specific, what does that mean as a student now? If you're practicing on data camp, if you're solving challenges will help you understand errors you make, will help you understand the solution of that challenge.

You have a personal tutor that will help you navigate through our curriculum, will help what the best content is for you to do next. And so that means our engineering and our product teams now need to have. A lot of skills around AI. And that's a general trend in the industry.

Right? Like any engineering and product team in the last 12 to 18 months has been focused on building better chatbots, better search, more personalization, new types of experiences. And we're no different, but I do think within education, the ROI of those investments is going to be really high.

but it goes broader than that. Like. I talked about what's happening in terms of our product and engineering teams. I think the most fascinating thing that we've seen in the last year and a half is adoption in different departments. So to give you a specific example from our marketing team, have people using generative AI and LLMs to analyze email data.

to help write copy. And we've had people in sales use generative AI, and that's a general trend as well. That we hear from our clients. There's in every department. There are now early adopters, and it's enabling people to do their job much better, much faster, much more effective. And that's obviously a huge opportunity for every organization, and it's a huge opportunity for data cam to help enable people.

And drive up that data fluency for everyone,

Richie Cotton: That's brilliant. I love that it's basically every department has had, has found some kind of use for this new technology. But with new technology, it means that people need new skills. And I feel there's there's a big difference between sort of general users and people who are developers.

So, Martijn, do you want to talk to me about what new skills the general users need in order to make use of generative AI?

Martijn Theuwissen: So maybe to make the comparison with again with data. I think in the past couple of years, like given everyone got access to data, people needed to start developing, data literacy skills, and they needed to understand, like, okay what kind of graph do I need to use?

How should I interpret this type of table? And I think the same thing is going to happen now with AI, meaning like, okay, everyone is going to need to develop some kind of AI literacy skills. They need to understand some of the jargon they need to understand, like, how to use some of the tools.

again, just looking at our own company, like you can already see that, like we recently did some analysis on our emails, like open late, click through rates, what kind of content really resonates with our learner base. And we use AI for that we interacted true uh, some prompt engineering, and we discovered a lot of interesting insights and people who are doing that are not necessarily like technical by nature.

Or know how to code. And so, that to me is AI literacy. And so when I think from like, what skills do people need to develop, does the general population or the general population will develop my. point is like it's going to be AI literacy, like understanding how to interact with AI, understanding like, okay, the jargon so that you can interact with an AI engineer if you need a certain feature or you want to build something.

So, think things are heading that way.

Richie Cotton: That does seem so important, just being able to have a conversation that's intelligent about AI, even if you can't do everything yourself. and so, going beyond the sort of, the skills for everyone, the AI literacy, if you're a developer, if you're working in the field, what sort of skills do you need?

Jo Cornelissen: maybe starting super high level. I think there's three areas if you want to get started as a developer. There's your large language model. There's your vector database and there's your A. I. Application development framework. So again, to stay super specific. What does that mean? What's the quickest, easiest way to get started is probably learn how to use the OpenAI API for language model, you learn how to use SpineCone, and then learn LangChain for the development framework.

And that's how a lot of people are getting started today. Now, you think about scaling that, if you think about putting that in production. What's going to happen is the costs are going to increase you'll get questions around and lock in control. And so at the same time, we're seeing the emergence of open source especially for the large language model components.

And we've seen kind of a hugging face community and ecosystem. Really take off. And so that's the second wave as you get deeper. As a developer, you probably should understand what open source models are taking off. You should become part of the hugging face ecosystem. And the advice so far was very kind of Practical.

But I think that the underlying trend is really interesting here, too. If you think about software developers overall, their job has been pretty deterministic. And software development timelines were relatively predictable tend to be two times longer than what people think, but they were fairly predictable nevertheless.

And what's happening now is a lot of what software developers are going to do in the future. is more stochastic and is less predictable at least in the next few years. And so for the first time, we're seeing developers come to data camp now realizing their skill set is getting outdated as well, and they have to kind of learn how to navigate this new worlds where they're building not just deterministic models, but they're building stochastic models and experiences.

And they might not know what the timeline look likes. for getting things in production. I think there's a lot of companies that got really excited about generative AI, but then have really struggled to hit the timelines they were hoping for about a year ago. So, yeah, I think it's really fascinating what's happening in the developer space.

Richie Cotton: Yeah, a lot of, like, very interesting trends there. fact that the data space and the software space overlapping more, that does seem like a very important trend to watch this year and,

Jo Cornelissen: Absolutely. I totally agree. I think the kind of historically with data can focus on this data analyst, data scientist and the citizen data scientists. And what's happening now is you think about the concept of an AI engineer. Some people come at that from a developer background, picking up the data and AI skills, but some people come at it from a developer background and actually probably the largest group of people are going to have a development background and they're learning kind of data and AI skills.

Totally agree.

Richie Cotton: Okay. That's definitely something to watch now. I have to say with all this kind of new technologies and new skills required, I found there are a lot of people who panicking a bit. They're saying the space is moving too fast. They don't know how to keep up. Do you have any advice for dealing with that?

Jo Cornelissen: Keep learning. Keep learning. because I think what's happening now is a very fundamental shift. Just like 40 years ago, people had to learn how to work with computers. And 20 years ago, you had to learn how to work with the internet and the software. And I think data and AI skills in the next five to 10 years are at a similar level.

Okay. Everyone is going to have to kind of upskill and upskill and reskill and keep learning. It's going to be fun.

Richie Cotton: I'm looking forward to it, having a few more people learning some skills. Now, Randy Bean came on DataFramed a few weeks ago, and he was talking about how most companies, they're still at the testing and experimentation stage with generative AI. So this alludes to what you were saying that Getting things in productions taking longer than people think.

How do you think these are going to change in 2024?

Jo Cornelissen: Yeah. So I think if you zoom out 2023 for a lot of larger companies was about experimentation. Large companies are like oil tankers and takes a while for them to change course. But once they do they're going to continue ahead. And so I think 2023 was about. experimentation 2024 is going to be about getting a lot of what's been tested and developed in production at scale.

So I, I do think we'll see a trend there. And one of the biggest bottlenecks has been well, internal skill building around a lot of this stuff and making sure things work as expected. But there's also still a cost question. If you scale up. GPT 4 in an application, it's still pretty slow and it's still pretty expensive.

So I think this year we'll see a drop in price and especially as people move towards open source I think we'll see an explosion in actual production use cases.

Richie Cotton: Yeah I haven't realized like just how expensive it is to even run these models. I mean, it's not trivial the cost once you start serving thousands or tens of thousands of users. Okay. So I'd like to move on. But so the other story of 2023 is really around text generation. So these large language models have been incredibly popular, but this sort of talk about how well multimodal AI is becoming a bit more popular.

Some of these image generation tools are getting better. Martijn, do you want to talk me through what you think is gonna happen with multimodal AI in 2024?

Martijn Theuwissen: Yeah, so, so as you said, like 2023 was really about tech generation like you. pose a question to chat GPT and you get some text back and seemed like, that's what everyone was doing uh, all the time. I think for 2024, image generation will definitely become a thing and then reach the type of maturity that text generation has today.

Again, just looking internally at data cam, like, okay, how we already started using it. You can immediately think of like different applications in the image generation. And you look at that Dolly, if you look at mid journey latest versions are really good.

And you can see production value in there. I think for the video generation, the story, my, my guess, the story is, going to be a little bit different. Like, maybe when we sit here next year it's going to be the big trend for 2025. It just. Much more difficult it's continuous.

So like the creative possibilities are more endless. There's a huge difference between creating a video of 30 seconds, a minute or half an hour or an hour and a half. So, I think that will remain still in its experimentation phase infancy uh, this year, for

Richie Cotton: Okay, absolutely agreed. And I like that we're storing up predictions for next year's episode already. That's good.

Martijn Theuwissen: I'm inviting myself again,

Richie Cotton: Excellent. Yeah. So I can certainly see how there's a big difference between creating a TikTok using Derntive Air and creating a Hollywood movie. so, before we move on from AI um, have you got any more trends Martijn, do you want to go again?

Martijn Theuwissen: maybe picking another one is like, I think the attention span of the general public it's not going to be two or three years. Like I think AI had his focus point but I actually think it's a good thing. I think what we'll see is that AI is going to get.

integrated into lots of tools, applications like our daily life and just like software or mobile phone, our laptop, like it just going to be part of our technical toolkit. I mean, don't really think like it's not that laptops are a hype today, but we probably still use them like multiple hours on the day.

And so, so my my other prediction would be that I think AI. Will start behaving more like, like laptops. So we're just going to become part of who we are, what we do how we use technology. And so it's going to leave them the collective mainstream thought, the active thought.

Richie Cotton: Okay, that does seem, like, very useful that AI is still present, but maybe just it just works, it's invisible. I am slightly disappointed, though, it's been quite nice just this year, saying, okay, I work in AI and it's cool, I'll go back to being uncool again next year. alright, uh, Jo, do you have any more AI trends?

Jo Cornelissen: Yeah. So one trend that a lot of people predicted year ago was the emergence of AI agents. I think that's one area that hasn't really materialized yet. I think if you look at the current AI agents they're not really that good yet. But are we going to see a trend towards or it's.

That's starting to work for specific use cases. I can see that happening this year where for kind of narrow areas, the AI agents get good enough to really go from kind of start to completion and start performing specific tasks for, people.

Richie Cotton: Okay. That seemed plausible. I mean, certainly like some of the demos have been like, Oh, you can set a calendar invite by using chat GPT. And it's like, well, that's okay. But it's not like a difficult problem that's being solved there.

Jo Cornelissen: Yeah.

Richie Cotton: All right. So let's move on from generative AI to talk about data. So, do you have any predictions for the world of data, Jo?

And it's a big, a big question there.

Jo Cornelissen: Yeah. And I see, I think the data predictions I have, they're not new, but there are things that. are under higher, a higher degree of pressure because of generative AI. So if you think about data quality, data culture, data availability, those were themes in the data space for like the last decade. But I think there is much more pressure on some of, in some of these areas and a much higher ROI in some of these areas as a result of generative AI.

So let's quickly talk through that. So, I think if you want to train custom AI models, data quality is really important. And so I think a lot of organizations are going to be waking up to the fact that their data quality is not at the level where they can actually train good customized models.

so I expect even more focus, more investment going into that area. The second one is data culture, right? I think a lot of organizations have realized the importance of data culture. There's been kind of an increasing level of awareness around it. Thanks to generative AI. More and more people are going to be able to do useful things with data.

And so the biggest bottleneck is going to be what's the data culture of the organization? What's the data fluency level of a lot of people, right? Because there is now a much higher ROI on building a healthy data culture. So that's, I'm very excited about that one. And then the last one is data availability links back to data quality.

Even if you have high quality data in a lot of organizations, it's very siloed. And we will continue to see huge investments and a lot of efforts to bring all of that data together. And again, I think generative AI is putting more pressure on that trend.

Richie Cotton: Okay. So this seems pretty positive that Maybe executives are starting to think a lot more about data quality, finally taking this sort of thing seriously and think about how do you get data driven decision making more widespread throughout your organization. so, let's talk about programming languages.

Martijn, are you seeing any trends here with programming languages for data?

Martijn Theuwissen: Yeah. So think I gave the same answer last year, like Biden still growing, still going very strong. They're doing really, like the language is doing really well also in the AI space. So it seems like industry is settling quite a lot on the combination Python SQL. Looking at another big language on the datacam platform R, you do see R leveling off.

I think they they, they missed the AI, uh, wave and that's definitely hurting them. If you look at the IB index, like, you also see R dropping in popularity there. And then, well, new technologies give, give rise to uh, new languages. So like we're keeping an eye on Mojo programming language mainly aimed at AI developers.

Not expecting a big breakthrough in 2024 but it has some future potential and maybe finally, like, JavaScript so, it's going to be Really interesting to see what's going to happen with JavaScript. You already see AI applications being built with JavaScript.

I think it comes back to some of the things that Jo mentioned earlier, like, okay, you have this whole developer community, software developers, like, they start to need to build AI applications, like, that overlap with the data space is becoming bigger and bigger.

They bring their own. Frameworks, their own languages with them of which JavaScript is really popular one. So, yeah, really interesting to see, like, okay what JavaScript is going to do this year.

Richie Cotton: Absolutely, and I have to say, with my background in the R community, I am sad that R is No longer the sort of one of the leaders in the data space. And yeah, it's definitely missed the boat a bit on generative AI. But yeah, it is going to be very interesting to see, will data people start learning JavaScript or not?

that's definitely one to watch this year. So moving on programming languages into sort of a general tools for working with data. Jo, have you seen the trends here?

Jo Cornelissen: Yeah, so I think the workhorse for a lot of data scientists is still the Jupyter notebook. It's still kind of the number one tool of choice for a lot of data scientists out there. In a sense that hasn't changed and that might not change. I do think there's a trend where Jupyter notebooks have key disadvantages.

And you've seen a bunch of companies step in to solve the challenges with Jupyter Notebooks data. We ourselves have a horse in the race here. We built data can workspace that makes it much easier to collaborate. It's a great playground for learners. It's a great way to do hackathons within organizations.

And I think As these notebooks in general move to the cloud and become more A. I. Enabled. That's gonna unlock a lot of potential because let's face it, like Jupiter notebooks are the workhorse for data scientists. But if you show that if you show a Jupiter notebook to the average person, their reaction is going to be, Oh, this is not for me.

Thank you very much. And I think what's changing with generative AI is that you can build a layer on top of that and make it much easier to get started. So OpenAI has their code interpreter. We are building an AI layer on top of data camp workspace as well. And we've seen people start to adopt that from all types of backgrounds to do things they didn't imagine they would be able to do.

So that's, I think, super exciting for the next few years is that kind of the total addressable market for data exploration is really going to explode. Thanks to that kind of UI simplification and the much more user friendly way. Transcribed interfaces that AI is enabling now.

Richie Cotton: Absolutely. I do find it fascinating that here at Datacamp and then at OpenAI, we've started from completely opposite places. Like, we've been doing sort of tooling for productivity for data scientists. OpenAI, I start with like, a chat tool. And then we're just converging in the same place. It's like A bit of chat plus some data analysis tools.

And yeah, both of those coming together. All right. So last year we talked about the rise of and putting machine learning models into production has become a much more popular thing back then. The tooling for that was very heavily in flux. Is it starting to stabilize?

Jo Cornelissen: MLOps is a really important curriculum area for Datacamp. And I think we've been holding back a little bit historically because the space was so fragmented. The space is still fairly fragmented. We've decided to step in and, and build out a great MLOps curriculum anyway. And there's the larger ML platforms Databricks and Snowflake.

There we have built curriculum now, so I would invite everyone to check out those courses. It's an area of upskilling for a lot of teams and organizations out there right now.

Richie Cotton: Okay, yeah certainly like, putting your data warehouse and your machine learning together in the cloud just seemed to be a huge trend. And we are seeing some sort of standard tools there. Excellent. Now, in terms of new roles that are coming up. There's been a lot of talk about prompt engineering and is prompt engineer a real job on that?

So, Martijn, do you have an opinion on this?

Martijn Theuwissen: Uh, Yes, uh, I don't think prompt engineering will become a role in itself. I think, it's going to fit more into the AI literacy story I was telling before. What I think is going to happen is that most people will need to learn how to do prompt engineering and some will become really good at it.

Some will be average at it, but like everyone will be interacting. with an AI in some capacity, so everyone will need to learn some prompt engineering tool. So I don't think it will become it's separate thing. do think there are other things like, like communicating data insights that, that will remain very important but also there is going to be a universal skill, a universal professional skill rather than like a dedicated job role would be my take.

Richie Cotton: Okay, so everyone needs to do a little bit of prompt engineering but probably it's not going to be a full time job for most people. Okay. Alright, so beyond this Jo, have you seen any new data or AI roles appearing?

Jo Cornelissen: What's fascinating right now is the space is changing really fast. In general there's the emergence of the AI engineer. I think in general, the trends I'm very excited about is the shifts in terms of the skill set for a lot of software developers. I think every software development developer is going to need varying degrees of confidence.

Kind data and AI skills in the future. And the ones that are at the intersection, maybe right in between the data scientists and a backend developer, that's a new role that's emerged. I, I would say called AI engineer. But I think ultimately every developer will be an AI engineer to varying degrees.

So I think that's a really interesting trend we're seeing right now.

Richie Cotton: Okay, so it just seemed like, the biggest new roles are really in that AI space then.

Jo Cornelissen: Yeah.

Richie Cotton: Okay. All right. So in terms of jobs, I think there was a huge sort of hiring rush during COVID particularly in the data space. And then in the last year, things have softened a bit. And the most common question I think I've had from interacting with users in our webinars has been just how do I stand out?

How do I get that data role? Marten, do you have any opinions on how people can stand out when they're trying to get a data job?

Martijn Theuwissen: Yeah, I think it's two things. It's it's credentials. And it is a portfolio. So, if you're thinking about, okay, how can you get credentials? have the vendors like Microsoft like Power BI from Microsoft but also like Tableau, Azure and so on, like that have their own certificates.

so you can go out there learn the necessary stuff to get that certificate, acquire the certificate and like you showcase and like, okay, I've done it. There are also more general certifications. Like for example, in DataCamp, you can find certifications for data scientists for data engineer, for data analyst.

So, showing that you have the necessary skills. Through this combination of credentials as well as like portfolio building, because when you're learning, you're most likely doing projects whether as part of the curriculum or not, or on your own. So if you collect that and you use that to showcase to your current employer or future employer then I think that's the way to stand out today.

And to give some credibility to your application.

Richie Cotton: Absolutely. So a lot of it's just about reducing risk for the hiring managers just to prove that you do have some skills. Okay. So I guess on the flip side of that Jo, maybe, do you want to take this? Like, if you are a hiring manager, how do you go about finding the best candidates for your data roles?

Jo Cornelissen: The good news for hiring managers, I guess, is that it's gotten slightly easier. The market was really hot two years ago. I think it's softened a little bit, so they will get candidates. And I think what's important to understand for both the hiring manager as well as for people who are trying to break into this industry or trying to reinvent themselves in their career is that there's just not enough people that have the traditional credentials in this space.

Right. There's been, an emergence of data science masters in the traditional educational system in the last five to 10 years. But a lot of what's taught in most of these programs is already outdated. So I think the main thing to realize for both sides is like, you will need to continue to learn.

You will need to continue to update your portfolio, update the credentials you're getting to be successful in this market. And that's true for both sides. It also, means that hiring managers have to stay up to date and understand kind of what credentials are out there. And exactly kind of which ones can you trust?

Richie Cotton: This is very much a field where you can't just say, okay, I've got a degree and I'm done. You do need to continue learning throughout your career.

Jo Cornelissen: Yeah, exactly. And I think the speed at which the kind of the relevance of skills is the king has increased in this space. I think that was already true for a lot of data and software development roles in the last few decades, but I think the speed at which you have to learn it has only increased in the last few years.

Richie Cotton: Absolutely. I have to say we're doing the DataFrame podcast and then we're doing DataCamp's webinars every week. I'm finding some instructor comes on or some guest and they're always showing something interesting that I know nothing about. Like, oh, I could spend years learning about this. So, yeah, always something interesting to learn.

All right. So, before we wrap up can you tell me what you're most excited about in 2024? So, yeah, final thoughts, Martijn?

Martijn Theuwissen: So, so 2023 was really about experimentation with all kinds of like the companies were trying to figure out like, okay, how am I going to use AI in the services, the products that I offer? And I think 2024 will really be about bringing those. experimentations, those thought experiments into production and making them available to us consumers.

So, yeah, really interesting to see, like, okay, if I open Netflix six months from now, like, okay how will they have incorporated AI? If I open Spotify, like, how will they have, like, am I going to be making my own music? six months from now. So what, what really excites me is, is, seeing like, okay how, how this whole, well, hype of 2023 will materialize in 2024.

I think that's like, yeah, that's what I'm looking forward to.

Richie Cotton: yeah, Yeah, things actually uh, hitting, approaching, I do especially like that idea of being able to make your own music. I have a sort of longstanding bucket list item to become a rock star. I'm horribly tone deaf and not musical at all. I've made no progress towards this. So it'd be nice if AI can take that on for me.

Jo uh, what are you most excited about?

Jo Cornelissen: Yeah, I had a similar one from a slightly different angle. I think I'm excited about what individuals and startups are going to be building. And data camp is ultimately in the business of enabling people, teams, organizations. And what I'm excited about at a high level is generative AI is enabling two new groups of people to really build.

Totally new experiences and get to insights in new way. So you have the citizen data scientists, the someone who doesn't have a technical background in any business role. They're going to be able to get to insights much easier, much faster, thanks to a lot of new tools that are emerging.

So what are they going to come up with? And then second of all, You have the software developers who were always building and shipping products. But they will be able to build experiences that we couldn't imagine years ago. So yeah, a bit similar to what Martijn was saying. Kind of like Seeing that go in production is going to be really exciting.

Richie Cotton: Absolutely, just bringing data and AI to the masses. It's exciting times. Excellent. All right. Thank you both for your time. Uh, Yeah. Thank you, Martijn and Jo for coming on the show.

Martijn Theuwissen: Perfect. Thanks for having us. See you next year.

Jo Cornelissen: See you next year, Richie. Thanks so much.