Episode 190: Understanding the Build vs Buy Dilemma in HR Tech (Interview with Craig Starbuck)
As the world of HR and people analytics evolves, how can we best leverage the tools, insights, and technologies available to us? In this thought-provoking episode of the Digital HR Leaders Podcast, David Green sits down with Craig Starbuck, Head of People Analytics at Roku, to explore the critical decision every HR leader faces: should you build your analytics capabilities in-house or buy them from external providers?
Leaning in on his ten year background building and leading people analytics functions to excellence, Craig discusses the benefits and challenges of each option, guiding listeners through the strategic considerations vital for making informed decisions.
The discussion also delves into:
The importance of aligning people analytics with finance to drive substantial improvements in business outcomes.
Framing analytics insights in the language of the business to effectively support strategic decision-making.
How Craig is integrating passive and active data at Roku across a multitude of use cases.
Insights from Craig's book, "The Fundamentals of People Analytics: With Applications in R" on upskilling HR teams in advanced analytics tools and methodologies
This episode is proudly supported by Worklytics, a people centric analytics solution that combines passive listening with Organisational Network Analysis (ONA) to help you understand how work is getting done.
Curious to see how it works? Worklytics is offering a free Collaboration Analysis to the first 10 qualified companies who express interest by clicking on the following link: www.worklytics.co/DigitalHRLeaders
[0:00:00] David Green: People analytics moves us beyond gut feelings, offering evidence-based insights that not only guide strategic decisions and enhance operational effectiveness, but also drive real business outcomes. This is why Insight222 research shows that 88% of CHROs consider people analytics a core component of their HR strategy. Our research at Insight222 also finds that one of the eight characteristics of leading companies in people analytics is that they undertake advanced analytics on the most important business priorities. The role of the people analytics leader in driving this agenda, building the team and influencing senior leaders in the organisation is paramount to the impact of people analytics.
As such, I'm delighted to be joined in today's episode by someone who has led and built multiple people analytics functions, Craig Starbuck, Head of People Analytics at Roku. Craig has spent over a decade in the people analytics field, at companies such as Equifax, Mastercard, Robinhood and now Roku. He has worked in established people analytics functions in large global companies, led people analytics in a scale up to an IPO, and at Roku is the founding people analytics leader. As such, in our conversation, Craig will unpack the critical build-versus-buy decision for people analytics technology. We'll also discuss the synergies between people analytics and finance, the importance of aligning analytics with business needs, and the value of passive and collaboration data for a range of different use cases. With that, let's dive into the conversation with Craig Starbuck.
Craig, welcome to the show. Could you kick things off by giving our listeners a brief introduction about yourself and your journey in people analytics?
[0:02:09] Craig Starbuck: Absolutely. Good to be with you, David. Thanks for having me on the podcast. Yeah, I've been in people analytics now for about ten years, which is hard to believe. I currently lead people analytics at Roku, where I've been for about two years. And funnily enough, Roku means six in Japanese, because it's our founder and CEO's sixth startup, and this is my sixth people analytics function that I've had the opportunity to lead. So I think in many ways, maybe the stars aligned for me to come to Roku and be a part of this organisation. But I do love people analytics. It is my calling and I feel a great sense of purpose in what I do. And hopefully for the remainder of my career, I'll get to be in people analytics in some form or fashion. So, looking forward to the discussion today.
[0:02:57] David Green: Well, great. I mean in terms of your journey on people analytics, where has it taken you over your ten years?
[0:03:04] Craig Starbuck: Yeah, great question. So, my first people analytics role was actually in an investment firm. It was a small, privately-held firm of about 4,000 people, and I was there for several years. That's where I learned all the different HR disciplines. It was my first foray into the space, and so it was a great opportunity to learn about talent acquisition and what HR business partners care about and HR compliance, and all the different, great work that's happening in talent management, in terms of talent development, performance management paradigms. And it was there that I got to lead, actually, my first employee listening programme. And employee listening is something that I'm super-passionate about. I think it generates a lot of great data. Perhaps we'll talk more about listening later on in the podcast. But that was my first opportunity.
From there, I went to TD Ameritrade, and then on to Equifax. And Equifax, I was in more of a product-facing role. We were building people analytics solutions that we were monetising and selling. And so, in that capacity, I was more of a consultant, travelling around, consulting with different heads of HR, showing what we were seeing in the data with respect to attrition benchmarking, compensation benchmarking, and some broader labour market insights. Then I moved back into an internal people analytics role at Mastercard. And so at Mastercard, we had a pretty significant investment in people analytics, which was great. I had about 20, 25 people, including an engineering team that was dedicated to supporting our internal people data infrastructure build. And then from there, I moved from this large behemoth of Mastercard into a smaller organisation, Robinhood, which was just an incredible experience, just some brilliant colleagues. We were doing some great work there. It was a hyper-growth startup. I joined just before IPO and got to experience the IPO journey.
Ultimately, I made the decision to come to Roku. They had a new opportunity to build a people analytics function, and building is what I enjoy doing. I've been in a build role at every organisation in which I've had the opportunity to lead people analytics. And so, yeah, really enjoying my time at Roku, like I said, coming up on two years now, and looking forward to sharing more about that.
[0:05:33] David Green: Given you've been in people analytics for a decade, I'd love to hear your thoughts on how people analytics has evolved as a discipline and how it's impacting organisations today.
[0:05:45] Craig Starbuck: Yeah, I mean, it's evolved quite a bit, right? I was unfortunately not at People Analytics World, but I can imagine the sentiment was probably the same. And as I think about how it's evolved, I would actually credit vendors a lot for that evolution, or at least the speed and acceleration of that evolution. When I think back ten years ago, I think most organisations were probably building their people data platforms internally, and there weren't a whole lot of vendors in the space that were doing that, at least doing it well. And so, fast-forward ten years to the present day, there's quite a lot of vendors in this space that can help accelerate our journey in terms of data integration, data management, the production of common dashboards that every organisation needs, things like headcount, career moves, attrition, diversity, org design, kind of the staple measures, and I think vendors have played a significant role in helping us do that.
[0:06:47] David Green: I'd agree with you on that. I think a lot of the innovation in the space has come from the vendors, and you're right. When we look back ten years ago, there weren't that many vendors around specifically focusing on people analytics. There were others starting to come into the space, I guess, from other parts of HR tech. And I guess a great indication of the growth in people analytics technology is, it's now almost a category on its own. And certainly if we look at the research that Stacia Garr and Priyanka Mehrotra do at RedThread Research on the people analytics technology market, it's exploded over the last few years. So, really interesting thoughts there, great. I've got two things around the last ten years to follow up on you. One is related to you and working in different types of organisations, I think, in the people analytics role and around the difference; and the second one, I'm going to come back then to what you've actually built at OrgAcuity as well, as kind of a side gig. I'd love to learn a little bit more about that.
So, let's start, you mentioned you worked at a fairly mature people analytics function at Mastercard, where you had a lot of the engineering as part of the team and quite a large team as well, 25 people, and I think that's continued to grow since your time there. And then obviously, you said then you went to Robinhood, it was a scale-up. Love to understand the differences between big enterprise, like Mastercard, scale-up, like Robinhood, and now you're a founding people analytics leader at Roku as well. Love to hear, because we've got people analytics professionals and HR professionals listening to this who are working at large companies, working at scale-up companies, where they might be the only person in people analytics. Love to hear some of your thoughts around some of the differences maybe between the three, but maybe also again, based on your experience, some of the key focus areas and how those differentiate perhaps between those different types of organisation.
[0:08:49] Craig Starbuck: I would say Mastercard and Robinhood, if I were just to juxtapose those two experiences against one another, they were quite different. And the first thing I would say is that at Mastercard, we chose to build our internal people data platform. And so all of that investment, what was internal, we didn't buy the people data platform; whereas at Robinhood, we actually purchased a vendor's platform. And Mastercard was a painful experience in many ways, because building is hard, number one; it's very costly, number two; and number three, it takes a great deal of time to architect a data model that's going to match the source system. And so what we found, I mean we had some great talent on this project. And what we found as time went on, there were just some cracks in the foundation that resulted in the source system data not matching what was coming out of our internal data platform. And as we dissected that, we learned there were retroactive changes, backdated changes, there were corrected events that were happening in the system, and our data model wasn't necessarily reflecting that. And so, it was through a series of kind of data reconciliation exercises, unfortunately reported by stakeholders that didn't see that the data aligning between the source systems and the data platform, that we realised we need to refactor this data model.
At Robinhood, it was a much better experience with respect to the data platform, because we were working with a vendor who had figured this out, right? And they had many repetitions on this, because they had received feedback from so many customers that by the time that we went live, it worked, right? It was a really good, easy and favourable experience. And so, that the speed to insight, there's no comparison. And I, at this point in my career, having built internal data platforms two different times at two different organisations, and having purchased off-the-shelf products from vendor partners, I have a very strong opinion on this. And if I were to advise someone building a new people analytics capability, I would advise them first and foremost to buy their people data infrastructure. Don't build, just save yourself the headaches, save yourself the time, save yourself the cost, just buy a solution, because that will enable you, like I said, these basic dashboards, like headcount, your basic HR measures, you can check that box, you can tick that box and then you can move on to more strategic initiatives, building those strategic relationships and cross-functional partnerships that you need to position people analytics as a strategic decision support function.
I think as long as we're spending our capacity and our resources optimising data models for our internal data infrastructure and continuing to field basic queries like, "How many active workers do I have; and, what's my attrition rate?" that's really unfortunate because people analytics can be so much more, and I just really don't see the ROI in building internal data infrastructure. I'm very much a proponent and an advocate of buying the people data platform.
[0:12:21] David Green: So, moving on, Craig, let's talk a bit about the work that you're doing at Roku, and then we'll get on to the partnership that you've got with Worklytics. So, firstly, with Roku, love you to give an overview on people analytics at Roku, what are your focus areas, how's the team organised? Obviously, you're building the function from scratch. Love to hear some insights about that.
[0:13:54] Craig Starbuck: Yeah, absolutely. So, Roku, my role now is actually working into more of an enterprise data and analytics role, which the primary focus is still people analytics. But now we're more and more starting to partner with other functions, like finance, which I think is a real benefit. I think finance is very tangential to the work we do, whether it's workforce planning, or it's shared metrics around headcount, or attrition assumptions for financial forecasts, or looking at capacity utilisation, location planning, talent planning, and how that feeds into the next workforce planning cycle. And so, really excited about that partnership. I love to collaborate with the finance organisation, and we have some very data-driven leaders in that part of the organisation. So, I've been looking for ways of, how can we connect the dots with what we're doing in the people analytics side, to what the CFO cares about, for example? And like I said, I think over the last ten years, I've really seen people analytics take centre stage in terms of being about the business.
One of the first things I always try to do, when starting a new role, is learn the business first and foremost. And so, that could be understanding our strategic priorities from a business perspective, if it's a public company, reading through our financial statements, meeting with stakeholders to learn what they do, how are we organised; why are we organised in that way; how do we make money; what are the different lines of business, the different business units that we operate? And then, think creatively about how can we tie into that; how can we enable those functions? Because while we may be a GNA function, and we're not generating revenue directly, indirectly we are supporting the folks who do, and we can surface insights that help them be more effective and more successful. And so, I think it's really important to build those strategic and cross-functional collaborations with stakeholders, because if we're sitting back and just taking orders and reacting to what people request, I think we miss the large majority of the opportunity to be strategic and to be more proactive.
So, what I've made a concerted effort to do is really focus on learning more about the business, especially at Roku. I think each passing people analytics role I take, I think gets a little bit better in some respects. I think that experience, that diversity of experience that I can draw upon, and the lessons that I learned along the way, albeit sometimes painful, I think helps in the next role. And I think I've brought all of that cumulative experience to bear at Roku in thinking about, how can we help make data-driven decisions about talent? We have made the decision, or I should say, we did make the decision to build our internal data infrastructure upon my joining. And so I've had an in of two in terms of build experiences, but we have fantastic data engineering partners. And I think if it weren't for that, we would not have been successful in building it. So, it is possible to build data platforms and have them be successful. But again, if I were to do it over again, and I were to take a new role, or I were advising someone building a new people analytics team, I wouldn't have to give a second thought to, just buy the people data platform and save yourself a lot of money, a lot of time, and a lot of headaches. I think the speed to insight is going to be that they're not even in the same ballpark; it's just much more favourable.
[0:17:38] David Green: Can you share a little bit more about maybe some of the use cases that you're planning using Worklytics, the passive data that can potentially enhance your people analytics work at Roku?
[0:17:50] Craig Starbuck: Yeah, absolutely. I mean, as I think about the vendor landscape, I mean what I've been talking about in way of a people data platform, I would say, are multi-source providers, right? They're taking data from Workday, or whatever HRIS the company is using, ATS, Learning Management System, and they're bringing that together and creating metrics and dashboards, and that's great, and that's a really important aspect of what we do in People Analytics. I think what Worklytics does is quite different, yet complimentary, in that Worklytics has kind of surveyed what are all the different platforms or applications in which people are conducting their work? And so, we've got email, we have calendar information, we have instant messaging, we might be corresponding, if you're a software engineer in GitLab or GitHub, there's content management systems like Confluence, there's ticketing systems like Jira. And each of these applications independently is capturing data about what we do. If you're using like MS 365, Microsoft is surfacing Viva reports on how you spend your day and what percent of your time is focus time versus in meetings. And that's great, but that's a very narrow picture as it relates to what we do day in and day out.
For example, when I was at Robin Hood, almost all of the correspondence between colleagues took place via Slack. That was a Google shot, but the only time I checked my email inbox was if I was expecting some correspondence from an external party who would not be Slacking me. No one internally ever sent any emails; 100% of it was Slack. And so, if we were to rely on Google alone to give me a picture of who I'm talking to and how often and the nature of our collaboration, it would look like I'm talking to no one, because my correspondence was in Slack. And so, Worklytics has acknowledged this, and so basically what they do is they take all of this data, albeit pseudonymised, because it's a very privacy-first mindset, which I really appreciate, pseudonymised data and only metadata, not the content, across all of these different platforms, bring it together to create a holistic picture of what people do, what are the patterns with respect to collaboration and work output and how we conduct ourselves in on-site settings versus remote settings. And that enables us to create archetypes of, what does an effective leader look like versus a not so effective leader? Or what does an effective employee, a highly engaged employee look like versus a highly disengaged employee?
So, I think that comes to the beauty of integrating the passive digital exhaust that we're collecting from Worklytics, or that we're planning to collect, and integrating it with the active signals that we capture from the likes of employee experience surveys. Because I think both independently provide significant value, but I think if you can bring together the active and the passive, there's so much more information and insight that we can glean. And so, we're thinking about things like, we have an increasingly geographically distributed workforce. And so, Roku more and more is expanding internationally, we're expanding in the UK, and places like Manchester and Cambridge and London and Cardiff, we have a number of sites there. We're expanding in India, we're expanding in other geographies, and that's creating some interesting questions around location strategy and, when does it make sense to hire a team that is located 10, 12 hours apart from the central team or perhaps the manager of that team versus when do we take a different strategy? And so, there's a lot of questions around, how do managers operate in this more geographically distributed environment; and what sort of tools and data-informed insights can we glean and share with them to make them more effective leading these more distributed teams? So, I think that's certainly top of mind for us.
There's a lot of things around the networks, and so your collaboration, your circle of influencers or your circle of close collaborators. Does that look different between folks who are fully remote versus folks who are in this hybrid model? How does it change between maybe a new hire's first 30 days onto their first 90 days, first six months; do we see them becoming more and more integrated into the board network? And so, what implications does that have for our onboarding programme; and how can we use that insight to ensure that people are receiving the support that they need to receive, and that they have a great sense of belonging and they're just as effective and efficient as they can be? And so there's a lot of questions that we're asking that really go to our efficiency and effectiveness and collaboration, and how a more geographically distributed environment is impacting that.
[0:23:31] David Green: I mean, bringing all that together, we're going to go into the skills of HR professionals. There's something you also said earlier, which is quite interesting, how your focus is on people analytics, but are increasingly looking to a more enterprise kind of role at Roku, and you mentioned the relationship with finance. So, it's quite interesting. When I was at People Analytics World recently, I shared some of the research that we've been doing at Insight222, which is trying to understand what are the practices of leading companies in people analytics, because it's a question we get asked a lot. There's eight characteristics, we won't go into all of those now, but one of them is around measurement. It's said that leading companies actually measure the impact of what they do, not every piece of people analytics work obviously, but certainly some, and that helps to make decisions around what products to scale potentially and where to invest more moving forward. And actually, what we identify with companies that do that was the relationship they have with finance.
So, if you're measuring the outcome of what you do, whether that's pure financial terms, whether that's looking at things like workforce experience, maybe societal benefit, whatever, there's a very close relationship with finance. And I think you've probably seen this throughout your ten years within the field. That is a really important relationship, isn't it, for people analytics leaders to have?
[0:24:49] Craig Starbuck: Yeah, I'll actually share just a quick story, David, to that. So, my first role in people analytics, as I mentioned, I led the employee listening programme. And the employee listening programme was actually sitting with HR compliance. And effectively, we had a vendor that was just giving us an Excel dump of favourability scores, and I thought, "We can do better than that, we have to do better than that". And so, I was pretty vocal about that, and the CHRO said, "Craig, now you're on employ listening, make it better". So, great! So, went out, we found a new vendor, and we launched quarterly poll surveys. And as part of that, the CHRO asked me to come speak to the C-team once a quarter on the results of the survey. And my first one, I'll never forget, it was a pretty enlightening moment. There were a couple of segments leading up to my segment, and they were talking about kind of the financials; it was a brokerage firm, they were talking about assets under management, some of the brokerage metrics that they cared about. And I was just looking around the table, this was back in the days when I was physically in the room with all of these folks, very different from today, you could tell by the non-verbals, this was very important stuff. Everyone was super-engaged, they were actively, vigorously taking notes. This was important content that was being discussed.
Then, we got to my section, and the CHRO kind of teed up my segment by talking about, "We want something to be effective, we want to increase employee engagement from 68% to 70%". And there was nothing about the outcome, the business outcome of doing that. It was just kind of this HR isolated goal. And as I was panning around the room, I noticed a distinct change in the tenor. People were checking their phones and they weren't as engaged, and I could tell they didn't take this as seriously as the brokerage analytics that we were just discussing; it was very clear to me. So, I gave my presentation, it went fine, but I went back to my desk and I started to reflect on this. And I thought, "How can I connect the work we're doing to business outcomes to financial measures?" Because if I cannot speak in the language of money, the likes of the CFO won't care what I'm presenting each quarter, and I want them to care because I believe in this stuff, and I know that decades and decades of organisational literature corroborates the influence that a highly engaged workforce has on business outcomes and financial performance.
So, since I was leading people analytics, I went into Workday and I looked up everyone with the word "Analytics" in their job title. And I found the Business Intelligence Leader and the Brokerage Analytics Leader and FPNA Leader. And I reached out to them and I said, "Hey, I'd love to form this collaborative, I'd love to get us all together and talking, and I'd love to learn what you're doing, love to share what we're doing and see if we can be a resource for one another". And my hidden motive was, I'd like to get some access to financial data that I could use to correlate with engagement data because I don't have it sitting in HR, but surely someone has it. And so, we ended up getting sales data, individual-level sales data, we had some quota attainment metrics, commission data. And the next time we conducted that survey, we loaded that in to the system and we started to do some driver impact analysis with respect to business outcomes, not just engagement outcomes, but business outcomes. And so, the next time I went before the C-team and I presented the results, instead of just talking about, "We went from X% engagement to Y% engagement", I was able to communicate that for every 1% increase in engagement, the average change in business outcome ABC is Y.
It was a very different reception to that message versus, "We're just increasing engagement", and there wasn't really the appreciation for what that meant previously.
[0:29:08] David Green: We hope you're enjoying this episode of the Digital HR Leaders podcast. If you are looking to continue your learning journey, head over to myHRfuture.com and take a look at the myHRfuture Academy. It is a learning experience platform supporting HR professionals to become more data-driven, more business-focused, and more experience-led. By taking our short assessment, you will see how you stack up against the HR skills of the future. Then, our recommended learning journeys guide you every step of the way, helping you to close your skills gap, deepen your knowledge, and press play on your career.
What skills do HR professionals need to use some of the advanced analytics tools that we've been speaking about?
[0:30:06] Craig Starbuck: I love the question, David. I've been thinking about this for a long time, actually. And in my career in people analytics, I've observed some gaps with respect to the technical skills, and it's actually one of the reasons I decided to write the book, The Fundamentals of People Analytics With Applications in R, because I wasn't aware of any comprehensive treatment from research designs, research methods, onto descriptive and inferential statistical methodologies, onto data visualisation and storytelling, kind of that full end-to-end. And I think there could be a misconception, unfortunately, with AI that we can relegate the work to a carefully engineered prompt, and I actually don't subscribe to that notion. I think as we talk about the skills for HR, number one, HR needs to be data-driven. I really do believe that data is the new currency for an HR professional. And so, I think it's like table stakes, like everyone in an HR role, whether you're a HRVP or you're a talent management leader or talent acquisition leader, whatever it might be, people need to be comfortable working with data and using data to tell a story.
I actually had a professor in grad school, I'll never forget this, it was one of my first days in class, who walked in and he said, "No one cares about your uninformed opinions". And then he paused, like this big dramatic pause. And I remember thinking to myself, "The hubris of this guy. What an arrogant thing to say!" But I later understood what he meant by it. He meant that you can come into a room with people you are seeking to influence, and you can have untested theories and untested assumptions and opinions about how you think things should work or better ways of doing things, but unless you have an evidence-based approach, something that's been tested and validated empirically, it's easy to discount it as an uninformed assumption or opinion. And so, I think enter into the equation people analytics, that's precisely what we are working to do, is arm stakeholders with the data and the insight they need to make data-based recommendations, so it's not just an uninformed assumption or an opinion, but it's something that we can validate and not just validate through research that may have been done on MBA students, that may or may not generalise, but in our specific organisation, our specific setting, here's what the data say, here's what the data show. And I think that is really the key benefit. But it requires a level of comfort.
I mean, going back to your question of what skills does HR need, it requires a level of comfort engaging with data, and people are in different places with respect to their data journey. And so, my goal in writing the book, The Fundamentals of People Analytics, was to make this accessible. That book can be purchased on Amazon or wherever you get your books, but I've also made it freely available online. So, anyone in the world with the initiative to learn, you can download the PDF freely available. Again, I'm not trying to make any money off of this book. I'm actually investing any royalties I get from the print book into OrgAcuity to make people analytics accessible to more organisations. But the book is really intended to help make people analytics accessible and to give people tools, new tools in their toolbox to make effective use of people data.
[0:33:51] David Green: We also found that actually, making the people analytics leader responsible meant that HR professionals were two and a half times more likely to invest the time in learning. Now, obviously that doesn't mean the people analytics leader has to do it on their own, they've got learning support around that as well. That was quite an interesting finding. But ultimately, this is about culture and change management, isn't it? It's not just, go off and do a couple of online courses and suddenly you'll be fantastic. You've got to build that confidence over time. And I don't know, obviously, if we think back ten years ago and we think where HR professionals are now about being data-driven, there clearly has been a lot of progress, but there's probably still some way to go. I'd love to hear your thoughts around that.
[0:34:43] Craig Starbuck: Yeah, I mean I think the key word there is "learning". My best advice to anyone in the workforce or anyone about to enter the workforce is to be a lifelong learner. A habit I developed many years ago, which I keep to this day, is at 5.00am every day, I learn. I do something apart from my day job, usually something technical in nature, to upskill and to keep current with trends. I love technical work. I have fewer and fewer opportunities to do it in my day job. Now it's just kind of a function of moving up in the organisation, and I don't think that's where I can best serve the team. It's building those strategic partnerships and representing the work my team does well, but I do enjoy the technical work. And so, I really think lifelong learning is the key.
I mean, we have the good fortune of having so many resources at our fingertips today, right? I mean, there's so many free resources. You don't have to necessarily take a traditional university route to gain those skills. There are so many bootcamps and trainings and free resources and guides. And for me, if I'm commuting in my car, I'm listening to an audio book or a podcast. If I'm in the gym, I'm listening to an audio book or a podcast. My wife and I, when we're cleaning up the kitchen after dinner, we have our AirPods and we're listening to a podcast or an audio book. I mean, there's so many ways to multitask and get the value of the information and the knowledge. And I just think it's critical, regardless of one's field, to always look for ways to keep current and to make sure that we're providing value and gaining new skills, because I think the pace of change is so significant.
I saw an article, or a report that was published by Goldman a while back, and I think they were estimating that around 300 million jobs will be impacted by GenAI. And if memory serves, that's about two thirds of all jobs. And they also, in that report, published that about one quarter of all current jobs will be altogether replaced by AI. And so, I think that could be a very gloom-and-doom headline. But the good news is, if you look across history, whether it's the Industrial Revolution or the Information Age, history has shown that the worker displacement from automation creates new jobs in the emergence of new occupations. And so, there's a lot of jobs that'll be coming up that we don't yet know. And I think it's exciting. I mean, AI is not only coming, it's here. And so, my advice would be to embrace it; how do we augment our capabilities? I think for people analytics, it's more about the augmentation of what we can do, becoming more efficient, becoming more effective, and less about the replacement.
I don't see GenAI as replacing people analytics. It will certainly change aspects of what we do. I mean, for example, I've used it to frame up an analysis. I'll write out an outline or a framework, in terms of how I want to structure an analysis, and I'll ask GenAI to write some R code for me. And it does pretty good. I actually did it a couple of weeks ago, and I was reading through it, and I found a new R package that I'd never heard about. So, I started researching it and learned that R package, not only does it not exist, it never has existed. And so, it's not that the large language model was trained on dated information in a package that used to exist has since been deprecated; it never existed. And so, it does hallucinate, we're not ready for ChatGPT to produce production-ready code that we can just blindly put into production.
I think the nature of our field, and admittedly maybe I'm a little biased here as a people analytics practitioner, but the nature of our field is very, very complex, right? Understanding people in social and organisational settings is very nuanced and very complex. And so, it's more than just technical work, it's applying decades of org theory to anecdotes that we're hearing from the business, or hypotheses or problem statements, and framing that analysis in a way that's going to deliver useful insights that we can put some stock in.
[0:39:29] David Green: So, if we look forward ten years' time, how do you see people analytics evolving over the next decade? And don't worry, I won't come back to you in ten years' time and mark your card on it, but how do you see the field evolving in the next ten years?
[0:39:43] Craig Starbuck: That's a really good question. Number one, there's no vendor or model that's going to give us answers to 100% of questions. Over the last ten years, I think the nature of questions that the teams I've led have tackled have become more and more complex and nuanced. And my guess is that over the next ten years, while I don't know what it's going to look like, it's going to become more and more complex. And so, we really need more sophisticated methodologies and approaches to addressing those questions effectively. And so, I think AI will certainly help to make people analytics more accessible. I mean, I already see some evidence that vendors are doing this, whether it's through a Slack integration, for example of, "Let me ask a question using an English prompt of, what is my attrition rate over the past 12 months for a certain subset of the organisation, certain functional org?" for example. You can get a quick answer, right, but those aren't really strategic questions, that's a basic HR measure.
So, as the nature of our work becomes increasingly complex, I think our systems and our technology will need to evolve commensurately. And so, as I think about something like, "Is training effective? Maybe it's a leadership development course or maybe it's some other intervention". The way we would typically frame that up is through some sort of A/B test, where you take a random sample so that you can control for all of the alternative influences on the outcome that we're trying to influence, and you partition the groups into a treatment and control group. And the treatment group gets the intervention, they get the training or whatever it might be, and the control group does not. So, business as usual for them. And at the end of the intervention, we would test, is there a significant difference between the treatment and control group? And this is the nature of experimentation, and I see us doing more and more of that in the future and less of the building dashboards. I think ten years ago, that was an important skill, but again, with the vendor landscape what it is today, I think largely we can automate away a lot of the dashboarding tasks, if not all of the dashboarding tasks.
So, it becomes more about, how do we take these increasingly complex challenges facing people and organisations and understand them, and present the results in a way that is actionable and can influence what it is we're trying to impact? And I think the art form in that is really critically important, because the way we present it to an HR business partner may be very different than how we would present it to someone in finance, for example. And I listened to your recent podcast with Cole, Storytelling with Data, just excellent insights that she shared about how to present compelling stories with data. It's as much an art as it is a science, or maybe even more so, and I think that is a critical skill that people analytics teams need to develop to meet these increasingly complex challenges that we will surely face over the next ten years.
[0:43:13] David Green: Really good. Well, Craig, thanks to calling out Cole's episode. It's always instructful listening to Cole. So, this is a question of the series. You now get to answer the same question that Cole actually answered as well, as well as Nickle Lamoreaux and Catherine Coppinger and Loren Shuster as well in this series. How can HR leaders harness the power of employee insights and analytics to revolutionise the workplace experience? And you'll probably maybe refer to some of the stuff that we've already talked about, I guess.
[0:43:48] Craig Starbuck: Yeah, I mean one thing that keeps me up at night, David, is unknown unknowns. So, there's some known unknowns and some unknown unknowns, meaning there's some things that I know I don't know the answer to. Maybe it's the characteristics of effective leaders. We need to research that, we need to glean that. But there's a lot of things I don't know that I don't know, and that's scary, right, because there's a lot of missed opportunity there. And so, I think the vast data that we have access to, there's a lot of subtle barriers to success that people analytics can help surface that can influence things that we don't know are problems. And so, I very much think like a researcher in terms of how I frame up a problem that is brought to me or that I observe, in terms of we begin with a good theory, and we frame it as a hypothesis, which we can test and bring the right methods to bear, and then we present it back to the stakeholder in a way that resonates. But what about all those things that we're not asking, those things that I don't know to ask, those unknown unknowns? And so, I think there's a lot of signals that we can capture in the data, whether it's through passive digital exhaust or through active signals or through the vast data that we have across other applications across the business, that can surface and highlight these unknown unknowns.
[0:45:18] David Green: How can people stay in touch with you? How can they find out more about OrgAcuity? How can they find out more about the book as well?
[0:45:25] Craig Starbuck: Yeah, well first of all, thanks so much, David. Really appreciate the dialogue today and thank you for having me on the podcast. I've really enjoyed the time together. LinkedIn's a great way to stay in touch with me. Not as active on there as I would like to be. Life is pretty busy with some of these initiatives, but LinkedIn is a great opportunity. The book is linked. Perhaps we can put it in the show notes, but you can find a link to the book either through orgacuity.com or through my LinkedIn. And then, if you're going to be at the TALREOS conference in Chicago next month, I am teaching an R Markdown workshop there. And so, looking forward to connecting with many in our community.
[0:46:07] David Green: Craig, thanks so much for joining us. I know that listeners are going to learn a lot from this conversation, so thank you very much.
[0:46:14] Craig Starbuck: Thank you, David.