How Roblox built a hiring test that feels like a video game

Ritti Bhogal had never seen an internship test like this before. “It was like a game,” says the second-year NYU student of her recent experience taking an online Roblox exam. She had guessed it would involve coding and take place on the Roblox platform, which attracts around 58 million daily active users. “I felt myself using the same kind of skills I use every day while coding, which is kind of bizarre because it’s not in the context you’d expect it.” In other words, she hadn’t anticipated being asked to predict how a disease might spread across a deer population.

Tech companies are known for their novel—and sometimes brutal—approach to job applications. Why are manhole covers round? How would you solve problems if you were from Mars? Google gets a lot of credit (and blame) for the modern incarnation of seeking out these obtuse displays of intellect, which has spread through the industry and into corporate America like, well, a disease transmitted through a deer herd. At its best, the hiring process at tech companies discovers unorthodox thinkers. At its worst, it unnecessarily stumps candidates and is ultimately arbitrary. For all the show at identifying a candidate’s “Googleyness” or whatever, the recruiting process has largely remained ripe with bias. Hiring managers favor “elite” schools, judge applicants based on their names on resumes, and respond more often to people who look or act like them.

Although a tech recession is underway and very few tech companies are openly hiring, Roblox, which generated $517.7 million in revenue in the third quarter of 2022, is an exception. With more talent on the market in years, as many companies have slimmed their workforces through layoffs, Roblox’s commitment to revamping its recruitment process comes at an opportune time. The gaming platform has been investing in assessments that would combat bias and inefficiency in its application process, weeding out subjectivity as much as possible for the company’s entry-level jobs, which it believes is critical to help it improve the diversity of its workforce.

“It’s got to be in that funny gray area between the test and the game,” says Jack Buckley, Roblox’s VP of people sciences.

“Even after taking [the test], I’m not 100% sure what they were testing me on,” says Bhogal. “So how do I prepare for that next time? I’m not really sure.”

Computer science majors know what to expect from a coding test, but they don’t know how to prepare for an assessment extracting the rhythms of their brains. For Roblox’s leaders, that’s kind of the point: They want to find employees outside of the typical comp-sci mold.

That doesn’t make the experience any less stressful. Nor is Roblox’s new testing regimen guaranteed to do everything it’s supposed to do. But, says Roblox CTO Dan Sturman, “I think this could be the future.”

The ‘best test’ or the ‘worst game’?

Roblox’s path to exploring the science and art of job assessments started when CEO David Baszucki approached a startup named Imbellus about taking on his company as a client. Imbellus built game-based assessments for clients such as McKinsey, and Baszucki wanted the company to develop a test that quantified the exact types of thinking necessary for success at Roblox. As it happens, Imbellus was in the middle of looking for a buyer, so Roblox bought it instead, formally acquiring it in December 2020.

“We needed something based on the way Roblox works that would test the intrinsic sorts of things that people bring to the table,” says Roblox CTO Dan Sturman. With Imbellus now in house, Buckley, who had been president and chief scientist at Imbellus, worked with industrial organizational psychologists to do a job analysis of entry-level software engineers and product managers. “You need to go study exactly what skills are necessary for these people to be successful at Roblox,” he says, devoting hours to surveys and employee interviews and observation.

Buckley’s team developed a list of around 15 critical skills. They decided the cognitive test could best measure “creative problem solving” and “systems thinking.” Buckley relayed the skills to Roblox’s lead game designer to incorporate into the assessment. To judge an applicant’s creative problem solving abilities, the designers devised a challenge where one would have to build as many robots as possible that could clear certain obstacles. For systems thinking, applicants would need to complete a “pathogen test” in which they would be asked to determine which animals out of a group were most likely to attract disease.

After the designers baked the skills into the assessment, the next step was testing. Roblox volunteers participated in “think-alouds” where they’d have to talk through their thought process in front of researchers recording their every utterance. (I did a think-aloud at the suggestion of the Roblox team, and taking an exam in front of others and having to verbalize your thought process is not for the faint of heart). The Imbellus vets then caucused with the designers to discern where the test may have been flawed, or if there’s a new way of interacting with the test that better captured the skill.

The next challenge: figuring out how to score. Unlike your classic multiple choice assessment, the “questions” in a game-based task are invisible to the test-taker. Roblox’s candidates are graded on certain decisions they make while completing the task. Buckley says candidates need a minimum score on the cognitive test, but the team uses an algorithm to balance out the score with the coding assessment. They typically try to be more generous if someone’s score is on the margins.

“We don’t hire people who are good at one thing,” Buckley says. “What that usually boils down to is, you have to figure out in the entire hiring process when to measure what.”

The cognitive test, which Roblox started using late last year, is somewhere between “the best test” you’ve ever taken or “the worst game” you’ve ever played, according to Buckley. “We do get feedback where people are like, ‘This is so much fun, I’d actually like to keep playing it,’” he says. “But for the most part, it’s got to be in that funny gray area between the test and the game.”

The company has paired the cognitive assessment with a coding one from CodeSignal, another developer testing tool. Roblox only requires the test for students applying to engineering and product management roles, and it administers it after a very basic resume screening that mostly checks for whether the student can work in the United States.

Once the applicant passes the tests, they’re invited to complete technical screenings and interviews. This is where some of the other skills come into play. “Measuring your empathy and communication is not easy to do in a game,” Buckley admits. “It’s much cheaper and easier to do in a conversation.”

No test is perfect

Roblox believes that this test will reduce bias in hiring because it’s based on the cognitive traits of successful employees, rather than identity or educational pedigree.

But creating an effective standardized test is not simple, and creating one completely without bias is nigh impossible. “The more subjective you attempt to make what you’re capturing, the more problematic it becomes,” says Akil Bello, a test-preparation expert who notes that bias can easily slip in when you’re not measuring practical, easily quantifiable skills like coding. “It conveys mushy things like fit, ‘good employee,’ and all of these things that start to create biases and reflect existing biases.”

Bello takes particular issue with the “cognitive” test label, as it refers to someone’s innate being, which is very difficult to capture in a standardized test. It’s a very “loaded title,” he says, stretching “beyond that narrow definition of, ‘these are highly relevant skills for what you’re doing.’” He also points out that using existing Roblox employees as the foundation of the company’s research inherently limits what a successful candidate might look like. “If you’re evaluating who’s a fit for your organization by looking at who’s in the organization,” he says, “then essentially you’re saying the old hiring process is going to be encoded through a system that we get to claim is objective.”

Roblox’s Buckley pushes back on this concern, saying his team analyzed the jobs—and not the people in them. “We do, of course, have to survey, interview, and observe people performing the tasks,” he says, “but the objective is not to build an idealized profile of a Roblox employee—it’s to make sure we are selecting people based on the skills and competencies necessary for success.”

Thus far, at least by one metric, the company is happy with the results. Roblox’s percentage of hires from “elite” schools decreased from 68% to 52%, and its “non-elite” hires grew from 32% to 48%. But Roblox did not clarify whether that growth is due to the assessment or how the assessment impacted diversity areas beyond schooling, as the company declined to share additional demographic data. Roblox also didn’t have statistics on the types of candidates who did well and who did poorly on the exam.

The next test for the test

Other Roblox candidates echo Ritti Bhogal’s first impressions of the cognitive test: pleasantly surprised by the experience, although caught off guard by a test that felt impossible to prepare for. “I had a hard time telling whether I was doing well or not doing well,” says Deepayan Sanyal, a sophomore at the University of Washington. “But I enjoyed the experience. If that was a game on its own, I’d probably play it.”

Slight apprehension from college students who are accustomed to taking exams is one thing. Trying to scale this approach company-wide is another. Sturman, Roblox’s CTO, says that rolling the experience out to mid or later-career professionals will be a challenge. “You ask [experienced job candidates] to take a test and they give you a funny look,” Sturman said.

Bello, the testing expert, has encountered a similar test hesitation among adults in academia as well. “You ask college admissions people to take the SAT,” he says, “they look at you like you just asked them to chop off their hand at the wrist.”

But Sturman is hopeful that the Roblox test will become more socially acceptable in the future. “There’s no reason,” he says, “that our type of assessment couldn’t be used by any company.”