Pathways

We’ve had an epic four months of product building! Our product is 10 times better than it was back in October. And the feature we put the most time and effort into and that I’m most proud of is Pathways.

Pathways are hyper-personalised to do lists to help people into each of their careers of interest, along with time, cost and funding information. It covers qualifications, work experience, professional bodies to join and more. We ask users about their education history, location, employment status, age and salary to personalise the information. We then link people to relevant places like their local college so they can take the next step. This is unique in the market and every user we tested with found it invaluable.

We also gave our whole site a makeover and built a whole basket of other cool features. Such as telling people their skills match to their careers of interest based on what jobs they’ve had in the past. And a marketplace of curated, personalised services to help people with things like coaching and CV writing. We then pulled everything together for our final application for Nesta’s CareerTech prize by the end of January.

I’ve literally never worked so hard in my life. But I’ve never been as productive either. This was a team effort from three of us full-time — me, Emma and Anna (our frontend developer). We had other people contribute too including Josh (our designer) and Leo (Forward Partners’ CTO). And I’m really proud of our progress and excited to see the impact of these new features, and in particular Pathways, in the months ahead.

Try it out at www.wouldyouratherbe.com. And if you want the details of the last four months, read on!…

Pathways

Mapping Highest Qualification Level

The origin of our Pathways feature started with us trying to map people’s qualifications to careers. Back in September we ran a series of product experiments that gave us a lot of insights. One key insight was that what people would value the most is to map their qualifications, skills, personality and values to careers of interest. So we thought we’d better get started by mapping what level of qualification is required for each of our careers.

I spent three full days gathering this information by going through entry requirements for our careers. I used the National Career Service website and various other sites. We then showed the level of qualification required for each of a user’s careers of interest on their final results. In addition, we researched whether each career had lots of jobs today or if it was negatively impacted by Covid. And we displayed this on a user’s final results too. We added these labels in our old app design back in October.

But we then wanted to explore going a level deeper. For example, we wanted to help users identify the specific subjects they needed degrees in for each of their careers of interest. And to map their existing qualifications too.

Qualification Data

So the first step here was to understand the full range of qualifications and subjects available to people in the UK. And it turns out there’s a lot!

For example, there are 15,763 Ofqual-recognised active qualifications below degree level. Ofqual’s dataset gave us this list along with the qualification title, level and some time, cost and funding information. But we had to write scripts to pull out the qualification name and subject into a structured form. The time and cost information was incomplete and not always accurate. The funding information in isolation wasn’t enough to tell us if a given user was eligible to receive funding. We also needed other rules defined in the Adult Education Budget. And most importantly, it gave us no information about whether courses existed to get these qualifications. Or where a user could study a given course.

There are also 35,709 university-course pairs in the UK. HESA’s dataset provided a fairly complete view of these. But it still required quite a lot of work to get the data into the right format for us to use and interrogate for our purposes. And information on Masters or PhD courses / subjects were much harder to come by.

All in all, Emma and I spent about 100 hours in total between us to gather all of this information so we’d have the building blocks we needed. We even pulled together all of the funding rules for everyone in the UK at all qualification levels.

Structured Pathway Data

We then changed tack and started deeply researching one career at a time. With the goal of identifying exactly what was required to get into each one. I started researching 3D Printing Technician and Emma took on Academic Researcher. We decided to go through our careers alphabetically. Then we’d tackle a good, broad spectrum of careers to ensure the structured data we captured would work generally for everything.

Our starting place was typically the National Career Service. But we also ventured into other sites like Prospects, job ads and more specialised sites. We considered the different pathway options at different qualification levels. And we considered non-qualification criteria like tasks and junior entry jobs. We referenced our previous research and datasets to gather information on course providers, time, cost and funding information. And then we defined a structured way of capturing this information in a Google Spreadsheet with multiple tabs for pathways, qualifications, tasks and entry jobs.

It actually took us a couple of days to complete the first three or four careers and to define the initial format for our spreadsheet. We then added and amended the spreadsheet’s structure a lot as we completed more careers.

A Much Bigger Feature

As we started pulling together this structured data for our first few careers in mid-November, we started to realise that our dataset would support a much richer feature than simply qualification mapping. In fact, the whole idea of “qualification mapping” didn’t quite make sense. People are usually more interested in knowing what qualifications they need, rather than the small list of careers they can access with their existing qualifications. And we were even going far beyond just qualification requirements. We were gathering data on time, cost and funding. As well as non-qualification criteria.

So we realised that we could be far more ambitious than just doing qualification mapping. We could build personalised to do lists for people to get into their careers of interest, with time, cost and funding information. It would be a lot of work, as each career may take us hours to complete. And we had 500 to do! But we were not aware of anyone else on the market providing personalised pathways into careers. And it’s what all our users said they wanted from the interviews we did back in May.

Completing Pathways for 81 Careers

We were also 10 weeks away from the deadline for our final submission of Nesta’s CareerTech prize, which we are a finalist of. Up to that point I thought we had a pretty good chance of winning. But if we pulled off this new Pathways feature, we’d have an amazing chance! I knew we couldn’t research all 504 of our careers in 10 weeks. But maybe we could complete enough to demo and test the feature.

So we decided to prioritise careers most relevant to Nesta’s target cohort for this prize. Which were essentially careers not requiring degrees. We also prioritised those with lots of available jobs today. And deprioritised any that were negatively impacted by Covid.

We came up with a list of 80 careers. Then Emma’s goal was to research pathways into these within the next month. Initially, her pace was 2–4 hours per career. But towards the end Emma increased her pace to about 1 hour per career. She hit our target by mid-December, completing 246 pathways in total, covering 81 careers 🎉

Designing the Pathways Feature

While Emma cracked on with the research, I pivoted towards designing the feature in mid-November.

Employing Professionals

At this point, Anna (our frontend developer, provided by Forward Partners’ Studio team) was already 6 weeks into rebuilding our site with our new visual design. She was about 2 weeks away from completing that work, so we decided to continue to employ her to build this new feature. Anna’s help would accelerate our efforts and ensure we kept the quality really high. And she also helped us move faster with the Nesta deadline looming!

We had a couple of other features we planned on building too, which included skills matching and marketplace (more on those later!). So we decided to create new designs with all of these features in mind, especially as the inputs required from the user to power these features would all be part of the same flow. And if Anna was going to build the features, we decided to employ Josh (our designer, also provided by Forward Partners) to design them to a really high quality too.

Wireframes

So I started by creating wireframes for what I thought the new flow could look like, factoring in everything I knew about the features and how they would work. This included mapping out the high-level user flow for the current and new design, as well as wireframing each screen. Here’s the flow I mapped and some of the wireframes…

Professional Designs

Josh then spent a week working on pixel-perfect designs for each screen and thinking through the overall flow. As usual, he used Figma, which is an amazing design tool that lets us easily collaborate in real-time. He shared the progress he made each day so that we had a chance to provide further inputs and refine the designs. Here are some of his designs that map to my wireframes above…

Building the Pathways Feature

And then towards the end of November I started writing the detailed specification for the frontend features (how they display visually), how they interacted with the backend (where the magic happens and the data comes from), and the backend design for the Pathways feature (the data structure and algorithm that powers it).

The technical design specification documents alone were really long, complex and time-consuming. The frontend spec document for all our new features, including Pathways, numbered 42 pages! Anna needed this so that she would know exactly what she needed to build. As Anna has only been on the project for a short period, she didn’t have much context. So it was important to provide all this detail in a spec doc. The backend Pathways spec doc was also really long, numbering 23 pages!

Frontend

Anna and I then worked furiously until the end of the year to build and ship each of the key features. Anna finished the new visual redesign of the site by the end of November. She then created the new input screens to capture location, situation, education and experience information. And then upgraded the final results screens to show a user’s skills match with their careers of interest, a marketplace section to show third-party services to help a user with their career and then the Pathways feature. Anna only took 3 days to complete the frontend part of the Pathways feature. But she had a huge amount of frontend work to do to support the new visual redesign and input screens that were also needed.

Backend

Most of the Pathways engineering work though was on the backend, which was used to power the feature and send the required data to the frontend. This involved turning the 23-page backend design doc into code. Half of the work was actually ingesting the spreadsheet of structured data Emma was putting together into our database and ensuring it was accurate and error-free. I then had to craft algorithms that computed hyper-personalised pathways for users.

Hyper-Personalised Pathways

There is far too much complexity in the Pathways feature to cover in a single blog post. But it’s worth highlighting some of the coolest features that make it so powerful…

Relevant Links

Many of the task cards in a user’s to-do list have hyper-personalised links to help them take the next step to complete that task. Here are some examples…

• For vocational courses that are typically available in Further Education (FE) colleges, we link to the user’s local college website so they can find and sign up for that particular course. We determine their local college by taking their postcode, converting it into latitude/longitude coordinates and then using trigonometry to compute the distance between their postcode and all 583 FE colleges in the UK. We then link to the closest college to where they live.
• For courses run by universities (e.g. degree courses and above), we link to Whatuni, which is a university comparison website. We pass in the qualification type, subject and automatically sort by ranking so that in one click the user can see the best universities in the country for that particular course. They can then use Whatuni’s filtering options to further refine their selection based on their past grades for example (i.e. their UCAS points).
• For tasks that involve joining professional bodies, we link to the specific websites that enable users to immediately sign up and join these bodies.

Time and Doing Tasks in Parallel

We automatically compute the total time it takes to complete all required tasks for a given pathway (ignoring those tasks the user has already completed). In order to do this, we research and record the exact time it takes to complete each task (whether they are qualifications, time spent in a junior role, work experience or other tasks). Some tasks can vary in time, so we capture this nuance too.

We even capture if a task is full-time or part-time. For example, an individual a-level is part-time and specifically takes on average 5 hours per week (we assume 25 available hours per week). Our algorithm then computes which qualifications and tasks can be done in parallel when computing the total time to complete all required tasks. Some qualifications are prerequisites of others too. So we ensure these are sequenced and can’t be done in parallel.

This enables us to display an accurate total time to complete a pathway that factors in all of this complexity. And enables a user to instantly see how long a given pathway will take into a career.

Cost and Funding

We automatically compute the total cost to complete all required tasks. Some tasks are free to complete, and others have costs like vocational courses, degrees and joining professional bodies. Emma deeply researched the average costs for these different tasks.

It was important for us to capture all government funding options available to the user too for qualifications we listed. With the exception of Scotland, there is no grant-based funding available for qualifications at degree level or above. So most of our funding logic applies only to lower levels of qualifications (e.g. vocational). Although, we also capture loan-based options like student loans and advanced learner loans.

It turns out that funding is extremely complex. The Adult Education Budget funding rules for 2020–21 is 100 pages of fairly complex rules. The rules depend on the user’s age, employment status and salary. And rules can vary based on location.

But we also need to map the rules to each qualification. There are different categories of funding, and each category only applies to a subset of qualifications. Conveniently, Ofqual’s dataset provides this mapping. But as there is no good mapping of these qualifications to actual courses, we can only approximate whether a given course is eligible for each of the funding categories.

Our algorithm takes all of this into consideration to compute whether a given user is eligible for funding for a given course. And we use that to compute total cost for a pathway by aggregating all costs that are not funded. A user can then immediately see exactly how much a given pathway will cost them given their circumstances. We even tell them if they would be eligible for funding if their circumstances change (e.g. they become unemployed).

Summarising Time and Cost per Career

We bubble up the time and cost information all the way up to the user’s final results screen next to each of their careers of interest. This enables a user to immediately see how much time and cost the shortest pathway into each career is for them, given their circumstances and background.

The logic starts with computing accurate time, cost and funding information for each task in each pathway in each career. We then compute total time and cost for each pathway using the logic defined above. The time and cost information from the pathway with the shortest time is then used on their final results screen.

So the time and cost information a user sees per career is extremely personalised, intelligent and accurate. And every user sees all of this information immediately as soon as their final results page appears. Which takes the average user only 10–15 minutes to get to from when they first reach our website.

Pre-ticking Completed Qualifications

We capture each user’s highest education level, and extra information that might be relevant to pathways. This can include the subjects they studied and the grades they obtained.

For example, if their highest education level is GCSE, we capture how many GCSEs at A*-C they have. And whether they have GCSEs in maths, english, science or a foreign language. Or if their highest education level is masters, we ask what subjects their masters and undergrad degree was in. These fields are autocompletes, which means that as the user starts typing, they’ll see a list of eligible options they can choose from. This improves the user experience, and enables us to map their inputs directly to our database of over 600 masters and degree subjects.

We then take this information and compute the full list of qualifications the user has in our database. We compute all prerequisite qualifications they must have. For example, if they have a degree in computer science we know they must also have 2+ A-Levels, including a STEM subject and 5 GCSEs including english, maths and science. We also compute any “parent” qualifications they have. For example, if they have a degree in computer science, they must also have a degree in any subject and a degree in a STEM subject.

We apply a fairly complex recursive algorithm to compute all prerequisites and parents to ensure we pick up on all of them, as we need to check each prerequisite and parent for their prerequisites and parents too!

And then we automatically pre-tick all completed qualifications in all pathways for all careers as soon as the user gets to their final results. Which means the times and costs they immediately see on final results factors in what qualifications they already have.

Updating Everything as Tasks are Ticked

Pathways is essentially a to-do list of tasks the user needs to complete to get into a career. As the user completes a task, they can tap on it and it is ticked and marked as completed. Once a task is completed, the time and cost for that pathway is updated and immediately reflected to the user. But as we bubble this information up, the user will also immediately see the time and cost information updated in their list of pathways for a particular career. And even on their final results page with the time and cost to get into each career.

In fact, tasks that a user completes may display in multiple pathways for multiple careers. For example, let’s say the user completes a degree in computer science. Once they tick off this task, we automatically tick it off in every pathway for every career it appears in. But we also know that if they tick off this task, they will have completed any relevant prerequisite and parent tasks too (as defined in the section above). Like A-Levels and GCSEs. So if these are not already ticked, we’ll tick them off too. And not only for this pathway, but for all pathways in all careers where they appear.

So one tap from a user ticking off one task has far-reaching, cascading impacts on many pathways across many careers. And it will instantly update the time and cost information for all of their careers on their final results page, showing them exactly how long and how much money each career will then take them as they make progress.

User Testing & Refining

We finished and shipped the first version of the Pathways feature on December 31st. Our goal was to finish by the end of the year, so we hit that goal 🎉

We then had 17 working days to user test, refine, polish and write up our Nesta application before the January 27th deadline. Over the next two weeks Emma set up and ran 18 user tests, all over Zoom. We learnt a tremendous amount, and she summarised the key items we could address quickly (across all our new features, not just Pathways). Over that time, Anna and I made 17 improvements based on this feedback, and we were able to test our improvements with each successive user test.

For example, we:

• Refined the order of careers on the final results screen
• Added instructional screens before and after the quiz to better guide people through the flow
• Refined the titles of the pathways
• Updated the home page to include our top features

We made many other bug fixes and improvements based on my testing too.

So we refined the product rapidly in January. And we’ve continued to do that in February too. We probably have about one more week of refinements to make. And then we can move our focus on to the next part, which will be to complete pathways for the remaining 423 careers. As the product and data structure will be well defined by then, we should be able to power through these in a couple of months.

If you have an engineering or product background, I can give you some insight into the scale of work since January, using two different ways of measuring units of work. We:

• Wrote up and completed 61 Jira tickets
• Submitted 93 PRs (Pull Requests)

Makeover

We didn’t just work on Pathways over the past four months. We gave our site a full makeover, massively improving its visual design. This was essentially an implementation of the design work Josh did last August (as detailed at the start of my last blog post).

We employed Anna at the start of October to implement these new designs. She has a lot of experience with React, which is the technology we use for the frontend of our site. The first time I ever used React was when I built Would You Rather Be back in 2019. So it was a relief to have someone with a lot more experience improve our site and code.

Rebuilding our site in the new designs took time. Anna worked on this for two whole months full-time, from the start of October to the end of November. She spent a month building all the individual components (e.g. buttons, inputs, graphical elements etc.). And then a month building all the screens, by laying out the components she created. Anna created everything from scratch, with minimal reliance on external libraries. This maximises our control over the look-and-feel of the site and our ability to extend it in the future.

Anna also optimised the site for both mobile and desktop. The design is responsive, which means that as the screen expands beyond a certain size, the design changes to make use of the extra space.

Here’s what our new home page looks like, which nicely illustrates the new visual design (and you can fully experience the product for yourself if you like!)…

And here’s the before and after of some of our key screens, so you can see the difference!…

Skills Matching

We started working on the Pathways feature because one of the top pieces of feedback our users voiced was wanting to map their qualifications and skills to careers. Pathways solves for the qualification mapping part, but not skills mapping. So we wanted to try and tackle this too.

Choosing a Skills Taxonomy

We started by reviewing a few different skills taxonomies including O*NET (freely available US-focused data set), EMSI (paid, proprietary data set) and ESCO (freely available European-focused data set). We settled on ESCO in the end because it was free, most relevant to the UK jobs market and it was the only one with a hierarchical skills taxonomy.

The ESCO Data Set

ESCO allow you to export a set of CSV files (spreadsheet format) that contain a list of:

• 2,960 occupations
• 13,485 skills
• 114,393 occupation-skill mapping pairs (so each occupation has on average 39 skills)

And it contains a further 673 skill groups. Every individual skill is part of one or more skill groups. And each group is part of one or more skill groups. There are 4–5 levels of hierarchy, with only 29 skill groups at the highest level. Some examples of the highest level skill groups are information skills, management skills and education.

Taking a full example, one of the 13,485 low-level skills is JavaScript (a software engineering skill, which happens to be the language that Would You Rather Be is coded in). This skill actually has three parents — “computer programming”, “web programming” and “software and applications development and analysis”. And each of these skill groups have parents, who also have parents, which go all the way up to the 29 top-level skill groups. For example, “computer programming” has two parents, one of which is “digital content creation”. This skill group has one parent which is “digital competencies”. And this skill group has one parent called “application of knowledge”, which is one of the 29 top-level parents.

Now the Javascript skill is linked to 53 different occupations, some of which have it as an essential skill and others as an optional skill. For this particular skill, the vast majority of occupations actually mark it as an optional skill, probably because most developer occupations can use any number of different languages. Some example occupations that have Javascript as an optional skill are software developer, software tester and web developer.

Using ESCO Data

The first step we took was to get all of this data into a single Google spreadsheet, split out across multiple tabs. We then put quite a lot of work into stitching the data sets together into the format that enabled us to interrogate it further.

Once we understood every aspect of the dataset, we were able to craft an algorithm that suited our needs. It’s worth taking a step back at this point…

What Matters To Us

An algorithm is only as good as the data you feed it. So the most important first step for us is to prioritise the dataset and ensure it is well understood. It’s then important to clearly identify what problem you’re trying to solve. In our case, we wanted to:

1. Capture the skills a user has likely picked up during their career so far requiring less than 60 seconds of time investment from the user.
2. Map these skills to all of their careers of interest to provide a single skills matching score. We only wanted to give a rough guide as to how well each of their careers of interest are to their skills. So a single score best achieves this rather than providing any more granular detail that may be more difficult for the user to parse and make decisions from.

Given that, we needed to be able to do the following with the data set:

1. Get a comprehensive list of occupations a user could do (more comprehensive and detailed than our existing list of 504 careers)
2. Have a mapping of these occupations to skills
3. Map our career database to this list of occupations so that we can do the final part of the skills mapping to a user’s careers of interest

Other attributes of the dataset were less important to us. For example, one of EMSI’s selling points was that they keep their skills taxonomy up-to-date, but ESCO is only updated every few years or so. But as our mapping is relatively high-level and not too granular, having the dataset updated less frequently isn’t as critical to us.

Our Algorithm

We then designed an algorithm that takes as an input the user’s past jobs and their careers of interest. And outputs a skills match score for each of their careers of interest. Our algorithm essentially computes the overlapping skills from their past jobs with their careers of interest in order to determine a score.

And this is where the hierarchical nature of the ESCO data set was really useful. Continuing from our previous example, if a user says one of their past jobs is a web developer, then we’d record some score for the Javascript skill. Let’s say they then identify Public Relations (PR) Officer as a career of interest. This career doesn’t have Javascript as an essential or optional skill. But it does have “compile content” as a skill. And two levels up, this skill shares a parent skill with Javascript, which is “digital content creation”. Which means there is some skills overlap between these occupations, but not as strong as if they both required Javascript as a skill.

So we factor this into our algorithm. Instead of just comparing low-level skills, we compare skills at every level of the hierarchy, all the way to the 29 top-level skills. But we apply different weightings to each level. So if two occupations overlap on low-level skills, they would get a higher score than if they only overlapped on higher-level skills in the hierarchy.

And we apply lower weightings if the skills are only optional rather than essential for the user’s past occupations or their careers of interest. Plus when we capture a user’s career history, we ask for their current job and past jobs. We then apply a stronger weighting to skills in their current job as they would have been exercised more recently, so are likely to be stronger.

Designing Skills Matching

We then designed the feature visually. Both the inputs where we capture the user’s career history and where we show the skills match to the user in their final results. Here’s how these screens look:

Updating ESCO’s Data Set

We then had to update ESCO’s data set to fully support our algorithm. For example, we had to manually map our 504 careers from our database to their occupation list. We decided to use their list of 2,960 occupations to power the autocomplete input fields where the user specifies their current and past jobs. But in order to map to a user’s final results, we still had to map from their list to our career list. And some didn’t map perfectly. So we then had to create new occupations, along with a manually curated list of skills (which we pulled together by combining the relevant skills from multiple similar occupations).

And then we made a number of small additions and edits. For example, the exact names of some of our careers were missing in their occupation list. We added these so they would appear as options the user could select when inputting their past jobs. And the ESCO data set was about 98% accurate and complete, but still had a few holes in it. For example, there were some gaps in the skills hierarchy that needed filling in.

User Testing Skills Matching

We tested this algorithm with ourselves and a few friends. I was really happy with the results. It worked much better than I expected and produced skills matching scores that seemed fairly accurate. And it’s fairly sophisticated too. To compute the skills match for the average user with 20 careers of interest and a few past jobs, the algorithm executes about 500,000 computations. But thanks to the speed of computing, this happens instantaneously. So it feels like magic as you can see the skills match score for all your careers of interest about 2 seconds after you enter your career history.

Marketplace

We’re a software company. And software is really well suited to solving some of the problems we’re working on. Like helping someone discover and get into a career that will make them happy. But it can’t solve all the problems a user will have on their career journey. For example, many people require some emotional support, encouragement or quite specialised coaching and mentoring. These problems are often best solved with people rather than computers.

We landed on the idea of building a marketplace of curated, personalised third-party services back in September, when we were half way through our product experiments. So that we could help people with the other problems we weren’t going to solve directly ourselves.

Researching Providers

Emma spent some time researching the best services that solve different parts of the career journey and created a curated list of services, with information about each one.

She researched services that help people more generally with their career, like the Prince’s Trust who provide work experience and training for young people, and PushFar who provide a mentoring service. And she researched services that help people get into specific careers, like Resume Worded who help with CVs, and Mock Interviews who provide practice video interviews with feedback.

She also captured lots of relevant, structured data for each one, including the price and who they are targeted at (e.g. age restrictions, education level and employment status). For example, one of the services is Beam, who crowdfund the cost of training to help homeless people back into work. This is only relevant for people aged 18 and over, who are unemployed and homeless. It’s also only relevant for 45 of our 504 careers, as defined on Beam’s career pathways page.

Building & Testing the Feature

So when we built the feature, we decided to only display the curated services which are relevant to a given user based on their age, education level, employment status, living situation and even the specific career they are exploring. This has resulted in a list of high quality, relevant providers. We then show this list in one of two places. On the final results screen for the more general service providers. And a different list on a career-specific screen for providers that help people get into a certain career.

When we tested this feature, it performed the worst of all our features in terms of engagement. From the feedback we received, it’s probably because it appears at the bottom of each of the screens, and looks like a list of ads or sponsored content. So users discard the content as not being too relevant. In contrast to this, we had extremely high engagement on third-party services we linked to from specific Pathway tasks, due to their relevance (e.g. their local college or a university comparison site). So we plan to merge these Marketplace providers into our Pathways feature in the future.

Nesta’s CareerTech Prize

Nesta is an innovation foundation. They provide grants to innovative companies who are working on socially impactful projects. One way they do this is through monetary prizes. We are a finalist in their CareerTech prize.

We were shortlisted as one of their 20 finalists last March, as I talk more about in this blog post. They gave us a grant of £50k and 9 months to build a solution. The target users of the prize are people in England aged over 24 who don’t have a degree and work in an insecure job. The focus of the prize was to use labour market information to support people to make more informed decisions about their future careers. And we had 9 months to build a solution. The deadline to submit our final application was January 27th. The winner gets a further £120k and lots of good PR. That was the catalyst for why we worked so furiously to deliver these features so quickly.

Strategically, the prize didn’t affect what we worked over the last 9 months too much. But we did spend most of January focusing on our final application. That’s partly why we ran 18 user tests and interviews that month. And not just to test our features and get feedback. But also to get measurable, qualitative impact data on the value of our solution.

We also tested our app with thousands of users. Over a two week period in January, we had 4,133 users complete our quiz, reaching the final results screen. Of those, 3,396 (82%) were either unemployed or unhappy in their job — our overarching target users. And of these, 1,026 (30%) were within Nesta’s target cohort for the prize. We gathered a lot of really encouraging engagement and impact metrics from these users…

Engagement

We worked hard to make our app as engaging and easy-to-use as possible. For example, we streamlined all the user inputs that powered all our personalisation features down to four screens. This captures their location information, employment status, salary, age, education information and career history. And from testing, users are able to complete these screens in 1–2 minutes.

For the more complex inputs, like subjects they studied and past jobs they’ve had, we use autocompletes. This means that as they start typing, suggestions appear underneath that they can select. And they must select one of the options we present so that we can map their inputs to our database.

We worked really hard to make these screens as optimal and easy-to-use as possible. Here’s what they look like:

Our Engagement Metrics

And here are our engagement metrics, which are all very encouraging:

• 83% of people (6,485) who started the quiz finished it. They answered 50 or 100 questions.
• 82% of people (4,484) who started the input screens (as above) completed all four and got to the final results screen.
• 57% of people (4,484) who started the quiz got to the final results screen. They went through all of our screens, including quality and career selection. Note that we count 4,133 (53%) as “valid” final results users. This is defined as having taken at least 4 minutes to complete the quiz and choosing at least 10 of each of the career buttons across all the quiz questions.
• 60% of people (2,467) who reached final results clicked “Explore” on at least one career.
• 28% of people (1,052) who had at least one career with Pathways enabled went on to view at least one Pathway.
• Of the people who viewed a Pathway, 30% (319) clicked at least one button to take them to the next step. And 16% (170) ticked at least one Pathway task to mark it as complete.
• 26% of people (647) who explored a career viewed jobs nearby.
• 24% of people (977) emailed their results to themselves voluntarily. This is a 3x improvement from our product a year ago, showing the dramatic increase in value.

Our Impact Metrics

To measure impact, all our users completed a short survey before they started our quiz. We then asked users to complete a survey a couple of minutes after they reached their final results. If they ignored this quiz, we’d automatically pop it up again every couple of minutes until they completed it. 431 users who reached the final results screen completed our “after” impact survey (13% of all users).

Here’s what our two surveys looked like:

And here are the outcomes of our 431 target users who completed both surveys:

• 24% (88 people) who didn’t expect to one day find a job they’ll love now do as a result of using Would You Rather Be
• 82% (355 people) feel they now have more information about careers, received tailored information or had their career horizons expanded
• 52% (223 people) feel they better understand pathways into a new career
• 60% (257 people) found the current and future job demand labels helpful

I’m particularly proud of the fact that 24% of people who were unemployed or unhappy in their jobs and who didn’t say “yes” to expecting to ever have a job they’ll love answered “yes” to this question after using our app (just 15–20 minutes later). It’s so encouraging to me to see the positive impact our software is already having in the lives of those 88 people.

We also had some encouraging feedback that people submitted through our app and email surveys. Here are some of those quotes:

Written Application & Video

Emma and I then spent a full week writing our 7,000 word final application and recording a 3-minute video summarising our solution.

This was a lot of work to pull everything together! We employed a professional videographer to help us with the video. And we’re really pleased with the result. I’m also really proud of our application. And it was a great way to reflect on the journey we’ve been on, what we’ve learnt, the progress we’ve made and the impact we’ve demonstrated so far.

Nesta will be announcing the prize winner and runner-up at their end of programme showcase event on March 23rd. So fingers-crossed for that!

Our Final Solution

I’ve shown you snippets of different parts of our upgraded app throughout this blog post. And of course, you can try it out for yourself here and discover your top careers along with your own hyper-personalised pathways into them. But I thought I’d also share the full end-to-end flow of our app so you can see what it all looks like put together:

Another Nesta Prize

Nesta launched another prize that was relevant to us in October called the Rapid Recovery Challenge. This was in response to the pandemic and its impact on employment. We applied for this prize, but sadly didn’t make the cut. We were probably a few months too early, as this was back in October before we had even updated the visual design of our site or even conceived of Pathways.

Other Work

So you can see we’ve been pretty busy the past four months building Pathways, Skills Matching, Marketplace and pulling together our final application for Nesta’s CareerTech prize. But we’ve also done a few smaller pieces of work in that time too. Here’s a quick snapshot of those…

3 More Product Experiments

My last blog post in October was all about the 15 product experiments we ran in 6 weeks. We ran another 3 product experiments soon after, bringing us to 18 in total.

For our 16th experiment, we ran another survey on 971 users to gather data on their age, salary and impact of Covid. This helped us with our application for Nesta’s new Rapid Recovery Challenge prize and gave us further insights on our users.

Our 15th experiment was to see if we could get our users to sign up to a Learning Curve Group course. And in return we’d be paid a commission. While a lot of our users clicked through, none actually signed up. So we tried an alternative approach for our 17th experiment by asking users to provide their phone numbers. The Learning Curve Group call centre team would then call them and try to get them to sign up. We had 480 people who were shown a course, and 9 gave us their phone number. But unfortunately none of these users then signed up for a course after they were called.

And our 18th experiment involved a tweak to our feature set to see if the Adzuna job ads performed better, which they didn’t.

Job Demand Labels

As I mentioned earlier, we added labels in our app back in October to say if any given career has lots of jobs today or is negatively impacted by Covid. In January, we added a couple more labels to say if any given career is future proof or not. We researched jobs that are likely to be in high demand in the future or at risk, using a number of sources. For example, we looked at:

• ONS occupations at risk of automation
• Skilled worker visa shortage occupations
• O*NET bright careers (based on the US jobs market)

And here’s what those labels look like in our app today on the career cards in a user’s final results:

Psychometric Research

In November we opportunistically hired someone with a PhD in psychometrics to do a week’s worth of research for us. Her name is Cat and she did an excellent job of compiling a summary of all the main research relevant to us.

At some point this year we want to map people’s personality and aptitude to careers. These two traits are typically called “psychometrics”. And it’s a natural progression of our current mapping of people’s interests, qualifications and skills.

But in the same way we built our existing mapping from first principles, using sophisticated algorithms and rich data to create an experience that really works, we want to take a similar approach to psychometric mapping too. So we thought it would be helpful to start by reviewing all the existing relevant research in this space so we can learn from that and determine what our approach should be for the specific problem we’re trying to solve.

Cat compiled 55 separate research papers for us as well as a list of 27 other companies who have an offering in this space. And she also summarised all of her research in a 21-page document. It was really excellent work in a relatively short space of time. And it will make for some interesting reading later this year!

No Degree Filtering

A lot of users don’t have degrees. In fact, the target user group for Nesta’s CareerTech prize are specifically people without degrees. But it turns out that 292 of our 504 careers (58%) typically require a degree to get into them. So if you don’t have a degree, you only have access to about 42% of available careers.

From a product perspective, we didn’t want to show these careers to people who didn’t have a degree or aspire to get one. So we asked them these questions at the very beginning.

And if they said “no”, we hid all careers that typically require a degree from the quiz and the final career selection screens. As the list of careers was so much smaller, we didn’t need to ask so many quiz questions to narrow down the careers suited to them. So the quiz for users who don’t have a degree or aspire to get one is only 50 questions instead of 100.

Supporting “Back”

We’ve always had some minor feedback that users sometimes want to be able to go back to change some of their answers in the quiz questions. But if they try pressing “back” on the browser, it will actually take them back to where they came from prior to Would You Rather Be, which is usually Facebook. And that’s a pretty terrible user experience.

It happens that way because I built it as a one-page web app. So as you move between screens, the URL (web address) doesn’t change. To support “back” properly, you need unique URLs for every screen.

So Anna and I worked on supporting this at the end of January. I thought it might be one day of work, but it took us both about two full weeks! It turns out that there’s a lot of complexity to deal with here, mainly due to the dependencies between screens based on the user’s input. Plus Anna had to rewrite a lot of the original code I wrote to support this new approach.

But it’s supported now, so users can go back, forwards or jump around URLs as much as they want (assuming they know how to do that). And this change brings other benefits in the future too, like enabling a user to update their location, situation, education or experience information really easily for example.

Tracking

So far we track what users do by recording user attributes directly in our database. I then build custom stats pages to pull out what we’re interested in. Every time I’ve reported metrics in past blog posts, including this one, that’s how we did it. And it’s a little time-consuming coding a new stats page each time.

So we finally implemented Amplitude, which was our tracking solution of choice to make this easier. It took a couple days of work to fully design the schema and implement. But now we can create new dashboards and funnels without writing any code. I used Heap over a year ago in the early days of Would You Rather Be, which was a little easier to set up, but was very expensive. And Mixpanel is probably the market leader in this space, but is even more expensive. With Amplitude we get to track 10 million user actions per month for free, so I’m not even paying anything yet!

We also implemented Hotjar, which I’d not used before. It literally only took a few minutes to implement. And it’s super-powerful. It basically records the entire screencast of a user’s journey through our app, so we can watch these videos back to see exactly what people do and where they get stuck. And it has some clever anonymising technology so it protects the user’s identity. I think this will really help us in drawing user insights going forwards as we release new features and updates.

And finally, we implemented Sentry, which was recommended by Leo, Forward Partners’ CTO. This software tracks crashes and major errors that happen as people use our app. And sends me emails as soon as they happen, so I can fix it straightaway. Which is brilliant!

Market Research

One of the target customers we’ll be exploring in the coming few weeks will be schools. We’ll start by talking to a number of career leaders and head teachers of schools. But as a starting point, we decided to do some competitor research to find out what products schools currently use for career provision.

Emma researched 33 different products, and went on to demo and explore more deeply the best ones and those most similar to ours. These were:

• Morrisby, which is probably the market leader. Their main focus is private schools. And they recently acquired a company called Fast Tomato who also have a popular product. Morrisby are particularly well-known for their psychometric testing.
• Cascaid, who have an older product called Kudos and a newer product called Xello.
• My Career Options, who are a newer company. They don’t have much traction yet, but seem to be doing interesting things in the area of psychometrics.
• Unifrog, who isn’t strictly a competitor as they focus more on supporting students into universities. But they have good traction in schools and a great product.

We also started talking to a few people who work at schools and universities. Emma and I spoke with a careers leader of a state school, a lady who is involved in careers work at a private school and someone who is involved in careers work at a university. We plan to talk to about 50 people in total by the end of March across schools, colleges, universities and training providers.

Hiring Employee #2

And the last thing we did was hire our second employee!

Here’s a brief history of employees at Would You Rather Be…

• I founded the company in June 2019 and worked part-time on my own for the first year
• I’ve been full-time since June 2020
• Emma joined me full-time in August 2020 and there’s only been two of us full-time since
• Anna also worked with us full-time on frontend development from October 2020 to February 2021

One of the big tasks ahead of us for the next 2–3 months is to complete pathways for the remaining 423 careers. We decided to hire someone to help us with this so that we can free up Emma’s time to work with me to figure out how to make money and get some commercial traction. This is a temporary position, but may become permanent depending on a number of things.

Hiring Process

Emma led the charge with this hire and we posted a role titled “Research and Data Officer”. We paid for prominent placement on Escape the City (which cost £160). As that site yielded the best candidates when I hired for Employee #1 (and it’s how Emma ended up working here!). We had 120 applicants in total, with Escape the City providing the best quality ones again.

Emma then prepared an excellent task that we asked our top 10 candidates to complete. We asked them to research all the pathways to become a Detective, and then input their research as structured data into a spreadsheet we provided (which was a cut-down version of our master Pathways spreadsheet). This task gave us a really strong signal on both competence and motivation, the two things we cared most about. Emma then interviewed the top 5, and I interviewed them too.

We both completed a scorecard at each stage of the process for each candidate against attributes we decided were most important. And then we discussed our feedback at length and made a decision. We extended an offer to an excellent candidate called Laura who accepted on Monday! So we’re excited to have her join the team next week! 🎉

Hiring Reflections

We put a lot of effort into hiring, onboarding and all people-related activities. Our people are the heart of our company and I believe it will be the biggest determinant for our long-term success. Of all the things we do and spend time on, hiring is one of the most important. Not to mention that our mission is career happiness, so it’s pretty important that we hire the right people for what we need and do everything we can to make sure they’re really happy in their roles.

So now we will be three people from next week. I supposed we’ve been three people for most of the last four months with Anna working with us on frontend development. But we knew that would be temporary as Anna is part of the Forward Partners Studio team. But there is a possibility we’ll extend Laura’s 2-month contract or that it might even become permanent, so it really does feel like we’re growing in size by 50% 😊

What’s Next

We have one very specific goal for 2021. And that’s to make £100k in revenue. Our focus this year is commercialisation so that we can sustain and scale our impact in the future. But we won’t compromise our social mission along the way.

Our thinking around this is fairly fluid at the moment, so it’s a bit too early to share. But that’s our direction of travel for the rest of the year.

If you’ve read this far, thanks for following our journey! And if you haven’t yet tried out our updated app, head over to www.wouldyouratherbe.com and give it a go!