Follow us on LinkedInThis article derives from a presentation from the Developer Marketing Summit, from July 2021. Watch this presentation, and others, via our OnDemand service.
My name's Anna Borbotko, and I'm the Product Marketing Manager at TomTom.
I'm going to share our Mission Possible: the science behind gathering feedback from developers, specifically on such products as APIs and SDKs.
Truly understanding developers have, for some reason, always been a challenge, especially for those who come from business backgrounds.
In this article, I’ll focus on:
Developers vs average humans
Here’s another question for you: How different is a developer from an average human being? Let’s look at some of their core traits, then you can decide.
Firstly, developers love simplicity. They deal with complex algorithms in their jobs, and they love solving problems, but in normal life, they want everything to be as simple as possible.
Now, if you look at the average human, our brains are wired for simplicity, as shown in research by Daniel Kahneman, author of Thinking Fast, Thinking Slow.
Another major characteristic of developers is having a sense of purpose and wanting to know that whatever they do has an impact. The topic of purposeful living took off about seven years ago and is still a hot topic for many of us, including developers.
I remember when I was working for a startup that built software for the EdTech industry; we wanted to help overcome the shortage of teachers with technology.
During lunch breaks, developers would often share how recruiters from big companies and banks had approached them and offered to triple their current salaries, but no amount of money would make them switch if they didn't see the purpose behind the role.
The next item on my list is that developers really don't like spam. I don't need to tell you that most developers have ad blockers, but they also ignore cold marketing, cold sales emails, and almost everything that's not relevant to them. If you look at most non-developers, they’ll also get pretty annoyed by this kind of spam.
Lastly, developers like to test things before they buy, and the same applies to us. If we buy shoes, we try them on first. Or, if they were shipped to our house and didn't meet our expectations, we return them.
So, the question is are developers that different from us? My answer is no. We’re all human, after all, so how can we use that knowledge to gather feedback from developers? Let me walk you through it.
But first, some context
I'm a product marketing expert at TomTom, a location technology provider. The project I'm working on right now focuses on gathering feedback in B2D and B2B environments in a way that’s scalable and accessible to all relevant stakeholders.
TomTom has changed over the last few years. It has transformed from a personal navigation device company into a giant that now competes with the likes of Google.
TomTom now has a diverse product portfolio. Its products have a range of different use cases, and they fit differently in different markets. We’ve got APIs, SDKs, and even map data.
We in the PMM team are working on our messaging and positioning and go-to-market strategies, and for that, we need input from customers.
On the other side of the table, we have product managers who are constantly asking for feedback and using that information to improve our products or even build new ones. Both functions constantly require access to some information from customers, which can be a challenge.
In many organizations, you have a dedicated customer success team constantly working on feedback activities, and if you don't have that, a PMM will typically take over that role – that's the case at TomTom. We don't have a dedicated customer success team.
We also don't have a user-facing application because we work in a B2D and B2B environment, so we can't really go behind the scenes to monitor the behavior of our users. All we have is our sales teams, customer support, and developer portal.
Customer feedback is available all over the place. We have internal teams constantly collecting information.
We have external sources like social media and different analytical tools. I could go on. The problem is not that we’re disorganized; it’s just that there are so many different channels that it makes our feedback tricky to analyze.
If you work in a high-tech company, the chances are that the majority of your colleagues are developers.
That’s certainly the case at TomTom, so we decided to check what they would advise us to do if we wanted to collect feedback from developers. We also spoke to business teams and product managers to see what their feedback requirements would be.
We gleaned many insights from those conversations, but one thing I really would like to highlight – and it's not rocket science – is that asking for feedback is one thing, but if you ask for it, you have to act on it.
Of course, it depends on the quality and quantity of the feedback but, generally speaking, feedback means responsibility.
If nothing happens for a while in response to feedback, people don't see the point of sharing it with you anymore, and this makes gathering feedback even more complicated.
The three-step approach to developers’ insights
Based on what we learned from our conversations with developers and other stakeholders, we came up with a three-step approach to feedback-gathering activities at TomTom.
- The first thing you need is a developer experience index (DXI). This is a set of metrics that are measured and analyzed regularly.
- Secondly, you need to establish a conversation with developers through actions.
- Lastly, you need to build a learning community, and that can be internal, external, or both.
Step one: The developer experience index
Let's start with step number one – the developer experience index or DXI. We didn't take this from a big consultant’s report or a clever book – it’s purely a TomTom improvisation. We're still refining it at the moment, so we don't yet have all the insights about how well it works, but bear with me.
We’ve broken our DXI metrics down across the five stages of the developer journey.
First is the discovery stage, which is related to everything your marketing teams are working hard on. This is where you’re answering questions like do I want to use this? Do I need this? And will it help me solve my problem?
The next stage that a developer will probably go through is onboarding. This is when they’ve decided that your product will probably work for them and they need you to help them get started.
The third stage, which is crucial for developers, is documentation. That's the stage that answers questions like how do I use these APIs? How do I ensure that what I will implement in my product is implemented well? If I struggle, how can I get help? And that's where our fourth stage, support, plays a big role too.
Our final stage, assuming the developer sticks with your company, is loyalty. You want to measure their loyalty to your brand and products.
Across the different stages of the developer journey, we collect quantitative metrics, such as net promoter score, customer satisfaction score, and customer effort score.
However, those metrics mean nothing unless they're supported by qualitative information, such as developer comments. That's where sentiment analysis comes in handy.
Why DXI?
You might wonder why it’s worth doing DXI in the first instance. Well, let me tell you.
First of all, it’s effortless for developers – just one click on the star rating or a thumbs up or thumbs down, plus a comment in the text field if they want to leave one.
DXI also happens in real-time as developers are building their products. They can react immediately and give you more insights into your products. It's consistent too because you collect the same data inputs all the time, so you can easily compare data sets.
Finally, DXI takes a pull rather than a push approach. Users are not forced to give feedback; they’re simply given a channel to share feedback if they feel like it.
How to collect DXI
So how do you collect DXI? Well, you establish different metrics on different pages.
For example, on each documentation page in our developer portal, we have a thumbs up and thumbs down button for the CSAT metric, and we also have a text field. That way, when people are going through the documentation, they can immediately tell us if they found what they were searching for.
We have a similar system in the welcome emails that we send them out once developers register with us, so they can give us a rating out of five to tell us if their onboarding experience was on par with what they expected.
All this comes together nicely in a dashboard that everybody has access to. Product managers, product marketing managers, and anybody who needs to know what developers are thinking can monitor the information they need there.
Now, I know that marketing has a lot of sophisticated tools for feedback gathering. We haven’t investigated those tools yet, but we’re planning to at some point.
For now, we’re using the tools that we already have in-house, so our dashboard lives in Power BI, we use Mopinion to collect the feedback, and we use Salesforce for any additional input that we’re unable to collect from Mopinion.
Our learnings
What do you think we’ve achieved in six months, thanks to DXI? Remember, this is still a pilot project, so don't get too hyped up, but there are definitely some results worth sharing.
We collected the data, analyzed it, and we figured out that some of our products are not positioned correctly. Right now, we’re busy trying to fix those positioning issues that we encountered.
We also learned that our documentation needs to be simplified. Now, I also know from talking to other experts, that your website is a living thing that needs to be constantly updated, but still, there's always room for improvement.
We've also realized that there are some product issues that we couldn't capture before. The product team has been working hard to fix those.
Finally, over the last two years, we’ve spent a lot of time working on the onboarding experience for all of our developers. We thought we did a pretty good job, but apparently, developers are still struggling. We’re fixing that right now, and hopefully, it will soon be off our roadmap.
Step two: Conversation through action
The DXI is already an awesome source of information, but we humans always want more. That’s why we came up with step number two, which relies on building a conversation through actions to gather developer insights.
We noticed that from time to time, our users churn and we have no idea why. Two years ago, we tried to interview some developers; we issued surveys with free credits and Amazon gift cards, but nothing worked – we would get very few results if any.
Again, we approached our internal developers and asked them for help. As you can probably guess, we had to go back to the drawing board.
We found out that our previous attempts at surveying developers were lacking single-click answer emails. This is crucial because developers’ perception of surveys is that they’re boring and will require a lot of effort from their side. We also learned that mandatory fields in the survey are really demotivating.
The next takeaway we got is that time is crucial when approaching developers. They’re not interested in investing their time in your company unless they know that their feedback will change something. For this reason, it’s best to approach developers when they're still building or testing APIs from your company.
Finally, developers don't like marketing in general, so marketing-generated emails are a definite no-go. In fact, they want a developer-to-developer approach in a more personalized email, where you’re genuinely asking for help. Writing concisely and with a humble tone of voice also helps.
Features upvoting
I’ve heard quite a few times from developers that they appreciate it when you have your roadmap open to them so that they can see what features are coming; that helps them with planning their own roadmap and feature releases.
You can also allow developers to submit feature requests, which the developer community then can upvote. That’s going to help your product managers with prioritization.
Step three: Always be a learning community
I recently read a book by DeveloperMedia where they interviewed developers on their marketing preferences. The conclusion was that developers can get hooked by ads if they’re cool or related to education or an event.
However, a lot of the time, learning happens in the community, so if you're a company that offers APIs, you should start thinking of building your own developer community or using external sources.
Stack Overflow, Reddit, and GitHub are the usual places developers go when they need to ask for advice, but there are other communities too.
For example, TomTom has its own forum and community activities, which we invest a lot of time in, and there are other communities dedicated to specific coding languages.
Coming back to the interviews we did with our developers, most of them confirmed that if you want to engage in conversation with developers and get feedback, email is not the best channel. Forums are the way to go.
Another area of exploration that’s often forgotten or underestimated is universities.
Although students don't have money right now, they might well be founding great startups or working for big corporations one day. It's extremely important to think of that community of future customers and show them your brand at that stage.
Perhaps students don't have resources, but they can be invaluable if you give them predefined tasks, let them build something with your APIs, and ask them about their experience.
Or, you can make a deal with a professor and let students go wild with their ideas, which might uncover other markets or potential for your products.
This method, of course, has its disadvantages. Students aren’t (yet) your target audience, and they’re not going to solve all the problems that your target customer might face with the product.
However, this approach will help with things like developer flow, making foolproof documentation, and creating a smooth onboarding experience.
Now you have the tools to…
- Effortlessly gather feedback from developers in real-time
- Establish a continuous dialogue with developers through the right channels
- Deliver experiences that developers love
But if you remember only one thing from this article, remember that developers are human and they like and dislike the same things we all do.
