The Community Builder’s Guide to Member Research Surveys
Survey research is fast, nearly free, and broadly accessible. But community builders need to know that survey research is only valuable to a point.
For instance, asking members directly what they want or what the community platform should be will only give you superficial, unreliable responses that could steer you in the wrong direction. That’s because when we ask others to self-report their wants and desires in a survey format, they tend to rush, lack self-awareness, or simply not know the answer.
When asked via survey, people will almost always say they want to participate in a community. And that you should create all kinds of community activities, events, and programs that, once launched, they will never attend.
What’s more, asking people what they want is a valuable action on its own - it helps you hear and see your members as full human beings. But if you then cannot deliver on those wants? You have asked someone to risk vulnerably sharing for no return. There is no faster way to deplete respect and trust.
However, member research surveys have a place in our toolkits.
Surveys are an excellent fit for four community-building goals*:
Understanding diversity of membership and setting benchmarks for community health
Rapidly inspiring new content and programming ideas
Ranking popularity of specific needs, goals, and programs
Gathering a list of people who want to get more involved
One short survey can give you a good starting point for further inquiry on all of these goals, but you can just focus on one for more depth. Once you know your goals for survey research, you can then write questions that get you the insights you need.
*My social scientist research experience requires I say: Quantitative survey research is a nuanced methodology and the best approach ultimately depends on the context and your research questions (surveys can answer some research questions but not all and always come with limitations). This post, however, simplifies surveys for community managers, especially those building communities from scratch or who are 1-3 years into the process. If you have hyper-specific survey questions and goals, contact me to explore working together.
Setting Goals for Community Member Surveys
Survey Goal 1
Establish Demographic
and Health Benchmarks
If your primary goal is to understand the common identities shared by your members, a survey can clarify and focus your member personas. Demographics give you at least a surface-level understanding of potential member identities and help you understand how diverse or homogeneous your community is (and make changes accordingly). You may also ask about psychographic information (personalities, interests, opinions) to gauge similarities and differences among members. (Shout out to Robert Yang for this idea and evolution of my thinking!)
It can be smart to include a "Decline to state" option here and to include open responses for any of these, ensuring more inclusivity in your survey design.
In addition, you may want to establish community health benchmarks. For instance, you can ask a Net Promoter Score (NPS) question and ask for qualitative insights about responses. Those qualitative insights allow you to address any low-hanging fruit.
You may also ask how connected members feel to your organization and to one another, establishing a connection benchmark.
Pro Tip: Be especially sensitive to yourself when reading open-ended community health score responses. They can be hard to read and not take personally, but the responses are not a reflection of you personally! So give yourself lots of space and rest before reading and then creating follow-up tasks.
One of my clients recently made a date with herself to read every single qualitative response to a benchmark survey - over a glass of wine. Then she followed up personally with everyone who gave harsh but legitimate criticism. She sets a high standard for the rest of us.
Survey Goal 2
Inspire New Content and Program Ideas
When members make no progress, they disengage. Surveys are great tools for understanding the progress your members want to make. Notice that is different from trying to understand what your members want. They may not know they want a specific program unless you can connect it to the progress it helps them make. When you understand these deeper desires, you can then design programs that spark engagement.
Start by creating a list of shared goals (e.g., getting support, getting feedback, learning a specific skill), and include an open option for a write-in.
With the answers, you can develop a list of the top challenges (which you can also rank, see Survey Goal 3). When you know challenges and have a clear community purpose and values, program creation becomes a natural process rather than forced and rigid.
For instance, if you learn from your survey that the top challenge members face is "scaling their work" and your values are personalization, authenticity, and diversity, you may then brainstorm for several minutes all of the ways you might gather people to address this challenge while honoring your values. You could, for instance, create an accountability group connecting people with different business backgrounds. Or you might create a member advisory board of diverse experts who can offer personalized mentoring to members while also giving them special access to your team. Let the ideas flow.
Pro Tip: If your survey software allows it, pipe in the answers from this question into a ranking question, as we will discuss for Goal 3.
Survey Goal 3
Rank Top Goals & Topics
While people's self-reported wants and desires may be unreliable, asking people to rank their goals or ideas can encourage them to be more thoughtful about their responses. So while you might have a long list of all the goals that members want to achieve together based on the questions for Survey Goal 2, you may not know how strongly members prioritize each goal. You especially may not understand how these goals vary across different demographics, like job titles, age, or industry. Running a cross-tabulation analysis will allow you to break down goals by demographics and psychographics and also see which goals are universal.
It can be a good idea to ask members to rank their goals and challenges, whom they want to meet, or the topics they most want to discuss. Equipped with this information, you can prioritize addressing challenges by order of ranking (this information often helps you decide how to develop channels, subgroups, event themes, or discussion categories).
Survey Goal 4
Gather a List of People
to Get More Involved
Surveys are not the end goal. Relationships are.
Per the advice of Erika Hall, user researcher and author of Just Enough Research (shout out to Sarah Judd Welch for bringing this book into my life!), I recommend that community builders primarily use surveys as screeners for further discussion.
Always end the survey by asking if people would be willing to talk further about their answers, and then ask for their email or other contact information. It is unlikely you will be able to talk to everyone who indicates they would like to talk further, so follow these next steps.
1. Follow up. For all those who said they would be willing to talk further, contact them to say (1) you appreciate their contribution and openness, and (2) you will follow up with them if they are selected.
2. Take your time deciding who to interview further. Get a diverse array of voices, choose them based on existing member personas and a set of criteria that indicates they would be a good fit for your community purpose and choose both advocates and detractors.
3. Schedule interviews in whatever format is most comfortable for them: some people like Zoom or a phone call (or IRL when possible) whereas others prefer email, IM, or even text. What you lose in candid responses from live interviews, you make up for in the ability to consider answers, ask follow-up questions, and gain intentional and considered answers via email.
You will likely want to interview a handful of people who did not complete the survey as well, but this initial group will give you a good sample of those who are committed enough to share their opinions. And in a community, commitment is key.
Pro Tip: If you have promised anonymity in your survey, be sure to indicate that, for those who provide their contact information here, their answers will not be connected to their identity. Then create a system to remove their identities from their answers so you never see them (usually you just download the data and replace all names with ID numbers).
6 Phases of Community Survey Research
1. Establish criteria for inclusion
Not everyone's opinions will matter. In fact, you want to be crystal clear about who you want to hear from before you send out surveys. There are a few common criteria to consider:
Do you want to hear from your entire community, or only a part of it (the most engaged, the least engaged)
Do members need to have taken some specific action before, or not?
Specific buyer or customer personas
If you are unsure about what criteria to create, ask yourself:
Who do you need to build deeper relationships with?
What voices are missing in your community today — and in your resulting strategy?
What is your primary goal for this survey, and what target groups will be most likely to help us reach that goal?
From there, establish clearly defined criteria and send the survey only to those members.
2. Formulate a few hypotheses.
For serious social science research, you need a theory-backed hypothesis for your research to be considered rigorous. Now, you don't work in that environment (most likely), but that does not mean you should be lazy and simply look for random patterns in data. That practice is known as p-hacking and it leads to biased (read: useless) insights.
You may not know academic theories to explain your work. And though I recommend digging in deeper (theories are far more helpful than random patterns! It is helpful to know them), you can at least think about your goals and what patterns you think will emerge from survey findings.
Don't forget to write down your hypotheses! It is easy to delude yourself into thinking that you guessed the answers to your questions correctly when you have no written proof of your hypotheses. The distance between your hypothesis and the results often also becomes a great story or aha! moment to share with others.
As you go through the next step, allow your hypotheses to inform your questions, refine them, and question your assumptions.
3. Write the survey
Now it's time to write the survey. Keep the survey as short as you can, ideally less than 10 questions. See below for tips I wish I had known when I began doing this work years ago.
Here is a starting point for simple, straightforward survey questions by goal:
If you want to ask demographics or psychographic questions:
Start out with “Which of the following best describes you/r [job role, caretaker status, geographic location, etc.]?”
If you plan to ask an NPS question, here is how you do it for a community:
How likely is it that you would recommend [community] membership to a friend or colleague? (0-10 Scale)
Is there anything else you would like to share? [Fill in]
If you want to ask about a sense of community, here is a brief version helpful for online communities:
How connected do you feel to other [community] members? (5-point Likert scale from very disconnected to very connected)
How connected do you feel to [our organization]? (5-point Likert scale from very disconnected to very connected)
Optional: Please explain why you selected these options.
Danielle Maveal has some great tips on health surveys too via her Substack newsletter - her version amends the Sense of Community Index (SCI) and adds two great qualitative questions: “If you could remove one element of the community, what would it be? If you could add something to the community, what would it be?”
If you want to ask about goals, topics, or other preferences:
Which of the following options best describe your [goals, topics, etc.] for [period of time]? (Select up to 3)
4. Send out the survey, diligently.
I recommend a window of no more than 5 business days to complete the survey, with 1-2 reminders in that timeframe. Ideally, leave it open over a weekend.
5. Analyze the results.
If you don't have statistical chops (most community managers don't, it's OK!), partner with a BI analyst in your company or hire a statistical analyst on a site like Upwork. Check references first; your mileage may vary. These people can help explain the concepts of statistical significance to you, help you understand where and when it is appropriate to segment data, and even help you develop insights from the data.
If neither of those is possible, use your survey software’s analysis tools. SurveyMonkey and Qualtrics both provide statistical significance for each question, even if you cannot take that further and do cross-tabulation analysis. Statistical significance is not telling you whether your answer is close to “the truth” or not, by the way, it’s just telling you the likelihood that a particular trend in your data is not happening by chance. Still, it’s a helpful start!
6. Present the insights.
Don't keep the insights to yourself! Share your methods and findings broadly in a deck that can be distributed around the organization.
Per the advice of community builder Adrian Speyer, heed Avinash Kaushik’s insistence that pies are for eating and not for making useless pie chart visualizations.
Final Tips for Writing Community Member Research Surveys
1. Trying to re-engage disengaged members is painful and hardly ever yields meaningful results. There are, however, two exceptions:
If you have redesigned the community and taken into consideration new value propositions, user journeys, and more, and want to let them know (and give insight on) the changes.
If your disengaged members do not even know about your community offering (they were invited in a haphazard way at one point), you can use the survey as a reminder and another point at which to listen to their insights.
2. You can use surveys as a crafty way to remind people what the benefits of your community are.
If you are doing research to figure out why people are not engaging, throwing in a question such as, "Which of our programs have you benefited from most recently?" and including a multiple-select choice of options alongside the option of, "I did not know about any of these," can help. I have seen clients gain hundreds of new participants after reminding them that the community has specific feature sets.
3. Only ask one question at a time.
For example, don't ask: "What are your thoughts and feelings about our monthly events program?" Thoughts and feelings are closely related, so you can see how easy it would be to dash off this question in a survey. However, most participants will answer one or the other question or give you otherwise untargeted, unhelpful information.
4. Don't create false binaries when asking a question.
For example, don't ask: "Are you joining the community because you love animals or because you are bored?" Neither answer may be true. A better way to rewrite that question would be a Y/N "Are you joining the community because you love animals? Y/N” or — far better — an open-ended or multiple-choice option with an "other" and fill-in selection.
Wrapping Up
If you ever feel overwhelmed when creating your survey, always come back to this: Focus. Focus ruthlessly. The more questions you ask, the less likely the data are to be reliable as people get fatigued.
Surveys may not be a panacea, but they can help you meet many important goals. Just make sure to clearly establish those goals at the outset. Then let the insights spark new ideas. You got this.
