Scoring Member Engagement

Mei-Lin Harbaugh, Director of Data and Analytics at the Council of Early Childhood Educators, walks host Amara Obiledu through the engagement-scoring system CECE built to track its 11,000 members, what signals actually predict a lapse, and where a model like this can mislead you if you trust it too much.

• Mei-Lin Harbaugh — Director of Data and Analytics, Council of Early Childhood Educators. Teaches engagement scoring in NANA's Learning catalog.

• 00:00 Intro

• 01:30 Why CECE built an engagement score

• 06:45 What signals actually go into the model

• 12:20 Weighting event attendance versus digital activity

• 17:50 A case where the score was wrong

• 22:40 Rolling it out to 37 staff who didn't build it

• 27:15 What she teaches NANA members about doing this themselves

• 31:00 Who at CECE actually sees the score

• 33:40 What it revealed about program investment

• 36:10 Is scoring becoming standard practice

• 38:20 The minimum viable version

• 39:40 Outro

Host: Mei-Lin, CECE has 11,000 individual members and a staff of 37, which is a lot bigger than most of the associations we talk about on this show. Why did you build an engagement score at that scale?

Mei-Lin Harbaugh: At 11,000 members, you cannot know your membership the way a 200-member association's staff can — nobody at CECE has met most of our members personally, and a gut sense of who's at risk simply doesn't scale. We needed something quantitative that could flag risk across the whole roster, because relying on individual staff intuition would mean most of our members never get looked at at all.

Host: What signals did you end up including?

Mei-Lin Harbaugh: Four categories, weighted differently. Event attendance — conference, regional workshops, webinars. Digital activity — logins, resource downloads, whether they've opened our email in the last 90 days. Credentialing activity, since a lot of our members are pursuing continuing-education hours toward state licensure requirements, and that's a strong loyalty signal. And tenure, because a member in year one behaves very differently from a member in year eight, and the model needs to know which one it's looking at.

Host: How did you decide the weighting between those?

Mei-Lin Harbaugh: Mostly by testing against members we already knew had lapsed in prior years, and seeing which signals would have flagged them earliest. Event attendance turned out to carry the most predictive weight, which surprised us a little — we expected digital activity to dominate, since that's the easiest data to collect. But someone who stops attending even one regional workshop a year is a stronger early signal than someone whose email opens dip slightly, probably because event attendance requires more deliberate effort to sustain, so dropping it means something.

Host: Tell me about a time the score got it wrong.

Mei-Lin Harbaugh: We flagged a cohort of members as high-risk two years ago because their digital activity had dropped sharply over a few months. When staff actually called some of them, it turned out a state licensing board had changed its continuing-education portal in a way that meant our members weren't logging into our system to track hours anymore — they'd moved that specific activity elsewhere, but they were still fully engaged members otherwise. The model read an external system change as disengagement. That's the risk with any score built entirely on behavioral proxies — it can't tell the difference between someone leaving and the world around them changing.

Host: How did you fix that, or did you just have to live with it?

Mei-Lin Harbaugh: We added a manual override layer — staff can flag a member as "score exception, reason noted" so the automated score doesn't keep re-flagging someone we've already confirmed is fine. It's not elegant, but it's honest about the model's limits, and I'd rather have an ugly workaround than a model people stop trusting because it cried wolf once too often.

Host: You mentioned rolling this out to staff who didn't build it. How did that go?

Mei-Lin Harbaugh: Slower than I expected. A membership team used to working from instinct doesn't automatically trust a number a data team handed them, and honestly they shouldn't, without understanding roughly what's in it. We spent real time just walking staff through what the score does and doesn't mean before we asked anyone to act on it, and I think skipping that step is the single most common way these projects fail — not the model itself, but nobody on the front line believing it.

Host: You now teach a course on this for NANA. What's the one thing you tell other associations before they start building something like this?

Mei-Lin Harbaugh: Don't build the model before you decide what you'll actually do with a flagged member. If the answer is "nothing changes, we just now have a number," it's not worth building. The value isn't the score — it's the specific outreach that happens because of the score, and if your organization doesn't have the staff capacity to act on flags, a scoring model just becomes an expensive way of feeling informed without being any more effective.

Host: Does CECE's chief membership officer, Yolanda Pruitt-Ames, use the score directly herself, or does it stay within your data team?

Mei-Lin Harbaugh: She sees the aggregate picture — how many members sit in each risk band, trends over time — but the individual member-level score is used operationally by the membership team, not by her directly. That split was deliberate. Yolanda's job is deciding whether the overall approach is working and what resources to put behind outreach; she doesn't need, and honestly doesn't want, to be looking at ten thousand individual scores. The membership team needs exactly that granularity to do their jobs day to day.

Host: Has building this changed how CECE thinks about non-dues revenue or program design more broadly, beyond just retention specifically?

Mei-Lin Harbaugh: It has, in a way I didn't fully expect going in. Once you can see which programs correlate with high engagement scores over time, it becomes a genuine input into which programs to invest more in, not just which members to call. Credentialing activity turned out to be one of our strongest engagement signals, which reinforced a decision the board had already been leaning toward — expanding our continuing-education offerings rather than treating them as a smaller side program.

Host: Last question. Is 11,000 members the threshold where this kind of thing becomes necessary, in your view, or does it apply at smaller scale too?

Mei-Lin Harbaugh: It applies well below 11,000 — I'd say anywhere staff capacity to personally track every member has already been exceeded, which for a lot of associations is a lot smaller than 11,000. The mechanics scale down fine. What doesn't scale down is the data team to build and maintain it, so a smaller association usually needs a simpler version, fewer signals, less automation, but the underlying logic holds at almost any size.

Host: Do you see this kind of scoring becoming standard practice across associations, the way benchmarking surveys already are?

Mei-Lin Harbaugh: I think it's heading that direction, slowly. When I started teaching this course through NANA's Learning catalog, most of the people in the room were larger associations like ours, just starting to think about it. The mix has shifted over the sessions I've taught since — smaller association staff showing up too, asking how to do a version that fits a team of three or four. That tells me the appetite is there even where the resources to build a full model aren't, which is exactly why I spend real course time on the stripped-down version.

Host: If someone in your course wants to build the simplest possible version of this, what's the minimum viable starting point?

Mei-Lin Harbaugh: Two signals, honestly — event attendance and whether they've logged into whatever system holds your member records in the last 90 days. You don't need credentialing data or a full behavioral model to start flagging the members who've gone quiet. I tell people in the course to build the crude version first, get staff comfortable acting on it, and only add complexity once that habit exists. Most associations that fail at this try to build the sophisticated version first and never get past the build phase.

Host: Mei-Lin, thanks for the walkthrough, warts and all.

Mei-Lin Harbaugh: The warts are the useful part. Anyone who tells you their model never gets it wrong hasn't looked closely enough, or hasn't been running it long enough to find where it breaks.