DYSARTISANIDEAS, FORGED INTO FORM.Search the episode library ↗

THE INNOVATION FORGE / From the Foundry

Understanding Engagement Scoring in Slate: define what the score means

Build an engagement rubric around clear behavior definitions, accessible opportunities, useful component scores and a dependable review process.

A lantern-lit drafting table with a brass compass, diagrams and a closed leather notebook.
Define the measure before using the score.

THE IDEA, IN A MINUTE

Decide what a point means before adding the points together.

In this From the Foundry reading of Chapter 1, David Dysart introduces engagement scoring and the choices behind it. Define what a score measures, identify noisy signals and plan a small, reviewable first rubric.

01 / THE RECORDING

Watch the episode.
Or find your moment.

Start with the question the score should help answer

Engagement scoring turns recorded interactions into a summary that staff can use. In Chapter 1 of The Innovation Forge, David introduces events, communications, portal activity and application progress as possible inputs. The useful starting point is the purpose of the summary, before choosing a point scale.

Write down the question a colleague will ask when they see it. Are they trying to understand a recent change, prepare a conversation or find an unanswered request? Identify what action the score could help inform and what other context that person needs.

A score describes the behaviors your system can observe. It is not a complete account of someone’s interest, circumstances or needs. Keep the underlying interactions available so a staff member can understand the pattern instead of treating the total as the explanation.

In the recording · 06:31 ↗

Define the behavior before counting it

An email attached to a student record is not necessarily an email from that student. The chapter specifically distinguishes student communication from counselor messages and documents, and notes that privacy protections can make engagement statistics noisy.

For every proposed input, specify who performed the action, what qualifies, which period it covers and where the evidence comes from. Decide whether the measure counts distinct events, repeated actions or the presence of an action at all. Keep automated or unrelated activity from quietly changing its meaning.

Test a few known examples before calculating the total: a student reply, an outbound message, a counselor attachment and an interaction with uncertain tracking. If two reviewers would count those differently, settle the definition first.

In the recording · 11:34 ↗

Check whether the opportunity was accessible

The chapter’s event example comes with an important qualification. A campus visit requires time and resources, so a rubric that heavily rewards visiting can reflect access to the opportunity as well as engagement. David points toward virtual events, recordings and other ways to experience the institution.

List the ways a person can take part and what each activity allows them to do. Review whether meaningful participation is recognized across those routes. Do not assume that attending in person always supplies better evidence of interest than participating remotely.

Apply the same care to portal activity. The reading notes that someone may complete requirements with little online interaction. A low count should prompt a look at the available context, including whether the person had a reason to return, rather than an automatic conclusion that they are disengaged.

In the recording · 10:34 ↗

Make the weights explainable and revisable

Assigning points involves a judgment about which behaviors matter. Chapter 1 recommends involving staff and other stakeholders, then informing the rubric with consistent historical data and an outcome that can be defined. A new activity may initially need a provisional judgment because no useful history exists yet.

Record the reason for each weight and the evidence supporting it. Separate a staff hypothesis from a pattern observed in prior data. Name the outcome and time period used in the comparison, and keep later activity from leaking into an earlier assessment.

Review a changed weight against several different behavior patterns. Look for an input that dominates the score through repetition or an apparent improvement caused by a recording change. A rubric needs a review date and an owner as much as it needs a calculation.

In the recording · 08:19 ↗

Keep the component scores visible

The chapter proposes component scores for areas such as events and communication alongside the overall score. Those parts help explain why two similar totals can represent different journeys.

Choose components that correspond to questions staff actually ask. Show which actions contributed and when they occurred. If one person has recent replies and another has older event activity, the same total should not erase that difference.

Treat missing information separately from an observed zero. A source that has not refreshed cannot establish that no activity occurred. Keeping that distinction visible makes the score easier to question and reduces the chance of acting on an incomplete picture.

In the recording · 13:31 ↗

Make refresh and review part of the design

Queries and source formats can support a scheduled scoring process, but the schedule should match the work that depends on it. Define when the inputs arrive, when the calculation runs and when staff can expect to see a current result.

Start with a small set of known examples and compare the stored results with the intended rubric. Record the version of the weights and the last successful refresh. Decide who investigates missing input or a surprising change, then revisit the score with the people using it. A dependable process keeps both the calculation and its interpretation open to review.

In the recording · 09:39 ↗

03 / A SMALL EXPERIMENT

Check three assumptions behind a score

Use these fictional cases to test what a rubric is counting, whose opportunities it recognizes and whether the result is current.

The rubric rewards only campus attendance

A fictional rubric gives substantial points for campus visits but none for an equivalent live virtual session. Staff use the total to choose whom to contact, even though some prospects cannot travel. What should they review?

Compare your reasoning

Compare the purpose and meaningful participation available in each route. Review whether the rubric is rewarding access to travel rather than the intended behavior, and include appropriate accessible ways to engage. Examine low scores with context instead of automatically withholding helpful outreach.

Revisit this moment · 10:34 ↗
The record contains more email than the student sent

A fictional communication count includes two student replies, three outbound staff messages and a counselor’s attached document. An email-open measure is also affected by privacy-protected loading. What must the input definition clarify?

Compare your reasoning

Define the actor, direction, qualifying interaction and observation period. Separate student replies from outbound messages and counselor material, and treat uncertain opens as a distinct noisy signal. Check the underlying examples before assigning points or interpreting the count as student effort.

Revisit this moment · 11:34 ↗
A score drops when an activity feed is late

A fictional event feed fails overnight. The scoring job runs on schedule and displays zero event points, which staff interpret as no participation. How should the process distinguish the problem?

Compare your reasoning

Check source freshness before calculating or presenting the result. Preserve the distinction between an observed zero and unavailable data, show the last successful update and notify the responsible owner. Test a normal refresh, a missing feed and recovery before relying on the score for follow-up.

Revisit this moment · 09:39 ↗
Build a test card to take with you

Notes stay in this tab. Reloading clears them.

04 / INSIDE THE STUDIO

The seams are part
of the story.

Behind the episode

The reading’s author notes put practical complications beside the scoring examples: access to campus visits, noisy email statistics and students who complete requirements with little portal activity. Those interruptions keep the meaning of the recorded behavior in view while the rubric takes shape.

Revisit the studio moment · 10:34 ↗

Art from the workshop

A lantern-lit table keeps plans, instruments and a notebook close at hand.

Open the artwork ↗

KEEP THE IDEA MOVING

Where would you take it next?

CONTINUE EXPLORING