Future of ID

Instructional Design Capability Lab

Module 3Design Practice

Learning Analytics and Evidence-Based Design

Use data to measure what is working, find gaps, and improve learning designs over time.

Module 3Estimated time: 20 minDesign Practice

Learning Analytics and Evidence-Based Design

Learner question

How can evidence help me make better instructional design decisions?

Use data to measure what is working, find gaps, and improve learning designs over time.

Takeaway: Evidence should guide decisions, not decorate reports.

Learning analytics is not reporting numbers for their own sake. It means using evidence to make better instructional design decisions when the data is incomplete, mixed, or uncertain. This module shows how to understand learner behavior, evaluate design decisions, and make improvements tied to performance and business outcomes.
+What You'll Be Able to DoView outcomes

By the end of this module, you should be able to:

  • Explain how learning analytics supports evidence-based design decisions.
  • Distinguish useful learning measures from vanity metrics.
  • Interpret dashboard signals in relation to learner outcomes and business goals.
  • Use analytics evidence to recommend targeted design iterations.
Layer 1

Theory

This layer shows how to use evidence to evaluate impact and improve the learning experiences you design over time.

+Capability Definition

Learning analytics involves collecting, interpreting, and applying data about your learners' activity, performance, behavior, and outcomes. For instructional designers, the goal is to use evidence to improve decisions about content, practice, support, assessment, and performance outcomes.

+Why It Matters

Modern L&D teams need to show whether learning experiences are working and where they should improve. Analytics helps designers identify friction, validate assumptions, prioritize revisions, and connect learning activity to measurable performance instead of relying only on completion rates or satisfaction scores.

+Measurement Strategy

A measurement strategy defines what evidence you need before you build the experience. Strong strategies connect business goals, performance expectations, learning objectives, practice, assessment, and post-launch data so you can make better design decisions.

+Dashboard Interpretation

Dashboards only matter when you know what decisions they support. Completion, time spent, quiz scores, drop-off points, practice attempts, confidence ratings, and performance metrics each show only part of what is happening. Your job is to compare multiple signals before deciding whether the design is working.

+Evidence-Based Iteration

Analytics is valuable only when it leads to better design decisions. The data may show that instructions need clarification, practice is too easy or too difficult, feedback lacks specificity, your learners need job support, or the assessment does not match the target performance.

+L&D Theory Bridge
  • Kirkpatrick

    Use Kirkpatrick to sort evidence across reaction, learning, behavior, and results. It reminds designers to look beyond whether learners liked the experience and toward whether behavior and outcomes changed.

  • Evaluation strategy

    An evaluation strategy defines what evidence you will collect, when you will collect it, and how you will use it to make decisions. It keeps analytics focused on decisions instead of producing disconnected reports.

  • Evidence-based iteration

    Use data to continuously improve your learning design. Data shows where your design supports your learners, where it creates friction, and which changes are most likely to improve performance.

+Evidence and Source Notes

These sources support the field definition and evaluation model; Bradley's design interpretation connects them to the module's decision sequence.

  • Current field definition — SoLAR (2025)

    The Society for Learning Analytics Research defines learning analytics as collecting, analyzing, interpreting, and communicating learner data to provide theoretically relevant, actionable insight that enhances learning and teaching. The 2025 definition supports the module's decision-focused use of analytics.

  • Established evaluation model — Kirkpatrick and Kirkpatrick (2006)

    Evaluating Training Programs: The Four Levels, 3rd edition, 2006, supports distinguishing reaction, learning, behavior, and results rather than treating one metric as proof of impact.

  • Bradley interpretation

    The Evidence → Interpretation → Decision → Measurement sequence is Bradley's professional synthesis for turning analytics into an instructional decision. It is a design model for this experience, not a universal research taxonomy.

Layer 2

Experience

This layer gives you practice reading analytics evidence and turning it into a focused design recommendation. Before recommending action, interpret the evidence, name what remains uncertain, and connect the decision to learner behavior or performance.

Dashboard Signal Review

Review a simple learning dashboard that includes completion, assessment performance, confidence, practice attempts, and drop-off patterns. Identify which signals are meaningful, which require more context, and which should not drive a design decision on their own.

Applied Decision-Making

Choose the most defensible design action based on the evidence. The decision should connect the data pattern to a learner need, instructional objective, or business outcome rather than defaulting to adding more content.

Feedback and Insight

Compare the recommendation against evidence-based design criteria. Strong recommendations explain what the data suggests, what remains uncertain, what design change is proposed, and how success would be measured after the change.

Situational mental model

From Evidence to Decision

  1. 01Evidence
  2. 02Interpretation
  3. 03Decision
  4. 04Measurement

Design practice

Choose a professional design decision, explain your reasoning, and use AI feedback as a structured critique. You can move in order or select the decision point most relevant to your work.

Scenario

You are reviewing analytics for a recently launched training program. Completion rates are high, and learners report positive feedback, but there has been no noticeable improvement in job performance. How would you interpret this data, and what would you do next?

0 / 1200 characters (20 minimum)

Layer 3

Reverse Build

Inspect the Build

You just experienced this as a learner. Now let's inspect it as an instructional designer.

Reverse Build reconstructs the learning experience, not the module topic or the code. It reveals the challenge you practiced, the thinking behind the design, how AI supported the work, and the trade-offs that shaped the experience.

The learning challenge
The decision you practiced
The AI-supported critique
Reflection and transfer

Reconstruct the Experience

Bradley's professional observation is that learning teams may have completion, score, or satisfaction data without a clear decision it supports. This module addresses that applied problem by moving learners from evidence, to interpretation, to decision, to measurement. It does not claim that every team has the same analytics maturity; it helps designers decide what evidence matters, what remains uncertain, and how a proposed change would be evaluated.

The Problem

What This Module Is Solving

What this experience was built to solve before any design choices were made.

Bradley's professional interpretation focuses on the gap that can exist between reporting and action. The experience moves learners away from treating dashboards as answers and toward the sequence of evidence, interpretation, decision, and measurement, without generalizing that gap to every learning team.

Instructional Foundations

Proven Practice vs New Approach

5 moves

How modern practice connects back to established instructional design and learning science.

Design Decisions

Key Design Decisions

5 moves

Why the experience was scoped, sequenced, simplified, or emphasized this way.

AI Collaboration

How AI Supported the Build

5 moves

Where AI accelerated exploration, variation, critique, or drafting without owning the decision.

AI supported the build, but it did not decide what the right answer was.

Design Trade-offs

Tradeoffs and Constraints

5 moves

Where the experience balances realism, clarity, complexity, guidance, autonomy, automation, and human review.

Design Trade-offs

Where This Could Break

5 moves

Where the experience balances realism, clarity, complexity, guidance, autonomy, automation, and human review.

This is not a replacement for a full measurement strategy. It is a way to make better decisions with the data you actually have.

Learning Science & Transfer

What Another Designer Can Reuse

5 moves

How this reasoning can transfer into future instructional design practice.

Learning Science & Transfer

Reflection Question

How this reasoning can transfer into future instructional design practice.

Think about a recent learning experience you worked on. What metric did you rely on most, and did it actually help you decide what to change? If not, what evidence would have been more useful?

Transfer to Your Practice

What might this way of thinking change in your own instructional design practice?

Reverse Build is not a correct-answer key. It is professional reasoning made visible so you can study it, question it, adapt it, and apply it. Professional judgment is learnable.

Reflection

Before revising a course, what learning design decision would you want evidence for, what evidence would you trust, what evidence would you still need, and what professional judgment remains yours as the instructional designer?

0 / 1200 characters (20 minimum)

Choose what you are curious about next.

Follow the next capability or return to the Capability Map. Learners may begin anywhere; there is no required sequence.

Explore Next Capability →Return to Capability Map