Data, probability and statistics I · Lesson 11 / 12

Before celebrating a model, compare

A fictional alarm gets 90% of ten test cases right. That sounds good until a rule that always says “normal” also gets 90% — while missing the only failure.

Let's discover together

1
There are nine normal cases and one failure. Always normal has nine correct predictions, one missed failure and zero false alarms. Accuracy is correct/total; here, 9/10 = 90%. Alone, it hides which case was missed.
2
The second rule detects the failure but also flags a normal case. It still has nine correct predictions: eight normal cases plus the failure. Now there are zero missed failures and one false alarm. Same percentage, different consequences.
3
The comparison panel below calculates these results; it does not train AI or evaluate real equipment. The choice depends on the goal and error costs. Ten invented cases do not demonstrate performance beyond this exercise.
4
Define the metric and baseline before comparing. In an AI project, examples used to tune a solution should not be its only final evaluation. Holding out cases and reporting errors by group helps reveal results hidden by an average.

Evidence lab

Experiment with fictional deliveries at Aurora station. Analysis runs in this browser. Nothing entered here is sent to the server.

The workshop resets when you leave or reload. Download data and report to keep your work, or send the report to the activity editor, which keeps a draft in this browser. Completion and XP belong to the account.

Example to investigate

Input data (simple CSV)

Header id,group,minutes; one delivery per row; unique id; groups A/B; decimal point; duration 0–1440 with up to two decimal places. Blank means unknown. Up to 100 rows. No quotes or decimal commas. Use invented data.

Record selectionSample sizeSeed (0–999999)

Selection uses the entered table, including missing values. A draw does not guarantee representativeness. The same seed reproduces selection; it cannot recover cases outside the table.

Make a prediction and analyze the table.

Selection results

Selection results
Group	Records	Known	Missing	Mean (min)	Median (min)	Range (min)	Within 5 min / known

Summaries rounded to up to two decimal places. Percentages use known durations; missing values remain outside that denominator. This does not remove possible missingness bias.

Individual durations (minutes)

Bars begin at zero and show up to 20 known durations from the selection. The table below contains all selected records, including missing values.

Inspect selected records

Inspect selected records
id	Group	Duration (min)

Virtual bag: model with replacement

There are four equally likely tokens; choose how many are blue. Each token returns before the next draw. This is a deterministic simulation, without prizes, money or real observations.

Blue tokens out of four (0–4)Draws (1–1000)Seed (0–999999)

Repeating the seed repeats the sequence. More draws do not guarantee a closer result in every run and do not validate the real-world model.

Two rules, the same accuracy

Ten fictional cases: nine normal and one failure. Rules defined for this example; no AI training or real equipment testing. Same number correct, different consequences.

Two rules, the same accuracy
Rule	Correct / total	Missed failures	False alarms

Your investigation report

Question, population, unit and predictionInterpretation and comparisonLimitation, new scenario, help and review

All recordsAnalyze selectionAlways normalBlue tokens out of four (0–4)Challenge: other routesData or selection changed. Analyze again before generating the report.Individual durations (minutes)Bars begin at zero and show up to 20 known durations from the selection. The table below contains all selected records, including missing values.Report sent to editor. Check and personalize it before saving or publishing.Correct / totalDownload input CSVInput data (simple CSV)Replace the current editor text with this report? Download any draft you want to keep first.Check the table or settings:Choose 0–4 blue tokens.Each row needs three comma-separated fields.Duplicate id. Check case identifiers.Include at least one record.Group must be A or B.Use header id,group,minutes.Id: 1–16 letters, digits, hyphens or underscores.Choose a valid selection method.Use at most 100 records.Sample size must be from 1 to the total number of records, without repeats.Seed must be an integer from 0 to 999999.Text must be at most 16000 characters.Choose 1–1000 draws.Use duration 0–1440, up to two decimal places with a dot, or leave blank.Example to investigateFalse alarmsFirst rowsHeader id,group,minutes; one delivery per row; unique id; groups A/B; decimal point; duration 0–1440 with up to two decimal places. Blank means unknown. Up to 100 rows. No quotes or decimal commas. Use invented data.GroupRecords in the input tableInterpretation and comparisonExperiment with fictional deliveries at Aurora station. Analysis runs in this browser. Nothing entered here is sent to the server.KnownLimitation, new scenario, help and reviewLineLoad exampleMean (min)Median (min)Record selectionDuration (min)Missed failuresMissingMissions: data and missingnessNo known dataObserved blue frequencyOpen index.html in a browser; all calculations work offline. Source links and your Rullst Academy account need a connection. The workshop does not automatically save edits: download CSV and report before closing.Analysis updated. Check units, missing values and limits.Within 5 min / knownOptional, if Node.js is already installed: run the tests below in the extracted folder. Node is not needed to use the workshop. Tests check calculations and technical limits, not student learning.Extreme valueQuestion, population, unit and predictionPseudorandom, without replacementRange (min)Make a prediction and analyze the table.Loading the example will replace the current table. Continue?Download Markdown reportThis report exceeds the editor limit. Download the complete report and prepare a shorter version to publish.Study report using fictional data. Selection and calculation do not establish causality, generalization or professional competence. The workshop does not verify human review; record it if pending.Your investigation reportSelection resultsSummaries rounded to up to two decimal places. Percentages use known durations; missing values remain outside that denominator. This does not remove possible missingness bias.Inspect selected recordsRuleTen fictional cases: nine normal and one failure. Rules defined for this example; no AI training or real equipment testing. Same number correct, different consequences.Two rules, the same accuracySample sizeThe workshop resets when you leave or reload. Download data and report to keep your work, or send the report to the activity editor, which keeps a draft in this browser. Completion and XP belong to the account.Second ruleSeed (0–999999)Selected idsSelection uses the entered table, including missing values. A draw does not guarantee representativeness. The same seed reproduces selection; it cannot recover cases outside the table.Settings changed; run the simulation to update.Repeating the seed repeats the sequence. More draws do not guarantee a closer result in every run and do not validate the real-world model.There are four equally likely tokens; choose how many are blue. Each token returns before the next draw. This is a deterministic simulation, without prizes, money or real observations.Virtual bag: model with replacementSimulate and compareSame mean, different distributionTheoretical probabilityEvidence labSend report to editorRecordsDraws (1–1000)Not filled in yetUnknown# My investigation ## Before calculating - Question, population, unit and prediction: - Fictional data, source and selection method: ## Check and compare - Missing data, duplicates and recorded decisions: - Manual calculation and workshop check: - Center, spread, proportion and representation: - Original variation and a case that changes my conclusion: ## Interpret and review - What the data supports: - What it does not yet support: - Simple baseline comparison, when relevant: - Feedback received or external review pending: - Help from people/AI and my own checks: - After a break, another table and explanation without copying: Download CSV and report. The editor draft is not an automatic copy of the workshop. GitHub is optional; do not publish personal data.

Official links and next step

Google · classification metrics

✦

YOUR MISSION

Compare a solution with a simple baseline and inspect error types.

Take it with you

Invent 12 cases with two failures and compare Always normal with your own rule. Count correct predictions, missed failures and false alarms. Explain which error would cost more in this fictional setting and what data is still missing.

At your own pace

My files and GitHub

Every lesson can become your own record: code, notes or a practice plan. Saving a file to GitHub is optional and does not change your XP.

Student guide

The editor and downloads open when this lesson is available to your account. Check access below or return to the track map.

Checking your access to this lesson…

Your path, step by step

Download activity materials

Download the lesson example and a workbook for your notes. The ZIP includes a guide to organizing activities in one or more repositories. Publishing is optional; eligible accounts can also use the GitHub upload button.

Your file for this lesson

Edit the example or record what you learned. This space does not run code or grade answers. The draft is saved only in this browser, separately per account; you choose when to download or upload to GitHub.

The example may be only a snippet or outline. Changing the extension does not make it a complete program. The activity ZIP contains starting material; the edited-file button downloads your text.

My code or notesFile format

Your text will be saved in this browser when you edit it. On a shared computer, clear your drafts when finished.

Direct upload accepts up to 8 KiB of text in one file. Use downloads and the manual guide for projects with multiple files.