Rullst AcademyRullst Academy/Lesson
AI: the curiosity lab · Lesson 03 / 06

The surprise test

Two models look good in training. Lumi saves new cards to discover which works better outside it.

Learning paths7 min

Let's discover together

  1. 1

    Training adjusts a model. Separate examples help evaluate what it does with new inputs.

  2. 2

    Memorizing examples can hide difficulties. This can signal overfitting: excellent training results, weak results on new data.

  3. 3

    Do not repeatedly tune the model using its final test. Four cards are a small demonstration, not proof of overall quality.

  4. 4

    Repeated cards on both sides can create a false impression of novelty. Separate examples before evaluation and compare against a simple rule. If results guide model selection or tuning, reserve other examples for the final check.

FROM LESSON TO PRACTICE

A guide to follow at your pace

Read the steps before using the reference below. Commands run in the indicated system’s terminal; signup happens on the official site. Rullst Academy exercises are practice and do not verify your installation or external account.

         training   unseen
model A    4/4       1/4
model B    3/4       3/4

Official links and next step

✦
YOUR MISSION

Compare correct predictions on new cards and calculate how many errors remain.

Now it's your turn

Discovery workshop

The surprise test

A got 4 of 4 training cards and 1 of 4 new cards right. B got 3 of 4 training cards and 3 of 4 new cards right. Compare only the new cards.
Build your answers and check when you are ready.
A moment to understand

Does perfect training guarantee future success?

I need a hint

Training and evaluation answer different questions.

Loading your journey…

Take it with you

Make four new examples and explain why they stayed out of training.

At your own pace

My files and GitHub

Every lesson can become your own record: code, notes or a practice plan. Saving a file to GitHub is optional and does not change your XP.

Student guide

The editor and downloads open when this lesson is available to your account. Check access below or return to the track map.

Checking your access to this lesson…

Rullstpetbot

AI can make mistakes. Check the lesson material.