Object Detection – Detect Objects in Scratch

🎯 Object Detection – Detect Objects in Scratch #

The Object Detection extension brings real AI-powered object recognition into Scratch.
It lets your project identify multiple objects from the live camera or Scratch stage, count them, read their position and size, and react when objects appear.
Because the models are already trained, students can start experimenting with computer vision immediately – no model training or external setup required.
Simple enough for beginners, but powerful enough for classroom AI, STEM, and interactive game projects. 🤖

 

---

🌟 Overview #

• Detect Multiple Objects: Recognize several supported objects in the same image or camera frame.
• Count What the AI Sees: Read the total number of detections or count only a specific object.
• Track Position and Size: Get the X/Y center, width, and height of detected objects in Scratch stage units.
• Filter What Matters: Detect everything the model recognizes, or create your own list of object names to focus on.
• Save and Sort Detections: Save a snapshot of selected detections and order them by confidence, position, or size.
• Trigger Events: Use hat blocks to react when an object appears or when the number of detections changes.
• Choose the Right Model: Switch between Fast, Balanced, and Wide detection options depending on your device, image detail, and the variety of objects you need.
• Choose the Input: Detect objects from the live camera or directly from the Scratch stage image.

---

✨ Key Features #

• Pre-trained, prompt-free object detection – no training step required.
• Detect many objects in one frame and work with each detection individually.
• Read object names, counts, confidence, position, width, and height.
• Choose between current live detections and a saved snapshot when building more advanced projects.
• Create a custom detection list to ignore objects that are not important to your project.
• Save and sort detections from left to right, top to bottom, largest to smallest, or highest to lowest confidence – or reverse any of these orders.
• Hat blocks make it easy to build object-controlled games and interactive scenes.
• Adjustable detection intervals help balance speed and device performance.
• Works fully in-browser – safe and private.

---

🚀 How to Use #

1. Go to: pishi.ai/play
2. Open the Extensions section.
3. Select the Object Detection extension.
4. Allow camera access when prompted and check that the preview appears.
5. If you prefer, use the Scratch stage as the input to detect objects in the stage image, including any camera video displayed there. For the best real-time performance, use Camera input, since processing the Stage may be slower.
6. The extension starts with Fast detection model. You can switch to Fast, Balanced, or Wide at any time to change the balance between speed, detail, and object coverage.
7. Try the “list of [current] detections” reporter block to see the object names the AI recognizes.
8. Then use counting, position, saved-detection, filtering, and event blocks to build your project.

Tips

• Use clear lighting and keep the object large enough to see easily in the frame.
• Start with one supported object before building a project that depends on several objects.
• Use the exact object label returned by the “list of [SOURCE] detections” reporter block when typing an object name into another block.
• The object vocabulary and detection labels are in English, so use English labels when entering object names.
• current means the latest live detection results. saved means a snapshot created with the save-and-order block.
• If the same kind of object appears more than once, use #1, #2, #3, and so on to work with individual detections.
• If detection feels slow on a device, try the Fast model or increase the classification interval.

---

🧱 Blocks and Functions #

 

📊 Detection Lists, Sources, and Counts #

Understanding SOURCE:

• current – the latest live detections from the camera or stage.
• saved – a snapshot stored by this block:
  save [OBJECT|text] from [SOURCE] detections, order by [ORDER_TYPE]

Using a saved snapshot is helpful when you want object order and positions to stay fixed while your project works with them.

 

Shared behavior: Reporter blocks that return a name, list, confidence, position, or size return an empty value when there is no matching detection, no saved snapshot is available, or the requested item number does not exist. Count reporters return 0, and Boolean blocks return false.

 

Reports the object labels in the selected detection source as comma-separated text.
Choose current to see what the AI recognizes right now, or saved to read the stored snapshot.
This is one of the best blocks for learning the exact object names the selected model uses in its detection results.

 

Reports the total number of objects in the selected source.
For example, it can count all live current detections or tell you how many objects are stored in saved detections.

 

Reports the special value all, which represents every detected object regardless of its label.
When you want a block to work with every detected object instead of one specific label, place it in an input of a block that asks for an object name (OBJECT).
You can also type all in English directly into the object-name input instead of using this reporter block.

 

Takes matching detections from the selected source, sorts them, and stores the result as the new saved snapshot.
This is especially useful when you need a predictable order such as leftmost first, largest first, or highest confidence first.

By default, the current detection list is already ordered from highest to lowest confidence.

OBJECT: enter an object label such as person, comma-separated text such as person, car, or use all.

• all: keeps all detections and saves them in the selected order. If you choose confidence high to low, the current detections are already in that order, so the block simply saves them as a snapshot.
• An object name, such as person: filters the detections to that object and then sorts the matching results.
• Multiple object names, such as person, car: combines detections matching any listed name and sorts them together.

SOURCE: choose current or saved.

• current: uses the latest live detection results.
• saved: uses the existing saved snapshot, allowing you to reorder it or filter it further.

ORDER_TYPE: order type can be:

• confidence high to low or confidence low to high
• left to right or right to left
• top to bottom or bottom to top
• large to small or small to large

After saving, choose saved in the SOURCE menu of other blocks to read the snapshot.
Saved detections stay unchanged until you save again, and switching the detection model clears the saved snapshot.

 

Reports the label of one detection by its position in the selected source.
INDEX: use 1 for the first item, 2 for the second, and so on.
If the item does not exist, the block returns empty.
For saved detections, item 1, item 2, and so on follow the order that was selected when the detections were saved.

 

Reports how many detections in the selected source match an object name.
OBJECT: enter an exact object label, comma-separated text, or use all to count every object in the source.
When multiple names are provided, the block returns their combined detection count. For example, person, car counts every detected person and car.
Use this for counting games, classroom experiments, or logic such as “continue when two people are visible.”

---

📍 Position, Size, and Detection Confidence #

Reports the confidence of a selected detection on a scale from 0 to 1.
For example, 0.87 means the model gave that detection a confidence score of about 87%.

OBJECT: enter an exact object label, or use all to select items from the whole source.

INDEX: selects the first, second, third, and so on matching item.

SOURCE: choose current or saved.

 

Reports the X position of the center of the selected detection box, mapped to Scratch stage coordinates.

Use it to move a sprite left or right with a real-world object.

Note: A detection box is the rectangular area around a detected object that shows where the object is located in the image.

 

Reports the Y position of the center of the selected detection box, mapped to Scratch stage coordinates.
Use it for vertical tracking, aiming, or object-controlled games.

 

Reports the width of the selected detection box in Scratch stage units.
A larger value means the detected object occupies more horizontal space in the image.

 

Reports the height of the selected detection box in Scratch stage units.

 

Returns true when the selected source contains at least one detection and false when it is empty.
With current, this checks the live scene. With saved, it checks the stored snapshot.

 

Checks whether a particular object and item number exists in the selected source.
For example, person #2 is true only when a second matching person detection exists.
You can also use all as the object value to test whether a particular item number exists in the whole source.

---

🎯 Detection Modes and Custom Lists #

Controls which detected objects are included in the results used by the extension’s other blocks and detection events.

MODE:

• all detections – keep every object recognized by the selected model.
• my detection list – keep only object names that you have added to your custom detection list.

Important: The my detection list is empty when the extension starts. You can switch to this mode at any time, but no detections will be reported until the list contains at least one object name.

 

Reports your custom object list as comma-separated text.
The list can also store a history of newly detected object names without duplicates. For example, when the detection count changes, add the list of current detections to my detection list. Adding a name that is already in the list, even with different capitalization, does not add it again.

 

Reports the object name at the selected position in your custom detection list.
INDEX: use 1 for the first item, 2 for the second, and so on.
If the item does not exist, the block returns empty.

 

Reports how many unique object names are in your custom detection list.
This counts configured names, not objects currently visible to the camera.

 

Adds an object label to the custom list.
Use the exact label returned by the model.
Comma-separated text adds each label separately.
Adding the same name again does not create a duplicate entry.

 

Removes the matching object name from your custom list.
Comma-separated text removes every matching label.

 

Removes every name from the custom detection list.

---

⚡ Hat Blocks and Reactions #

Runs the attached script when the stable number of any detected object increases.
It can trigger when the first object appears, when a new object label appears, or when an additional instance of an already visible object is detected.

 

Runs when the stable count of the named object increases.
For example, it can trigger when the first person appears and again if another person is newly detected.
Enter comma-separated text to run when the stable count of any listed object increases.
Using all here gives the same “when any object detected” behavior as the previous hat block.

 

Runs when the total stable number of currently detected objects changes, either up or down.
This is useful for counting games and scenes that react when objects enter or leave the camera view.

 

Note: detection hats wait for the same count to be seen in 2 consecutive frames before treating it as stable. This helps reduce one-frame flicker and accidental triggers.

---

🧠 Model and Confidence Controls #

Switches the active object detection model.
The extension starts with the Fast model. Students can switch between models depending on whether their project needs faster detection, more image detail, or support for a wider range of object types.

↔ Swipe left or right to view full table on mobile

 

Tip: Start with the default Fast model for responsive detection. Choose balanced for more image detail, or wide when you need many more object categories.

Important: switching models clears saved detections. The current minimum confidence setting is kept when switching models.

 

Sets the minimum confidence required before a model result is reported as a detection.

The setting uses a scale from 0 to 1:

• 0.35 – default.
• 0.5-0.7 – stricter; can reduce weak or false detections.
• 0.2-0.3 – more permissive; may help with difficult detections but can also increase mistakes.

This is a useful way to teach students that AI predictions are not simply “right” or “wrong” – each detection comes with a confidence score.

 

Reports the current minimum confidence setting on the 0-1 scale.

Remember: both “minimum confidence” and “confidence of [OBJECT] #[INDEX] in [SOURCE] detections” use the same 0-1 scale.

---

⚙️ Classification Controls #

• classify [INTERVAL] - Choose how often detection runs:
  • every time this block runs
  • continuous, without delay
  • continuous, every 50–2500 ms
• turn classification [on/off] - start or stop continuous detection.
• classification interval - reports the current interval in milliseconds.
• continuous classification - reports continuous detection is “on” or “off”.
• select input image [camera/stage] - choose camera or stage.
• input image - reports the active input source.

---

🎥 Video Controls #

• turn video [off/on/on-flipped]
  • on: shows the camera preview in a mirrored view (like a typical webcam or mirror).
  • on-flipped: shows the camera preview in a non-mirrored view — directions appear as in the real world.
  • off: turns off the camera preview. In stage input mode, detection continues to run.
• set video transparency to [TRANSPARENCY|text] — adjusts how visible the camera preview is:
  • 0: fully visible (solid image)
  • 100: fully transparent (invisible but active)
  Use a value from 0–100 to adjust the camera background’s opacity and help sprites appear clearly over the camera preview.
• select camera [CAMERA] — chooses among available cameras on your device. The dropdown lists all detected cameras, and the extension switches automatically to the one you select.

• show detection boxes [on/off]
  Turn the detection box overlay on or off. When it is on, boxes appear around each detected object so you can see what the model is tracking. This is useful for testing the camera, checking how well the model is working, and debugging a project in real time.

Detection boxes are enabled by default for this extension. Use the detection-box control when you want to show or hide the visual boxes around recognized objects.

---

🎓 Educational Uses #

• Introduce Computer Vision: Students can see how an AI model turns images into object labels and locations.
• Teach Confidence and Uncertainty: Compare predictions at different confidence thresholds and discuss why AI sometimes makes mistakes.
• Explore Coordinates: Connect detected X/Y positions to sprite movement and Scratch’s coordinate system.
• Explore Size: Use width and height to see how large an object appears in the image.
• Practice Lists and Indexing: Work with the first, second, or third matching object and compare multiple detections.
• Practice Sorting: Save detections and order them by position, size, or confidence.
• Teach Events: Use hat blocks to show how software can react to changes in the environment.
• Discuss AI Limitations: Compare what students see with what the model recognizes and investigate false positives, missed objects, lighting, angle, and object size.

---

🎮 Example Projects #

• Object Greeter: Make a sprite say hello when a chosen supported object appears.
• Object Counter: Show the number of visible objects and trigger a celebration when the count reaches a target.
• Follow the Target: Use the X/Y center of a detected object to move a sprite toward it.
• Left or Right? Save matching detections from left to right and use item 1 to find the leftmost object.
• Biggest Object Challenge: Save detections from large to small and make Scratch react to the largest one.
• AI Confidence Lab: Display confidence values and test how lighting, distance, angle, and background affect predictions.
• Classroom Object Hunt: Give students a list of supported objects and score a point when the AI finds each one.
• What’s in Frame? Display the “list of [SOURCE] detections” reporter block so students can observe how the model describes a changing scene.

🧩 Try it yourself: pishi.ai/play

 

---

🔧 Tips and Troubleshooting #

• No camera?
  • Make sure your camera is connected and browser permission is allowed.
  • If the camera is blocked, enable it in your browser’s site settings and reload the page.
  • When the extension loads, if no camera is detected, the input image automatically switches to the Scratch stage so you can continue using and testing the extension's features.
• No detection?
  • continuous classification: Use this reporter to see if classification is active.
  • If it is active, improve lighting and face the camera directly.
  • turn classification [on]: Use this block, if classification is not active, then recheck the classification status with the above reporter.
  • In camera input mode, when the camera is turned off, classification is also stopped - you must turn the video back on or switch input to stage.
  • In stage input mode, the system classifies whatever is visible on the stage - backdrops, sprites, or images. You can turn off the video completely and still process stage images.
  • Stage mode is slower than camera input, so lower your classification interval (e.g., 100–250 ms) for smoother results using this block: classify [INTERVAL]
  • In stage mode, “left” and “right” landmarks are swapped because the stage image is not mirrored - coordinate space represents a real (non-mirrored) view.
  • Classification can also restart automatically when you use blocks such as:
  turn video [on] / classify [INTERVAL] / select camera [CAMERA] / select input image [camera/stage].
• Flipped view?
  turn video [on-flipped]: Use this to show the camera without mirroring. “on” mirrors like a selfie; “on-flipped” shows real left/right orientation.
• Laggy or slow?
  Use classification intervals between 100–250 ms or close other browser tabs to reduce processing load.
• WebGL2 warning?
  Try Firefox, or a newer device that supports WebGL2 graphics acceleration.
• Analyze stage instead of camera?
  select input image [stage]: Use this to analyze the Scratch stage image instead of a live camera feed. The stage can include the camera video, as well as any sprites and backdrops currently visible on the Scratch stage.
  Note: Processing the Scratch stage may be slower than using the camera directly.

🎯 Object Detection Specific Tips #

• No objects detected? Make sure classification is running, improve the lighting, move the object closer, and check that the selected model supports that object label.
• The object has the wrong name? Each detection model uses its own learned object vocabulary. Check the “list of [current] detections” block to see the exact label it returns.
• Too many false detections? Increase minimum confidence gradually, for example from 0.35 to 0.5.
• A real object is often missed? Try better lighting, a clearer angle, the Balanced model, or a slightly lower confidence threshold.
• Need an object that Fast or Balanced does not recognize? Try the Wide model, which uses a much larger class vocabulary.
• Only interested in a few objects? Add those exact labels to my detection list, then switch “look for [MODE]” to my detection list.
• My detection list mode suddenly detects nothing? Check that my detection list is not empty and that its names match the model labels exactly.
• Item 2 or item 3 returns empty? That item number only works when enough matching detections currently exist.
• Saved positions seem old? Saved detections are intentionally a snapshot. Run “save [OBJECT] from [SOURCE] detections, order by [ORDER_TYPE]” again, usually with current as the source, to refresh them.
• Project is slow? Use the Fast model, increase the classification interval, close other heavy browser tabs, or reduce how often your Scratch scripts read detection data.
• Camera is not available? Switch the input to stage and test with a backdrop, sprite, or imported image.

---

🔒 Privacy and Safety #

• Everything runs locally in your browser.
• No images or video are uploaded anywhere.
• Model files are loaded directly from Pishi.ai when the extension starts, so no third-party model servers are used.
• Always ask a teacher or parent before using the camera.
• Anytime, you can safely turn video [off].

---

🧪 Technical Info #

• Model Type: Pre-trained ONNX object detection models.
• Framework: onnxruntime-web.
• Browser Execution: Desktop browsers prefer WebGPU when supported, with WebAssembly (WASM) fallback; Android uses WASM.
• Model Options: Fast (320×320, 365 classes), Balanced (640×640, 365 classes), and Wide (640×640, 4,585 classes).
• Starting Model: Fast (320×320) on every supported device.
• Default Minimum Confidence: 0.35.
• Detection: Multiple objects can be reported from the same frame.
• Confidence Reporter: Returns a value from 0-1.
• Coordinates: Detection-box centers are mapped to Scratch stage coordinates; width and height are reported in stage units.
• Sources: Reporter blocks can read either current live detections or saved snapshot detections.
• Filtering: Detection mode can keep all detections or only labels from the user’s custom detection list.
• Saved Detections: A snapshot can be filtered and ordered by confidence, horizontal position, vertical position, or size.
• Hat Stability: Detection count changes must remain the same for 2 consecutive frames before object-detection hats are treated as stable.
• Detection Boxes: Visual detection boxes are enabled by default and can be controlled from the extension.
• Inputs: Live camera or Scratch stage image.
• Processing: Detection runs locally in the browser after the required model files are loaded.

---

🔗 Related Extensions #

• 🖐️ HandPose – detect hand landmarks and gestures.
• 🕺 PoseNet – track full-body pose and movement.
• 😎 FaceMesh – detect facial landmarks and expressions.
• 🖼️ Image Trainer – train your own custom image classifier directly in Scratch.
• 🏫 Google Teachable Machine – import pre-trained custom classification models.

---

🖼️ Object Detection vs. Image Trainer #

↔ Swipe left or right to view full table on mobile

 

💡 Why Choose Object Detection? #

Choose Object Detection when your project needs to find real objects and know where they are.
It is especially useful when students want immediate results without training a model first.

• Recognize several objects in the same scene.
• Count objects and compare their positions or sizes.
• Build projects that react to changes in the real world.
• Explore confidence, filtering, sorting, and computer vision in a Scratch-friendly way.

💡 When Should You Use Image Trainer Instead? #

Use Image Trainer when you need categories that are specific to your own project – for example, your own hand gestures, classroom cards, custom commands, or objects that a pre-trained detector does not know.
Image Trainer lets students create the labels themselves and learn directly from their own examples.

In short:
Use Object Detection to find and locate objects the AI already knows.
Use Image Trainer when you want to teach the AI your own visual categories.

---

📚 Learn More #

• AI Extensions Help at pishi.ai
• ONNX Runtime Web Documentation
• Scratch Ideas

---