物体检测——在 Scratch 中检测物体

🎯 物体检测 - 在 Scratch 中检测物体 #

物体检测扩展将真正由 AI 驱动的物体识别功能带入 Scratch。
它可以让你的项目从摄像头实时画面或 Scratch 舞台中识别多个物体，对它们进行计数，读取它们的位置和大小，并在物体出现时作出响应。
由于模型已经预先训练完成，学生可以立即开始探索计算机视觉，无需训练模型或进行任何外部设置。
它简单易用，适合初学者，同时也足够强大，可用于课堂 AI、STEM 和互动游戏项目。🤖

 

---

🌟 概述 #

• 检测多个物体： 在同一张图像或同一摄像头画面中识别多个受支持的物体。
• 统计 AI 看到的物体： 获取检测结果总数，或只统计某一种物体的数量。
• 追踪位置和大小： 以 Scratch 舞台单位获取检测物体的中心 X/Y 坐标、宽度和高度。
• 筛选重要物体： 检测模型能够识别的所有物体，或创建自己的物体名称列表，让检测专注于指定物体。
• 保存并排序检测结果： 保存所选检测结果的快照，并按置信度、位置或大小进行排序。
• 触发事件： 使用帽形积木，在物体出现或检测数量发生变化时作出响应。
• 选择合适的模型： 根据你的设备、图像细节以及所需识别物体的种类，在快速、平衡和宽域检测选项之间切换。
• 选择输入源： 从摄像头实时画面或直接从 Scratch 舞台图像中检测物体。

---

✨ 主要特点 #

• 使用预训练模型检测物体，无需提示词，也无需任何训练步骤。
• 在同一画面中检测多个物体，并分别处理每个检测结果。
• 获取物体的名称、数量、置信度、位置、宽度和高度。
• 在构建更高级的项目时，可以选择使用当前的实时检测结果或已保存的快照。
• 创建自定义检测列表，忽略对项目不重要的物体。
• 保存检测结果，并按从左到右、从上到下、从大到小或置信度从高到低的顺序进行排列；也可以反转其中任意一种顺序。
• 帽形积木让创建物体控制的游戏和互动场景变得更加容易。
• 可调节的检测间隔有助于在检测速度与设备性能之间取得平衡。
• 完全在浏览器中运行 — 安全且保护隐私。.

---

🚀 如何使用 #

1. 前往： pishi.ai/play
2. 打开扩展程序部分。
3. 选择“物体检测”扩展。
4. 出现提示时允许访问摄像头，并确认摄像头预览已经显示。
5. 如果愿意，你也可以将 Scratch 舞台用作输入源，检测舞台图像中的物体，包括舞台上显示的摄像头视频。为了获得最佳实时性能，建议使用摄像头输入，因为处理舞台图像可能会更慢。
6. 扩展启动时使用“快速”检测模型。你可以随时在“快速”“平衡”和“宽域”模型之间切换，以调整速度、图像细节和物体覆盖范围之间的平衡。
7. 试试“[当前] 检测列表”报告积木，查看 AI 能够识别的物体名称。
8. 然后使用计数、位置、已保存检测结果、筛选和事件积木来构建你的项目。

尖端

• 保持光线充足，并让物体在画面中足够大，以便清楚识别。
• 在构建依赖多个物体的项目之前，先从一个受支持的物体开始。
• 在其他积木中输入物体名称时，请使用“[SOURCE] 检测列表”报告积木返回的准确物体标签。
• 物体词汇和检测标签均为英文，因此输入物体名称时，请使用英文标签。
• “当前”表示最新的实时检测结果。“已保存”表示使用保存并排序积木创建的快照。
• 如果同一种物体出现多次，请使用 #1、#2、#3 等编号分别处理各个检测结果。
• 如果设备上的检测速度较慢，请尝试使用“快速”模型，或增大分类间隔。

---

🧱 积木和函数 #

 

📊 检测列表、来源和数量 #

Understanding SOURCE:

• current – the latest live detections from the camera or stage.
• saved – a snapshot stored by this block:
  save [OBJECT|text] from [SOURCE] detections, order by [ORDER_TYPE]

Using a saved snapshot is helpful when you want object order and positions to stay fixed while your project works with them.

 

Shared behavior: Reporter blocks that return a name, list, confidence, position, or size return an empty value when there is no matching detection, no saved snapshot is available, or the requested item number does not exist. Count reporters return 0, and Boolean blocks return false.

 

Reports the object labels in the selected detection source as comma-separated text.
Choose current to see what the AI recognizes right now, or saved to read the stored snapshot.
This is one of the best blocks for learning the exact object names the selected model uses in its detection results.

 

Reports the total number of objects in the selected source.
For example, it can count all live current detections or tell you how many objects are stored in saved detections.

 

Reports the special value all, which represents every detected object regardless of its label.
When you want a block to work with every detected object instead of one specific label, place it in an input of a block that asks for an object name (OBJECT).
You can also type all in English directly into the object-name input instead of using this reporter block.

 

Takes matching detections from the selected source, sorts them, and stores the result as the new saved snapshot.
This is especially useful when you need a predictable order such as leftmost first, largest first, or highest confidence first.

By default, the current detection list is already ordered from highest to lowest confidence.

OBJECT: enter an object label such as person, comma-separated text such as person, car, or use all.

• all: keeps all detections and saves them in the selected order. If you choose confidence high to low, the current detections are already in that order, so the block simply saves them as a snapshot.
• An object name, such as person: filters the detections to that object and then sorts the matching results.
• Multiple object names, such as person, car: combines detections matching any listed name and sorts them together.

SOURCE: choose current or saved.

• current: uses the latest live detection results.
• saved: uses the existing saved snapshot, allowing you to reorder it or filter it further.

ORDER_TYPE: order type can be:

• confidence high to low or confidence low to high
• left to right or right to left
• top to bottom or bottom to top
• large to small or small to large

After saving, choose saved in the SOURCE menu of other blocks to read the snapshot.
Saved detections stay unchanged until you save again, and switching the detection model clears the saved snapshot.

 

Reports the label of one detection by its position in the selected source.
INDEX: use 1 for the first item, 2 for the second, and so on.
If the item does not exist, the block returns empty.
For saved detections, item 1, item 2, and so on follow the order that was selected when the detections were saved.

 

Reports how many detections in the selected source match an object name.
OBJECT: enter an exact object label, comma-separated text, or use all to count every object in the source.
When multiple names are provided, the block returns their combined detection count. For example, person, car counts every detected person and car.
Use this for counting games, classroom experiments, or logic such as “continue when two people are visible.”

---

📍 位置、大小和检测置信度 #

Reports the confidence of a selected detection on a scale from 0 to 1.
For example, 0.87 means the model gave that detection a confidence score of about 87%.

OBJECT: enter an exact object label, or use all to select items from the whole source.

INDEX: selects the first, second, third, and so on matching item.

SOURCE: choose current or saved.

 

Reports the X position of the center of the selected detection box, mapped to Scratch stage coordinates.

Use it to move a sprite left or right with a real-world object.

注意： A detection box is the rectangular area around a detected object that shows where the object is located in the image.

 

Reports the Y position of the center of the selected detection box, mapped to Scratch stage coordinates.
Use it for vertical tracking, aiming, or object-controlled games.

 

Reports the width of the selected detection box in Scratch stage units.
A larger value means the detected object occupies more horizontal space in the image.

 

Reports the height of the selected detection box in Scratch stage units.

 

Returns true when the selected source contains at least one detection and false when it is empty.
With current, this checks the live scene. With saved, it checks the stored snapshot.

 

Checks whether a particular object and item number exists in the selected source.
For example, person #2 is true only when a second matching person detection exists.
You can also use all as the object value to test whether a particular item number exists in the whole source.

---

🎯 检测模式和自定义列表 #

Controls which detected objects are included in the results used by the extension’s other blocks and detection events.

MODE:

• all detections – keep every object recognized by the selected model.
• my detection list – keep only object names that you have added to your custom detection list.

重要提示： The my detection list is empty when the extension starts. You can switch to this mode at any time, but no detections will be reported until the list contains at least one object name.

 

Reports your custom object list as comma-separated text.
The list can also store a history of newly detected object names without duplicates. For example, when the detection count changes, add the list of current detections to my detection list. Adding a name that is already in the list, even with different capitalization, does not add it again.

 

Reports the object name at the selected position in your custom detection list.
INDEX: use 1 for the first item, 2 for the second, and so on.
If the item does not exist, the block returns empty.

 

Reports how many unique object names are in your custom detection list.
This counts configured names, not objects currently visible to the camera.

 

Adds an object label to the custom list.
Use the exact label returned by the model.
Comma-separated text adds each label separately.
Adding the same name again does not create a duplicate entry.

 

Removes the matching object name from your custom list.
Comma-separated text removes every matching label.

 

Removes every name from the custom detection list.

---

⚡ 帽形积木和响应 #

Runs the attached script when the stable number of any detected object increases.
It can trigger when the first object appears, when a new object label appears, or when an additional instance of an already visible object is detected.

 

Runs when the stable count of the named object increases.
For example, it can trigger when the first person appears and again if another person is newly detected.
Enter comma-separated text to run when the stable count of any listed object increases.
Using all here gives the same “when any object detected” behavior as the previous hat block.

 

Runs when the total stable number of currently detected objects changes, either up or down.
This is useful for counting games and scenes that react when objects enter or leave the camera view.

 

注意： detection hats wait for the same count to be seen in 2 consecutive frames before treating it as stable. This helps reduce one-frame flicker and accidental triggers.

---

🧠 模型和置信度控制 #

Switches the active object detection model.
The extension starts with the Fast model. Students can switch between models depending on whether their project needs faster detection, more image detail, or support for a wider range of object types.

↔ 在手机上左右滑动即可查看完整表格

 

提示： Start with the default Fast model for responsive detection. Choose balanced for more image detail, or wide when you need many more object categories.

重要提示： switching models clears saved detections. The current minimum confidence setting is kept when switching models.

 

Sets the minimum confidence required before a model result is reported as a detection.

The setting uses a scale from 0 to 1:

• 0.35 – default.
• 0.5-0.7 – stricter; can reduce weak or false detections.
• 0.2-0.3 – more permissive; may help with difficult detections but can also increase mistakes.

This is a useful way to teach students that AI predictions are not simply “right” or “wrong” – each detection comes with a confidence score.

 

Reports the current minimum confidence setting on the 0-1 scale.

Remember: both “minimum confidence” and “confidence of [OBJECT] #[INDEX] in [SOURCE] detections” use the same 0-1 scale.

---

⚙️ 检测控制 #

• 分类 [INTERVAL] - 选择检测运行的频率：
  • 每次运行此积木时检测一次
  • 连续检测，无延迟
  • 连续检测，每隔 50–2500 ms 运行一次
• 将分类 [ON_OFF] - 开始或停止连续检测。
• 分类间隔 - 返回当前检测间隔，单位为毫秒。
• 连续分类 - 返回连续检测当前是“on”还是“off”。
• 选择输入图像 [camera/stage] - 选择 camera（摄像头）或 Stage（舞台）。
• 输入图像 - 返回当前正在使用的输入源。

---

🎥 视频控制 #

• 将视频 [off/on/on-flipped]
  • on：以镜像方式显示摄像头预览（就像普通网络摄像头或镜子一样）。
  • on-flipped：以非镜像方式显示摄像头预览——方向与现实世界中的左右方向一致。
  • off：关闭摄像头预览。在 Stage 输入模式下，检测仍会继续运行。
• 将视频透明度设为[TRANSPARENCY|text] — 调整摄像头预览的可见程度：
  • 0：完全可见（不透明图像）
  • 100：完全透明（不可见，但仍在运行）
  使用 0–100 之间的值来调整摄像头背景的透明度，让 sprite 能够清晰地显示在摄像头预览上方。
• 选择摄像头 [CAMERA] — 从设备上的可用摄像头中进行选择。下拉菜单会列出检测到的所有摄像头，扩展会自动切换到你选择的摄像头。

• 显示检测框 [开启/关闭]
  打开或关闭检测框显示。 打开后，每个检测到的物体周围都会显示一个方框，让你可以看到模型正在跟踪哪些物体。 这对于测试摄像头、检查模型的识别效果以及实时调试项目都很有帮助。

Detection boxes are enabled by default for this extension. Use the detection-box control when you want to show or hide the visual boxes around recognized objects.

---

🎓 教育用途 #

• Introduce Computer Vision: Students can see how an AI model turns images into object labels and locations.
• Teach Confidence and Uncertainty: Compare predictions at different confidence thresholds and discuss why AI sometimes makes mistakes.
• Explore Coordinates: Connect detected X/Y positions to sprite movement and Scratch’s coordinate system.
• Explore Size: Use width and height to see how large an object appears in the image.
• Practice Lists and Indexing: Work with the first, second, or third matching object and compare multiple detections.
• Practice Sorting: Save detections and order them by position, size, or confidence.
• Teach Events: Use hat blocks to show how software can react to changes in the environment.
• Discuss AI Limitations: Compare what students see with what the model recognizes and investigate false positives, missed objects, lighting, angle, and object size.

---

🎮 示例项目 #

• Object Greeter: Make a sprite say hello when a chosen supported object appears.
• Object Counter: Show the number of visible objects and trigger a celebration when the count reaches a target.
• Follow the Target: Use the X/Y center of a detected object to move a sprite toward it.
• Left or Right? Save matching detections from left to right and use item 1 to find the leftmost object.
• Biggest Object Challenge: Save detections from large to small and make Scratch react to the largest one.
• AI Confidence Lab: Display confidence values and test how lighting, distance, angle, and background affect predictions.
• Classroom Object Hunt: Give students a list of supported objects and score a point when the AI finds each one.
• What’s in Frame? Display the “list of [SOURCE] detections” reporter block so students can observe how the model describes a changing scene.

🧩 自己试试看： pishi.ai/play

 

---

🔧 提示和故障排除 #

• 没有摄像头？
  • 确保摄像头已经连接，并且浏览器已获得摄像头访问权限。
  • 如果摄像头被阻止，请在浏览器的网站设置中启用摄像头权限，然后重新加载页面。
  • 加载扩展时，如果未检测到摄像头，输入图像会自动切换到 Scratch 舞台，也就是显示背景和角色的画布，这样你仍然可以继续使用和测试扩展的功能。
• 没有检测结果？
  • 连续分类：使用这个 reporter（报告积木）查看检测功能是否处于启用状态。
  • 如果检测已经启用，请改善环境光线，并正对摄像头。
  • 将分类 [ON]：如果检测没有启用，请使用这个积木，然后再次使用上面的 reporter 检查检测状态。
  • 在 camera 输入模式下，当摄像头关闭时，检测也会停止——你必须重新打开视频，或者将输入源切换到 Stage。
  • 在 Stage 输入模式下，系统会检测 Stage 上所有可见的内容——背景、sprite 或图像。你可以完全关闭视频，同时仍然处理 Stage 图像。
  • Stage 模式比 camera 输入速度慢，因此可以使用这个积木将检测间隔调低一些（例如 100–250 ms），以获得更流畅的效果： 分类 [INTERVAL]
  • 在 Stage 模式中，“left”和“right”对应的 landmark 会互换，因为 Stage 图像没有镜像——坐标空间表示的是真实的非镜像视图。
  • 当你使用以下积木时，检测也可能自动重新启动：
  将视频 [开启] / 分类 [INTERVAL] / 选择摄像头 [CAMERA] / 选择输入图像 [camera/stage].
• 画面方向翻转了？
  将视频 [镜像开启]：使用此积木可以显示非镜像的摄像头画面。“on”会像自拍画面一样进行镜像；“on-flipped”则显示真实的左右方向。
• 卡顿或运行缓慢？
  使用 100–250 ms 之间的检测间隔，或者关闭其他浏览器标签页，以减少设备的处理负担。
• 出现 WebGL2 警告？
  尝试使用 Firefox，或者使用支持 WebGL2 图形加速的较新设备。
• 想分析 Stage 而不是摄像头？
  选择输入图像 [stage]：使用这个积木分析 Scratch Stage 图像，而不是实时摄像头画面。 Scratch 舞台可以包含摄像头画面，以及当前在舞台上可见的任何角色和背景。
  注意： 处理 Scratch 舞台的速度可能会比直接使用摄像头更慢。

🎯 物体检测专项技巧 #

• No objects detected? Make sure classification is running, improve the lighting, move the object closer, and check that the selected model supports that object label.
• The object has the wrong name? Each detection model uses its own learned object vocabulary. Check the “list of [current] detections” block to see the exact label it returns.
• Too many false detections? Increase minimum confidence gradually, for example from 0.35 to 0.5.
• A real object is often missed? Try better lighting, a clearer angle, the Balanced model, or a slightly lower confidence threshold.
• Need an object that Fast or Balanced does not recognize? Try the Wide model, which uses a much larger class vocabulary.
• Only interested in a few objects? Add those exact labels to my detection list, then switch “look for [MODE]” to my detection list.
• My detection list mode suddenly detects nothing? Check that my detection list is not empty and that its names match the model labels exactly.
• Item 2 or item 3 returns empty? That item number only works when enough matching detections currently exist.
• Saved positions seem old? Saved detections are intentionally a snapshot. Run “save [OBJECT] from [SOURCE] detections, order by [ORDER_TYPE]” again, usually with current as the source, to refresh them.
• Project is slow? Use the Fast model, increase the classification interval, close other heavy browser tabs, or reduce how often your Scratch scripts read detection data.
• Camera is not available? Switch the input to stage and test with a backdrop, sprite, or imported image.

---

🔒 隐私与安全 #

• 所有处理都在你的浏览器本地运行。
• 不会将任何图像或视频上传到其他地方。
• 扩展启动时，模型文件会直接从 Pishi.ai 加载，因此不会使用任何第三方模型服务器。
• 使用摄像头之前，请务必先询问老师或家长。
• 你可以随时安全地使用此积木 将视频 [关闭].

---

🧪 技术信息 #

• Model Type: Pre-trained ONNX object detection models.
• Framework (框架)： onnxruntime-web.
• Browser Execution: Desktop browsers prefer WebGPU when supported, with WebAssembly (WASM) fallback; Android uses WASM.
• Model Options: Fast (320×320, 365 classes), Balanced (640×640, 365 classes), and Wide (640×640, 4,585 classes).
• Starting Model: Fast (320×320) on every supported device.
• Default Minimum Confidence: 0.35.
• Detection (检测)： Multiple objects can be reported from the same frame.
• Confidence Reporter: Returns a value from 0-1.
• Coordinates (坐标)： Detection-box centers are mapped to Scratch stage coordinates; width and height are reported in stage units.
• Sources: Reporter blocks can read either current live detections or saved snapshot detections.
• Filtering: Detection mode can keep all detections or only labels from the user’s custom detection list.
• Saved Detections: A snapshot can be filtered and ordered by confidence, horizontal position, vertical position, or size.
• Hat Stability: Detection count changes must remain the same for 2 consecutive frames before object-detection hats are treated as stable.
• Detection Boxes: Visual detection boxes are enabled by default and can be controlled from the extension.
• Inputs (输入源)： Live camera or Scratch stage image.
• Processing: Detection runs locally in the browser after the required model files are loaded.

---

🔗 相关扩展 #

• 🖐️ 手部关键点 – detect hand landmarks and gestures.
• 🕺 PoseNet – track full-body pose and movement.
• 😎 人脸网格 – detect facial landmarks and expressions.
• 🖼️ Image Trainer – train your own custom image classifier directly in Scratch.
• 🏫 Google 可训练机器 – import pre-trained custom classification models.

---

🖼️ 物体检测与图像训练器对比 #

↔ 在手机上左右滑动即可查看完整表格

 

💡 为什么选择物体检测？ #

Choose Object Detection when your project needs to find real objects and know where they are.
It is especially useful when students want immediate results without training a model first.

• Recognize several objects in the same scene.
• Count objects and compare their positions or sizes.
• Build projects that react to changes in the real world.
• Explore confidence, filtering, sorting, and computer vision in a Scratch-friendly way.

💡 什么时候应该改用图像训练器？ #

Use Image Trainer when you need categories that are specific to your own project – for example, your own hand gestures, classroom cards, custom commands, or objects that a pre-trained detector does not know.
Image Trainer lets students create the labels themselves and learn directly from their own examples.

简单来说：
Use Object Detection to find and locate objects the AI already knows.
Use Image Trainer when you want to teach the AI your own visual categories.

---

📚 了解更多 #

• pishi.ai 上的 AI 扩展帮助
• ONNX 运行时 Web 文档
• Scratch 创意

---