Control Pishi with Your Shoulders in Scratch

Use your shoulder tilt to move Pishi left and right, catch apples, avoid donuts, and score as many points as possible before time runs out.

⭐

Difficulty
INTERMEDIATE

🕒

Estimated Time
20–35 MINUTES

🧩

Extension

POSENET

📖 What You Will Learn

How to use PoseNet to detect a person’s body
How to measure the angle between the left and right shoulders
How to control a Scratch sprite using body movement
How to change movement speed based on shoulder tilt
How to create falling objects with random positions and speeds
How to use shared variables to control a timed game

PoseNet tracks body keypoints such as the shoulders, elbows, wrists, hips, knees, and ankles. In this project, the angle between the shoulders controls Pishi’s horizontal movement.

🧰 What You Need

A computer with a working camera
Chrome, Firefox, Edge, or another modern Chromium-based browser
Pishi.ai Scratch (start a new project)
PoseNet extension (view the help guide)
A Pishi sprite
An Apple sprite
A Donut sprite
The following variables:
- score
- time
- angle
- apple speed
- donut speed

Use good lighting and keep your upper body visible to the camera so PoseNet can detect both shoulders clearly.

🛠️ Build Your Project

1. Set Up Pishi and Shoulder Control

Place Pishi near the bottom of the Stage and create these variables:

score
time
angle

When the green flag is clicked:

Set score to 0
Move Pishi to x: 0 y: -130
Reset the timer
Continue the game until the timer becomes greater than 20

Inside the loop, first check whether PoseNet has detected at least one person. Then set angle to: abs of (angle between left shoulder and right shoulder) – 90 Using abs allows the script to work whether PoseNet returns an angle near 90 or -90. Use a small neutral zone to prevent unwanted movement:

If angle > 3, change x by 5 + angle
If angle < -3, change x by -5 + angle

The shoulder tilt controls both direction and speed. A larger tilt makes Pishi move faster. With this setup, Pishi moves toward the shoulder that is higher than the other. While the game is running, continuously set time to the current Scratch timer value.

2. Add the Falling Apple

Add an Apple sprite and create the variable:

apple speed

When the green flag is clicked:

Set time to 0
Repeat until time = FINISH
Place the apple at a random x position above the Stage
Show the apple
Set apple speed to a random value from -10 to -5

Move the apple downward until:

Its y position becomes less than -160, or
time = FINISH

While the apple is falling, check whether it touches Pishi. When the apple touches Pishi:

Increase score by 1
Play the Chomp sound
Hide the apple
Move it downward so the current fall ends

The outer loop then places the apple at another random position and starts a new fall.

3. Add the Falling Donut

Add a Donut sprite and create the variable:

donut speed

The Donut script follows the same structure as the Apple script.
When the green flag is clicked:

Set time to 0
Repeat until time = FINISH
Place the donut at a random x position above the Stage
Show the donut
Set donut speed to a random value from -10 to -5

Move the donut downward until:

Its y position becomes less than -160, or
time = FINISH

While the donut is falling, check whether it touches Pishi.
When the donut touches Pishi:

Decrease score by 1
Play the Bite sound
Hide the donut
Move it downward so the current fall ends

The main differences from the Apple are the speed variable, sound, and score change.

4. Feed the Fish to Pishi

After the timer becomes greater than 20 seconds, the main repeat loop ends.

Then Pishi:

Sets time to FINISH
Says Time’s up
Plays the Meow sound

The Apple and Donut scripts both use:

repeat until time = FINISH
Setting time to FINISH tells both falling sprites to stop.

The Apple and Donut also set time to 0 when the green flag is clicked.

This prevents either sprite from immediately leaving its loop if it starts before Pishi resets the value from a previous game.

🚀 Try Your Project!

Click the green flag and allow camera access.
Then:

Stand where the camera can clearly see your upper body.
Keep both shoulders visible.
Raise one shoulder and lower the other to move Pishi.
Tilt further to make Pishi move faster.
Catch apples to increase your score.
Avoid donuts so you do not lose points.
Try to get the highest score before the 20-second timer ends.

🤖 How Does It Work?

PoseNet detects the positions of the player’s left and right shoulders.

The project follows this process:

Your shoulders
↓
PoseNet detects both shoulder keypoints

↓

The extension measures the angle between them
↓
Scratch uses the absolute angle and subtracts 90

↓
The result controls Pishi’s direction and speed

↓
Pishi catches apples and avoids donuts

The Apple and Donut sprites repeatedly appear at random horizontal positions and fall from the top of the Stage.

After they are caught or leave the Stage, they return to the top at a new random position.

💡 Make It Better!

Raise Both Hands to Activate a Shield Use the Y positions of both wrists and shoulders. When both wrists are above their corresponding shoulders:
- Activate a shield
- Prevent donuts from reducing the score
- Change Pishi’s costume
- Play a power-up sound
This introduces wrist tracking while keeping shoulder tilt as the main control.
Use One Raised Hand to Control Falling Objects Detect whether the left or right wrist is above its shoulder. For example:
- Raise the left hand to slow the donuts
- Raise the right hand to slow the apples
- Lower both hands to return to normal speed
This allows the player to control different parts of the game with separate body poses.
Duck to Avoid a Flying Obstacle Add an obstacle that moves horizontally across the Stage. Use the Y position of the nose or shoulders to detect when the player crouches. When the player ducks:
- Make Pishi briefly shrink
- Move Pishi lower
- Allow the obstacle to pass overhead
This adds vertical body movement to the game.
Add a Jump Power Use the Y positions of the ankles, knees, hips, or shoulders to detect a jump. When the player jumps:
- Make Pishi jump
- Collect all nearby apples
- Avoid a ground-level obstacle
- Activate a short score bonus
This demonstrates PoseNet’s lower-body tracking capabilities.
Add a One-Leg Balance Bonus Use the ankle or knee keypoints to detect whether one leg is raised. While the player balances on one leg:
- Apples give two points
- Donuts fall faster
- A bonus meter increases
This turns the project into a balance and coordination challenge.
Create Pose Combinations Combine multiple PoseNet conditions to trigger special actions. Examples:
- Both hands up → shield
- Left hand up → freeze donuts
- Right hand up → attract apples
- Crouch → avoid flying obstacles
- One leg raised → double-score mode
- Arms stretched sideways → clear all donuts
This shows how several body keypoints can work together to create more advanced full-body controls.

🔗 Share Your Creation

Finished your project? Add your own sprites, sounds, backgrounds, and PoseNet controls. Try creating a different theme, such as:

fish and trash
stars and asteroids
healthy food and junk food
treasure and falling rocks

Then share your project with friends or classmates.

OPEN PROJECT & REMIX