Gesture Controlled Visuals

Hand Gesture Controlled Abstract Visuals

VIEW LIVE LINK

The integration of intuitive interfaces in digital media is a burgeoning field, enhancing user interaction through innovative methods. This project proposes the use of ml5.js, a user-friendly machine learning library built on TensorFlow.js, to create a real-time hand gesture recognition system. This system will control abstract visual elements, providing an immersive and interactive experience for users. The application will enable users to manipulate visuals on a screen through simple hand movements, bridging the gap between traditional input devices and gesture-based interaction.

Resources

Mr Bean regression

Handpose_Webcam

Final Project Progress Update Log

Log 1

• _{Reference visuals found and modified}
  • _{“noPersp01” by KomaTebe Link}
  • _{“tw_moebius” by KomaTebe Link}
• _{Learned HandPose source code on ml5's website}
  • _HandPose
• _{Researched examples of Hand Pose Controls}
  • _{Mr Bean regression}
  • _{Handpose_Webcam}
    • _{Event-Driven Programming: The use of the .on("predict", callback) method demonstrates an event-driven programming style. This method sets up an event listener that updates the predictions array every time new hand pose data is available.}
    • _{Manipulation of Global Variables: The predictions variable is used globally to store the hand pose data from the handpose model.}
    • _{Interfacing with Hardware: The example illustrates how to interface with hardware components (like a webcam) using createCapture(VIDEO) and process this input through a machine learning model. This is important for applications that require direct interaction with hardware sensors.}
    • _{Graphics and Visualization: Drawing keypoints on the canvas using the ellipse function teaches basic techniques for visualizing data points.}
    • _{User Interface Manipulation: By hiding the original video feed and displaying only the canvas (video.hide()).}
    • _{Asynchronous Loading and Initialization: The modelReady callback function, which logs "Model ready!", indicates model loading copleted.}
  • _{Rock, Paper, Scissors with Handpose and KNN}
    • _{Interactive Machine Learning: Users train the k-Nearest Neighbors (kNN) classifier by providing hand gestures as input, which the model uses to learn and classify gestures like rock, paper, and scissors.}
    • _{Dynamic Data Collection: The example demonstrates how to collect data in real-time from the user. Users add training data by performing gestures in front of the webcam and pressing corresponding buttons.}
    • _{Real-Time Prediction: The system classifies hand gestures on the fly and provides instant feedback by displaying the corresponding emoji.}
    • _{User Feedback and Instructions: The application provides instructions if no hand is detected in the camera's view, guiding users for better interaction.}
    • _{Visual Feedback: It visually represents the keypoints detected on the user's hand, helping users understand how the model interprets their gestures.}

Log 2

_{Attempt 1: MySketch v1}

• _{What I did}
  • _{Added the default visual adapted from “tw_moebius” by KomaTebe Link}
  • _{Added User Interface & style.css adapted from Rock, Paper, Scissors with Handpose and KNN}
  • _{Added Zoom in&out function - IT WORKED! - I tweaked it a couple times but i didn't log it}
• _{My Next Steps}
  • _{Add in the other visual}
  • _{Figure out a way to make the transition smooth between the two}
  • _{Tweak the two to make the background color / lighting consistent}

Log 3

_{Attempt 2: MySketch v2}

• _{What I did}
  • _{Added the NEW visual adapted from “noPersp01” by KomaTebe Link}
  • _{Added a FULL SCREEN Button - but the fonts are not consistent in this version so i changed it in the style sheet on my next version}
  • _{Tried to use a gesture to draw new visual but failed.}
    • _{I tried to use the finger pointing up gesture to draw the new visuals but it did not work, so I changed it in my next iteration.}
• _{My Next Steps}
  • _{Try to figure out how to draw the new visuals}

Log 4

• _{I did not save the TOO MANY attempts but I did the following}
  • _{I switched to the Scissor gesture to draw the new visuals, it works a bit better but sometimes when i was just trying to perform the zoom in and out function it will trigger the draw new visual function}
  • _{I fixed the fullscreen button's UI design so that it will match the other one and overall pleasure for my (and users') eyes}
  • _{I changed the background color and lighting settings to make the two visuals consistent, that way when it transitions between the two it will not seem glitchy and unpleasant}
  • _{I also added a window resize function for the fullscreen function}

Log 5

_{Attempt No.I CANNOT COUNT: MySketch v3}

• _{What I did}
  • _{The draw new visual function worked this time - I also encountered an error where I will have to keep the gesture for the new visual to stay on the screen without changing back and I realized the problem is with the logic so I fixed it.}
  • _{The Zoom in and out function works pretty well this time.}
• _{My Next Steps}
  • _{I want it to switch back to the default visual with another gesture.}

Log 6

_{My Final Iteration!! Kara_Final}

• _{What I did}
  • _{It took me a few attempts to figure what sets of gestures i should use for drawing the new visual and the default visual without the two affecting each's performance - and I finally landed on the palm up and down gestures, and used the orientation of index finger and alm to do it.}
    • _{I also tried thumb up and down function but it will affect the zoom in and out function so I gave up on this.}
    • _{I also tried adding thresholds for how many frames the gesture should hold before the function draws but it didn't work very well so I also did not move forward with this.}
  • _{I changed the instructions to make it more clear to the users.}
  • _{Also other small tweakings to make it more fun and pleasant. AND There goes my final project!!}
• _{**What I wish to do after the class ends **}
  • _{I felt like there are more potentials to the hand pose controls than what I am doing right now (of course) and I wish to dig into it more.}
  • _{Keep iterating on this to make it more smooth and add in more visuals and try out other ways to switch or play with the visuals or animations.}
  • _{Keep learning ml5 it's fun.}

‍