Goals
Design and build an interesting and engaging interactive sensor-driven dynamic art installation. The installation consists of a combination of three kinds of original art — sound, mechatronics, and video — that all respond to participant motion in a synchronized way. Key points in participants’ poses, extracted in real time from camera output, are processed and then streamed over Ethernet to control the images, sound, and mechatronic movement. The goal is to give the participants a sense of wonder and intrigue as they experiment with how their body movements change the art.
Issues Involved or Addressed
• Fabrication and motion control of mechatronic elements, such as blooming flowers, talking Snowy Owl, honeycomb cells that individually light up and move forward and backward, and a Dancing Snake
• Creation of interactive videos of scenes, animals, and plants
• Lighting control for timed spot lighting and strobe (to simulate lightning)
• Implementation of interactive music and sound effects
• Write C# programs to control the music, video, and mechatronics in all sections.
• Integration of a bar-code printer at the entrance to the exhibit and four bar code scanners, one scanner at the entrance to each section, to track the player’s progress in the game
• Implementation of LIDAR to detect occupancy in each section
• Implementation of speech/text-to-gesture software to control movements of video and mechatronic characters
• Detecting the player’s pointing direction using a depth camera
• Set building and decoration
• Networking the elements of the exhibit
• Website development
The first generation of this installation was entitled “Raise Your Hand” and ran in a two-week public show in the lobby of the Ferst Center for the Arts, November 1-14, 2022. (see https://www.youtube.com/watch?v=5bcnNZb77v8 , https://www.youtube.com/watch?v=gC2FGTFGS2c , or https://www.youtube.com/watch?v=qtsGjhYW__A).
Our second show, which will take the form of a life-sized four-section interactive video game with mechatronics, will be in the Ferst Center in Spring 2025. The participant must find the clue in each of the first three sections through game play. A different interactive mechatronic narrator in each section coaches the participant and converses with Mr. Bee, the main character, who has become lost because of a storm. The participant will help Mr. Bee find his way back to the hive by body tilting to avoid “attack clouds” in Section 1, pointing to the right sequence of mechatronic blooming flowers in Section 2, assuming a series of poses that unlock keyholes in Section 3, and using the clues to unlock the entry to the hive in Section 4.
The new students will be mentored by more experienced students. The Sound Design sub-team will work with stems and sound effects created by a professional music and sound effects composer.
The overall project team includes this VIP team and may also include one or more interdisciplinary senior capstone design teams and undergraduate research students.
Methods and Technologies
Academic Majors of Interest
- Computing›Computational Media
- Computing›Computer Science
- Computing›Human-Computer Interaction
- Design›Industrial Design
- Design›Music Technology
- Engineering›Computer Engineering
- Engineering›Electrical Engineering
- Engineering›Mechanical Engineering
- Engineering›Robotics
- Ivan Allen›Literature, Media, and Communication
Preferred Interests and Preparation
The student should be interested in the intersection of art and technology and in building things. Object oriented programming experience will be helpful.
Meeting Schedule & Location
Team Advisors
- Electrical and Computer Engineering