Electronic ARTrium

2021 ~ Present | Potential sponsors are being contacted

Goals

Design and build an interesting and engaging interactive sensor-driven dynamic art installation. The installation consists of a combination of three kinds of original art — sound, mechatronics, and video — that all respond to participant motion in a synchronized way. Key points in participants’ poses and facial expressions, extracted in real time from camera output, are processed and then streamed over Ethernet to control the images, sound, and mechatronic movement. The goal is to give the participants a sense of wonder and intrigue as they experiment with how their body movements change the art.  

The overall purpose of this project is to explore the most effective mechanisms for interactive multimedia art.  A secondary purpose is to attract young people to STEM fields (science, technology, engineering, and mathematics).

The first generation of this installation was entitled “Raise Your Hand” and ran in a two-week show in the lobby of the Ferst Center for the Arts, November 1-14 (see https://www.youtube.com/watch?v=gC2FGTFGS2c and

https://www.youtube.com/watch?v=qtsGjhYW__A).  We expect to have another show in the Ferst Center in Spring 2024 and we will also seek the opportunity for shows in other venues at Georgia Tech or in metro Atlanta. The student will be expected to help with set building, transportation, set up, and breakdown for shows, as well as serving as a guide for participants during shows.

 

Going forward, we will continue aiming to create an entertaining interactive multi-media experience for the participant, while building on the successes and using the feedback from the first show.  We expect to continue using cameras and MediaPipe for pose and facial key points information, Ethernet for UDP packet communication, Unity 3D for the video, Max or Max for Live for the music, mechatronics driven by microcontrollers, and an invisible LIDAR screen to protect participants from the mechatronics and vice versa.  Changes we anticipate are having more and separated sections, more mechatronics, use a larger set of gestures, expand to recognizing pose of multiple people per section, and improve the facial expression recognition.  We are also interested in interaction between a mechatronic figure and a similar figure in the video, in addition to interaction between participants and the exhibit. We also want to experiment with new ways to use music with mechatronics, such as incorporating motor sound as percussion and musical composition around or in concert with the motion of mechatronic figures.

Issues Involved or Addressed

Design considerations include originality and artistic appeal, immediacy and clarity of interactive response (i.e., avoiding participant confusion), safety, ruggedness and reliability, and cost.

 

Methods and Technologies

  • SolidWorks or your favorite CAD for mechanical design
  • Arduino IDE (similar to C++) to control motors and LEDs
  • Ethernet networking with UDP for communication between devices and applications
  • Max or Max for Live for receiving the UDP packets and mapping the gesture parameters to music control messages
  • Ableton for sound generation
  • Unity 3D for interactive scene/image generation (uses C# scripts)
  • MadMapper for projection mapping
  • MediaPipe for pose and facial expression tracking
  • Python programming for receiving and processing gesture information and generating UDP packets

Academic Majors of Interest

  • ComputingComputational Media
  • ComputingComputer Science
  • ComputingHuman-Computer Interaction
  • DesignIndustrial Design
  • DesignMusic Technology
  • EngineeringComputer Engineering
  • EngineeringElectrical Engineering
  • EngineeringMechanical Engineering
  • EngineeringRobotics

Preferred Interests and Preparation

The student should be interested in the intersection of art and technology and in building things. Object oriented programming experience will be helpful.

 

 

 

Meeting Schedule & Location

Time 
9:30-10:20
Meeting Location 
Van Leer 465
Meeting Day 
Thursday

Team Advisors

Dr. Mary Ann Weitnauer
  • Electrical and Computer Engineering
Thomas Martin
  • Georgia Tech Research Institute

Partner(s) and Sponsor(s)

Potential sponsors are being contacted

Related Sites