How it works
The controller has 3 seperate systems that all work together:
- Electronics
- Python
- Unity
The electronics consist of a number of analog potentiometers which all lead into an AD7490 Analog to Digital Converter. This ADC takes in all of the analog values and sends out the data over the SPI bus. These SPI signals go into an FT232H breakout board which is able to talk to the computer over USB. A Python program then communicates with this board and uses it to send and receive messages from the ADC.
Here is the assembled circuit board
and here is a picture of the data coming from the board. The “Select” line indicates that the microcontroller wants to talk to the ADC. The “D In” and “D Out” lines show the data being read or written.
The python program asks the ADC for the values of all 16 channels and gets back the values. It also gets the coordinates from the Kinect from another python program through a named pipe, also known as a FIFO. This main python program then sends out a string of all of the data to two more named pipes, one for the paint-brush/hand model, and another for the canvas.
Spatial tracking of the hand is done using a Kinect, which has a python program controlling it. It gets the depth image, a black and white picture where the value of each pixel is how far away they are from the sensor, and the color image. Using the following series of filters, only pixels that are black remain.
Check out this post to learn more about how it works.
Once it has isolated all of the black pixels, it uses OpenCV image analysis to find the large blobs of pixels are that are the same distance away from the kinect. It finds the center of this blob and uses those coordinates to find the depth at that point. Those three coordinates are then sent back to the main python program.
In Unity, a C# script reads all of the information sent over the pipe. It turns all of the voltages into rotations, and moves objects around according to the position.
For the mounting of the potentiometers, I 3D printed little holders the snap together. Here is the first prototype:
Later, I refined them to make them stronger and also added a ring around them to keep them on your fingers:
Once they were all connected and working, I hot glued a glove inside of all of the rings to keep it all together.
What I Would Do Differently
- Connectors. The 1.5mm ZH series JST connectors are good at what they do, they are small and they stay in pretty well, but they are a major pain to get crimped. I was using a generic tool instead of the official one, but I still broke more than half of the crimps that I used. It looks like it is possible to get precrimped wires with the the connector attached, so if I did this again I would either buy those or use different connectors.
- Board layout. Not that I would do it differently, but I would pay more attention to the layout. The breakout board that turns the SPI signals into USB signals is on the wrong side of the board, and the voltage reference for the ADC was set to the wrong mode. None of these stopped me from completing the project, but I had to put the breakout board on upside down and cut one of the traces on the board with a knife.
- Different potentiometers. I am not quite sure if there are better ones to use for this purpose, but the ones are used are definitely not desgined for this. These potentiometers are designed to be soldered to a circuit board by a robot and used with some sort of shaft going through them. Because they were meant to be soldered to a circuitt board, there is no other way to mount them securely. The snaps that I used kind of worked, but I never figured out the right thickness for the walls, and the 3D printers we have didn’t really do a good job of printing it. I ended up hot gluing them anyway.
- Different mounting. I only used the wool glove because I found it in a box and it fit underneath the rings that I had already made. It works fine for the potentiometers that are attached to the rings, but the one for the first knuckle doesn’t really work at all because the whole glove just twists around it. I would like to have a glove entirely made of plastic so that there is no flexibility.