robot news, ircam app, october deadline madness

First things first: I found a chair for the robot, it is nice and has its own character. The way its put together is definitely changing my vision for the way the robot will move. I think I might have to saw some parts away, too.

Also, I realized I need a separate power source for my servos... I very likely have enough torque with the large servos I got from sparkfun (oh good!!) and I learned that NYC has shit for electronic components stores. I actually need to order break away headers and such since Radioshack doesn't have them and my searching for electronic components in NYC only came up with a very sketchy store-ish thing I'm not sure still exists.

Here's my ircam research proposal. Yessss.... kinda sloppy (esp. references) but dude, it was last minute and I kinda had to turn it in as is. I won't print my personal statement since it was a horrible amalgam of previous personal statements copy and pasted and hastily made to fit oh, in about the 15 minutes I had before the midnight paris time deadline. Also, yes its long. AND somehow when I copy and pasted it inserted strange line-breaks.

Project Description and Work Plan for Ircam Music Research Residency
N is for Neville who died of Ennui : An Interactive Performance System to Explore Musical Learning in Humans and Machines

Context and Description
The process of learning an instrument involves many small failures of the body. For example, one’s fingers may not press hard enough on the string on the guitar to let the note fully sound, or one may pluck the wrong string entirely with the other hand. Such incidents disrupt the mode of embodied interaction, that is, the subjective experience of the body interacting with the external world. The beginning guitar player becomes aware of her body in these moments of failure, and she must develop a set of mental representations for how and where to move her hands and fingers. In contrast, once she is an accomplished player, she does not have to think about where to put each finger in particular.

This project is an interactive performance system and series of musical works for this system, which will explore this process of learning an instrument by changing the mappings between the performer’s action and resulting sound during the course of performance. This system will change the mappings at the time when it can successfully predict the human performer’s musical gestures. The title of the project refers to boredom as a metaphor for this process, that is, the idea that the musical instrument gets bored with its performer, so it changes the pattern.

There has been much research in creating novel interactive systems for controlling music via gesture recognition. From The Hands (1984), by Michel Waisvisz, which used two wooden frames for the hands, to the more recent EyesWeb, which attempts to extract emotional affect from musical and dance movement. Most of these systems use gestural recognition to control sound in a controlled, pre-specified way, and the machine learning is performed off-line before the system is used in performance. Often, fuzzy logic is used in order to accommodate performers and make the novel musical paradigm more intuitive for the performer. This will be a completely novel system in that it adapts to the performer’s movements during performance, in order to take the performer out of her comfort level. The goal is not to create an intuitive instrument but to explore how musical gestures and movements progress from feeling unnatural to being intuitive. In this sense, it is somewhat similar to the i-Maestro gestural learning system, AMIR, although N is for Neville is not an educational tool, but an artistic exploration of embodiment during musical
learning and performance.

Research questions
1) How does a musician’s relationship to her body change when her actions do not have the intended effect? Does it seem foreign and externalized? Does this sensation go away as she learns how to play the instrument in its new form? When does this process of learning start to occur?
2) What changes in musical mappings can a performer adapt to quickly, and which one take a long time to learn? Which changes lead to an interesting experience for the performer? For the audience?
3) To what extent can particular neural networks predict repetitive gestural information in an improvisatory context? Does this prediction correlate with a performer’s mode of embodied interaction with the system? Are there additional measures from the gestural inputs that correlate with the mode of embodied interaction?

Aims and objectives
1) To explore how the experience of embodied interaction with a musical instrument
changes during the process of learning.
2) To explore the connections between machine learning of musical gestural and
movement with human embodied learning of the same system.
3) To gain new insight into how people learn to play music, and how their gestures and movement changes as they start to acquire skills and automaticity.
4) To develop a system that can learn to predict the periodicity of musical gestures in the context of a changing interactive paradigm in real-time.

Detailed Work Plan and Methodology
I wish to spend six months in Paris working with the Real-Time Musical Interactions
team at IRCAM to develop and research. gestural analysis techniques to create a novel interactive system. I would prefer to conduct my research and development during the dates of March 2012 to August 2012.

First, I plan to set up a system for motion capture of the right arm using sensors, such as accelerometers and image tracking. Then, I will develop a set of elementary motions of the hand and arm that will form a basic alphabet of the gesture recognition system. I will also address the problems of motion segmentation, and the representation of motion, possibly by using or modifying existing gestural analysis systems, such as the one used by i-Maestro’s AMIR.

Second, I plan to develop the sounds and sound parameters that the performer will be able to control with her movements. I will then compose a simple musical work for this changing musical instrument, and experiment with learning to play this work using the motion capture system with different sound-to-movement mappings. Then, I will implement a system that introduces novelty into the mappings between gesture and sound, and experiment performing my simple piece while the mappings change. In this stage, the mapping will change at timed stages; no prediction of gestures is occurring. I plan to generate novel, unpredictable mappings via a network of nodes with weighted edges. The inputs to this network are the descriptor values of the motion and the outputs are mapped to sound parameters. Each time a value passes through an edge, it is multiplied by the weight of the edge. When two or more values are sent to the same node from different edges, those values are summed before being sent to the next node. This map could have varying numbers of inner nodes and edges, which connect the input nodes to the output nodes. Changing the values of the edges will change the mapping between the motion and the sound, in an unpredictable way. Any number of output nodes may not be initially connected to the input nodes if there are zero edge values, but adding weight to the edges will allow the performer to suddenly be in control of a new aspect of sound.

This design allows flexible, unpredictable mapping in which the amount of change
between different mappings can be carefully controlled. I will be researching and designing the predictive system that will learn and train in real-time concurrently with the first two stages of the implementation. In the third stage, I will begin implementing and developing the first system to predict the performer’s gestures. The input to this system will be sequence of symbols from the gestural alphabet
developed in the first stage, that is, output of the gestural analysis system after motion segmentation and representation. This predictive systems will first be tested with manufactured input that has known patterns in order to judge its effectiveness in a controlled setting. Initial measures and values for prediction success will be developed at this time. After initial development, this predictive system will be integrated into the musical interactive system and this system will determine when to change musical mappings. Then, through a series of experiments, the effectiveness and results of the first version of N is for Neville who died of Ennui will be tested and explored. Different performers will learn and test the system trying to play the musical work that was previously composed and tested with the timed system of changes. Further, the systems will be tested on during improvisatory sessions, to see if they can learn performer’s gestures settings in tat context as well. After this stage is complete, I will evaluate the predictive system and determine if it needs changing and refinement, and if there are ways or new approaches that will improve and refine its predictions. After new approaches and refines, further testing will follow. I will also keep a record of my experiences with embodied interaction while performing with my interactive system, and will ask other experimental subjects (performers) to write down their experiences as well. During this last stage, I will compose a work of music for this novel interactive system which will be informed by my experiences developing, testing, experimenting with the system.

Outcomes
The outcome of this project will be a novel interactive system that changes the mappings from movement to sound when it can predict the performer’s gestures, and at least two musical compositions written to explore the potential of this system. I also anticipate that this project will lead to better information and knowledge about the predictive behavior and training phases of neural networks and other pattern recognition algorithms in the context of pattern recognition.

Further, I hope to gain from this process a better understanding between learning, bodily movement, and the mode of embodied interaction.

Bibliography
Benbasat, Ari Y., Joseph A. Paradiso, An Inertial Measurement Framework for Gesture Recognition and Applications, Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction, p.9-20, April 18-20, 2001.

Bevilacqua, Frederic. Fabrice Guédy , Norbert Schnell , Emmanuel Fléty , Nicolas Leroy, Wireless sensor interface and gesture-follower for music pedagogy, Proceedings of the 7th international conference on New interfaces for musical expression, June 06-10, 2007.

A. Camurri, B. Mazzarino, M. Ricchetti, R. Timmers, G. Volpe
Multimodal analysis of expressive gesture in music and dance performances, in A.
Camurri, G. Volpe (Eds.), "Gesture-based Communication in Human-Computer
Interaction", LNAI 2915, Springer Verlag, 2004.

Camurri, A., B. Mazzarino, G. Volpe. Analysis of Expressive Gesture: The EyesWeb
Expressive Gesture Processing Library, in A. Camurri, G. Volpe (Eds.), "Gesture-based Communication in Human-Computer Interaction", LNAI 2915, Springer Verlag, 2004.

Cumming, Naomi. The sonic self: Musical subjectivity and signification. Bloomington: Indiana University Press, 2000.

Merrill, D. and Paradiso, J. A Personalization, "Expressivity, and Learnability of an
Implicit Mapping Strategy for Physical Interfaces", Proc. of CHI 2005 Conference on
Human Factors in Computing Systems, Extended Abstracts, ACM Press, Portland, OR, April 2--7, 2005, pp. 2152--2161.

Rolf Inge Godøy1, Egil Haga1 and Alexander Refsum Jensenius1, "Playing 'Air Instruments': Mimicry of Sound-Producing Gestures by Novices and Experts", Gesture in Human-Computer Interaction and Simulation, February 2006.

Schroeder, Franziska. "The old and the new and the new old: A conceptual approach
towards performing the changing body". Hz #7 December 2005

———. "Re-situating performance within the ambiguous, the liminal, and the threshold:Performance practice understood through theories of embodiment." Diss. U of Edinburgh, 2006.

Waisvisz, Michel. The Hands, 2005

Okay, the end. My Other Minds project proposal was quite a bit less dry:

For the Other Minds Festival, I propose to create a piece for violin, cello, live electronics, and two robots made from found objects.

The frame of the first robot is made from a small, used wooden chair, and it makes percussive sounds by walking with its floor legs hitting the floor. This robot is intended to be precarious and seem child-like. The chair will travel around the space in loping circles, and sometimes I will have to physically restrain it move it back in place if it tips over… however, this robot will be able to avoid most obstacles using the input of a sonar sensor. I also intend mic the floor (using my contact microphones) in order to amplify and process the sound. The second robot will use a bicycle wheel to create percussion effects. When the wheel turns enough, it’s spokes will hit across metal bars. These sounds will also be amplified and processed by my laptop running SuperCollider.

I will control the motions of the robots via the laptop and include their percussion parts in my score. During the performance, I will be handling the live electronics and keeping the robots out of trouble. The performance should last around 8-10 minutes. I would be able to send a recording of the robot parts prior to the performance for rehearsal purposes.

I intend to write a score including many syncopated rhythms inspired by tango music and dancing. I am a tango dancer, myself, and I am particularly interested in the physicality of the piece: the juxtaposition of the awkward mechanical robots with skilled, embodied musicians. I also intend for this piece to invoke the connections between rhythm, movement, feeling, and how we perceive rhythm and movement in both human and vaguely human-like mechanical motion and sound.

AND... I totally still need to finish my origami piece. I ended up sending all old pieces to festivals but Oct. is the month of mad deadlines, so.