kinect camera to control stop-motion video flow

In this post I would like to explore embodied ways on how one could interact with a visual stream. More specifically, this is about controlling the flow of a video stream with one own's body movements.

The system I am using is based on Microsoft's Kinect camera. On the computer side I am using the Quartz Composer to connect different streams of information on the one side body related parameters and on the other parameters related to the video stream.

In the following video, I am showing a simple situation where the movie is controlled by vertical movements of my hand.


Kinect controlled video playing from sonat on Vimeo.

Testing Stimulus Presentation Program using Emulated Scanner Pulses

If you fancy Psychophysics toolbox to present stimuli during an fMRI experiment, you will need to know exactly at which time point the volume acquisition occurs with respect to important events of the experiment, such as for example stimulus onsets. 

Scanners usually send pulses and these are needed to be recorded by the stimulus computer in order to solve this problem. In the specific setup I am using, these pulses are collected in a circuit, called cogent-box, and accumulated. This cumulated count is sent to the serial port of the stimulus computer in the form of 2 bytes. The stimulus program can now correctly time events from different sources e.g. stimulus onsets and the start of volume acquisitions.

The downside is generally that one would need to go to the scanner's console room and run few dummy experiments with the scanner until the presentation code is devoid of any bugs.

However, it would be much nicer to test the stimulus presentation program in the office without the need of having a scanner sending pulses. Therefore some labs have a mobile version of the cogent box with the aim of testing your presentation program without the need of having a scanner to send pulses. But even then, today many modern computers do not have serial ports, making impossible the use such mobile boxes. 

A software solution can circumvent many of the problems associate to a mobile box. In a modern computer, one may create any arbitrary number of virtual ports (Serial or Parallel), read and write data to and from these ports. There are softwares that let you specify any number of virtual ports easily, for example "Virtual Serial Port Deriver". 

Once you have this software you can create a pair of virtual serial port, say COM1 and COM2. You can fire the Matlab, use COM2  and write information in the form of two bytes (to emulate what the cogent-box would be doing in response to pulses received from the scanner at each TR). These two bytes imitates the situation where the cogent-box is counting the number of volumes that have been so far acquired. In another instance of Matlab, running in parallel to the first one, you can set COM1 and read data from it. This will be analogous to the situation you are in the scanner room, without the need of a mobile cogent box or a scanner.

The following psychophysics toolbox function can be used to send pulses.

function SendPulses(N,TR)
%function SendPulses(N)
%
%Sends N pulses to COM2 serial port. TR is the interpulse TR (that is
%the "TR").
%
%Selim Onat, 24-Jan-2013 10:55:53

IOPort('CloseAll')
box.port = IOPort('OpenSerialPort', 'COM2');
counter  = 0;
%
t        = GetSecs;%now
t        = t:TR:t+TR*N;%time of the future pulses
for n = 1:N
    tobesent = [floor(n./255) n];%two byte to be sent
    WaitSecs('untiltime',t(n));%wait and fire.
    [~,t_]=IOPort( 'Write' , box.port , [char(tobesent(1))+1,char(tobesent(2))]);
    counter = counter +1;
    fprintf('Pulse No: %3d sent at %0.10g was due to at %0.10g\n',counter,t_,t(n));
    drawnow%give some time for Control-C
end

The Famous Lenna appears in Woody Allen Movie

Anybody who has ever read a book or paper about Sobel Filters, kernels and digital filtering in the context of image processing should have seen at least once the following test image.

This is the famous Lena picture.  But who is Lena? Interestingly, back then in 1973 Woody Allen used the same photograph in one of his famous movies, The Sleeper. For me this was this funny coincidence unrevealed the origin of this test picture. It is actually taken from Playboy Magazine as Woody Allen says in the movie. 

Following a quick search I was able to find this article (http://www.instantfundas.com/2009/04/story-of-lenna-famous-test-image.html), which is just about Lena's history in science. This also explains why the image is generally so strangely cut.

Well let's state the facts here... This is a picture of a women from the Playboy Magazine, where women are paid to show their body parts in erotical manners for men who need to see these type of stuff.  This picture has been used for decades by generations of researchers in the field of computer vision to showcase their algorithms. They described their algorithms in greatest details by describing the way how they transformed this image of a playmate pixel-by-pixel with a grin on their face in papers and conferences to other scientists.

The question that follows from these facts is why didn't they used another image such as Nixon, Woody Allen or maybe Rambo? One valid answer is that this community was dominated strongly by males as in many other academic fields. And today I think it is clear to many of us that the remarkable thing about this image is that it is one of the hallmarks of the patriarchy played in the field of computer vision.

Kaleidoscope + Stopmotion + Photographic Content + Minimal Electro

This was a project I was working on since long time. I am really happy that it is now here finally "published". These photos were taken as in a typical stop-motion setup. In Quartz Composer, I connected the speed of the photographic flow to the amplitude of audio signal at a given frequency band (usually low frequency, because beats are located there). This makes the video move faster with each beat. In addition to this, I applied different visual symmetry effects, known as kaleidoscope. The outcome is visually unusual when typical urban or natural scenes are used and fits perfectly to minimal electro. Two performances follow, the first illustrates the concept using photographs taken in an urban area, namely Osnabrück, Germany. And the second one, which uses images taken in a natural park.
 

Real-Time Halftoning with Perfect Circles

How to transform an image in real-time into an array of perfect circles (or other geometrical shapes) with their diameters proportional to the local image brightness below the circle?

This is classically known as halftoning and one of the most ancient digital image manipulation methods, still used today in printing business.

I managed to do this using Quartz Composer, here is an example video. In this video, the resolution i.e. the number of circles used, changes over time. Video starts with black and white circles, continues with circles colored with original pixel values and finishes with random colored circles. It is all done in real time without any noticeable delay.

 

Perfect for VJing purposes. When real-timing is not necessary the content of the video can be a political figure for example...


Halftoning AngelaMerkel from sonat on Vimeo.

The concept can be generalized to other politicians or other half-toning shapes...

 
Halftoning Politicians from sonat on Vimeo.


Halftoning AngelaMerkel Hearts from sonat on Vimeo.


Cross modal integration with auditory modality could be implemented by adding a noise to the diameter of the circles that is proportional to the amplitude of the instantaneous sound signal.. 

Programming a Weather Station with Quartz Composer

This is a weather station clock programmed in Quartz Composer. There are 6 different parameters that are taken into consideration. The temperature, air pressure, rain drop, humidity, wind speed and direction. Weather information is downloaded from Osnabrueck University Physics Department's web page and an XML parser extracts different parameters. These parameters determine then the background color (reddish for warm and bluesh for cold), diameter of the turning rainbow wheel (high air pressure), global blurriness (humidity), speed of the wheel (average wind speed). Two examples illustrates how the clock behaves under different weather conditions:

Hello Quartz Composer

One of the first things I have programmed with Quartz Composer was this rainbow clock. I currently use this as a screen saver.

 
Clock from sonat on Vimeo.

Varela's View of Cognitive Science..

The most integrative figure of the cognitive science ever made. 

This figure has been published in "The Embodied Mind" (1991) by Varela, Thompson and Rosh.

Different directions depict different subdisciplines of the cognitive science. The distance from center represents the philosophical flavor.  While Constructivists are located around the outer shell of the circle, the center part is occupied by Representationalists. The way these latter conceive the mind, is compatible with the view that the mind i.e. intelligent behaviour is a symbolic representational machine, pretty much like a computer. 

These views have completely different opinions on evolution as well.