Computational Portraiture Final

Conceptual Approach:

My conceptual approach was informed by a number of personal experiences leading up to the project.  I took to heart the notion that a portrait should convey a specific character trait, or represent an element of the subject's personality.  I also came to realise that the environment in which the subject is depicted can be just as important as the subject themselves.

As the project began to evolve, this became an important part of the conceptual development.  It became clear to me that capturing a subject in an environment familiar, or rather important, to them was essential to establashing context for the portrait.  For this reason, I wanted to combine the techniques of photgrammetry, for capturing the environment, and the use of the depthkit for capturing the subject.

Selecting a subject:

Subject 1:

I'd initially had the idea of taking a portrait of the buskers who perform on the trains.  I used to see them often, late at night, during my trips back to Brooklyn on the L train.  Before I had the opportunity to approach them, I witnessed a performance by a young street musician drumming on plastic paint drums in the Union Square subway station.  My interest was sparked when I observed a young woman film the musician using her smart phone for an extended period of time.  When her train arrived at the platform, she promptly stopped filming and stepped onto the train.  I found myself getting annoyed that she hadn't so much as acknowledged the young boy, let alone left him any money, as a sign of appreciation for her performance.  This immediately led me to wonder how often that happens, and how situations like that make him feel.  My train arrived in the middle of his performance, so not wanting to interupt him I boarded my train and returned to the station later.  Thankfully I found him upon my return, and I approached him regarding my idea to take his portrait.  Due to the fact that he was a child, I wanted to get permission from his parent/s before moving forward, so I gave him my number and asked him to get them to contact me.  Later that night I spoke with his mother, who declined my offer, without providing a reason for doing so.  That was the end of subject 1.

Subject 2:

I'll be collaborating with a choreographer and group of dancers for an interactive dance performance in the future.  I attended a rehearsal, during which the lead dancer performed one of her solos.  I was captivated by how intensely personal and intimate the experience was, as if she was sharing with us something extremely private.  This idea was reinforced by the fact that she never broke eye contact with herself in the large studio mirror.  This was an important conceptual leap forward, as it brought forth the idea of "experiencing" a portrait through the eyes of the subject by ways of a mirror.  From this point onward, the mirror theme became an integral part of the concept.  My plan was to capture the studio using photogrammetry and record the dancer's solo using the depthkit.  Unfortunately, on the scheduled day of shooting, the dancer injured herself at a rehearsal earlier that morning and was unable to perform for me.  That was the end of subject 2.

Subject 3: A proof of concept

Unfortunately by this stage I was running short of time and did not have the luxury of finding another subject and curating the experience of their portrait.  Wanting to continue with the theme of performing, or rehearsing, I instead asked a friend here at ITP if he wouldn't mind performing for me while I recorded him as he played guitar and sang to himself.  I set about recording a few takes using the DepthKit setup and photoscanned a section of the ITP floor that I felt could represent a generic studio space.


I took a series of photographs and imported them into PhotoScanPro.  I used the software to generate an .FBX file that I then imported into Unity.  Once I had the environment setup, complete with custom lighting, I set about rendering out a series of stills using the DepthKit Visualizer software.  After converting these files to an .mp4 using the custom shell script, I imported the necessary files into Unity.

The next step of the process was creating the custom mirror inside of Unity.  I found a shader in the Unity forum that was based on the built-in water reflection shader and modified it ever so slightly to suit my needs.  I applied the shader to a plane that I created, and placed it in my scene up against a wall, as though it were a mirror.

I created an empty game object to serve as my character controller.  I attached my video object to this, and created a camera with a slightly offset position, creating the illusion of first-person perspective.  By placing the camera slightly in front of the video object, I was able to create the illusion of the video object, or rather my subject, being reflected in the mirror.  The video below illustrates the proof of concept:

Moving forward:

While I feel that this demonstration serves as a proof of concept, I am far from happy with the final result, and I am choosing rather to view it as an initial step.  For one thing, I'm unhappy with the quality of the reflected image.  The reason for this is threefold: the built-in video player for Unity is decidedly unimpressive, as is the shader I adapted.  In addition to this, the depth kit shader for Unity has been written to display the full texture of the video, while I am aiming for a more minimal aesthetic.

Given that nothing I am trying to achieve expressly requires Unity, the next step in the process is using Cinder to create my intimate self-portrait experience.  This will require the adaption of the DepthKit oF shader, as well as writing a custom mirror shader.  I also want to utilise the Occulus Rift to turn it into a VR experience, thus enabling the user to embody the subject of the portrait, by viewing the portrait as a reflection in the mirror, instead of just on a screen.