Art, craft and design

In search of the super mind…

Archive for November 2011

Pictures and their meaning

leave a comment »

 This is the second post in the series Pictures, people and machines.

  1. Pictures, People and Machines
  2. Pictures and their meaning

.

.

For what a picture’s worth

An interesting perspective, to eye catchy colours of an abstract, contrast and the grain of a black and white, to photographs of our friends and family; we have several reasons to click and keep photos. As part of the project we did for Intel, we brainstormed around the various roles a photograph plays in our lives. [the following map tries to map out the objective of photos!] The significance of this exercise was not only to create such a map but to find areas where interventions are possible, –not to mention, Intel was interested in distributing the computing power of [their] processors to creation and consumption part of the spectrum as well, where as now the intermediate editing stage consumes most of the processing power of a computer.

These first thoughts were the reason I started thinking about the difference between what machines see and what humans see in a photo. With the advancements in computer vision, machines can detect and find objects, people and their faces; track objects in a video stream, estimate the 3D pose of an object etc. These algorithms still fall short when it comes to making sense of a photograph or seeing the symbolic meaning it has! Perhaps one of the nicer things computers can do is to recognise the people, objects or space [symbols] that people see in a photograph and try to augment the static photograph in a meaningful way.

A meaningful way

“We are in a picture together. We are in the same place and time and I am keeping this picture to remember that. We are family, friends. Remember that time when we took the road trip?…”. Photographs more than anything captures a moment in time. Frozen in that moment are our family and friends, things that we care about. And that is so important to think about the time itself, because the meaning of that photograph depends so much on that moment it was captured. People and objects age and what they mean to us may change over time. Moreover a photo can take us instantly back in time, bring back memories and lets us tell the story.

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects?

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects? A meaningful way to explore these questions could be to explore the scenarios assuming such an ageing photograph exists. This explorations could feed itself back into designing something tangible to answer those questions. There are certain innate qualities of a photo that has to be maintained, the context, relationships between the subjects and the aesthetics of the image itself. Also this new medium derived from a static photo will have some new qualities as well, how would the subjects age and change, what are the interactions possible with it when it changes?

One possible way is to think photographs as a collage –a collage of people, things and the space. Then introduce the concept of ageing, changing time. And recreate this collage as there are new photos taken of the same subjects, by recognising them and introducing them into the former picture. Just like making a collage from magazine or newspaper cuttings! The meaning and context of the old picture more or less remains unchanged, but the people and things in the photo is how they are now!

Years later…

“At CIID we are nearing the end of the course and everyone toiling away on their final projects. People are making their prototypes and codifying mobile apps, laser-cutting planks of wood and plastic and making boxes, assembling electronics to fill those boxes, shooting videos and preparing the presentations to explain the concepts.  There is a quiet feeling that lingers. I think we are closer to each other than ever. There will be a photo shoot in a few weeks when all of these end, may be at our graduation ceremony. After the ceremony and party everyone will go home. Months and years would pass and we will get on with our lives and careers.

Our group photo changed over time as we changed. Everyone kept the photograph with them, it is on the school’s website. Occasionally I see someone in a wedding dress or covered in confetti at a birthday party. It brings back so many memories. My friends haven’t changed even a little in my imagination, but they have changed so much, I know, I can see that in the photo. It is now a wonderful collage of everyone’s pictures overlapping each other. I kept it on my phone and running my finger across its screen, I can go through a small slice of everyone’s past all the way to the day the photo was taken in Copenhagen. Everyone’s doing well. Usually coming across this old photo ends up in me texting or calling someone or visiting their Facebook page.”

More than just pixels

Moving away from the technical and implementation details was helpful in developing on the core idea and how it would affect the people and their behaviour. I did various experiments that I imagined would help me in implementing the concept without losing the experience of the basic idea. These experiments included explorations into gestural interfaces, the aesthetics of an ageing photograph, and explorations into various computer vision algorithms for extracting people and objects from photographs and identifying them.

The above image explains the direction I chose towards further technical exploration and implementation. I am using Java with the help of Processing framework and some 3rd party libraries to make a prototype. OpenCV is a great toolkit for computer vision. With quite a bit of help from JavaCV I got C and Java talking to each other. JavaScript could be valid option to code the user interface; Processing.js project would make my life a lot easier. Besides then I can make a web page showcasing some of these ideas and get some feedback.

Few key learnings from the original set of explorations were presented to Intel. And the idea has come a long way ever since I started my solo project, with the kind permission from my team-mate Wan-Ting to take this idea further and explore it.