Art, craft and design

This is the second post in the series Pictures, people and machines. Pictures, People and Machines Pictures and their meaning .

.

For what a picture’s worth

An interesting perspective, to eye catchy colours of an abstract, contrast and the grain of a black and white, to photographs of our friends and family; we have several reasons to click and keep photos. As part of the project we did for Intel, we brainstormed around the various roles a photograph plays in our lives. [the following map tries to map out the objective of photos!] The significance of this exercise was not only to create such a map but to find areas where interventions are possible, –not to mention, Intel was interested in distributing the computing power of [their] processors to creation and consumption part of the spectrum as well, where as now the intermediate editing stage consumes most of the processing power of a computer.

These first thoughts were the reason I started thinking about the difference between what machines see and what humans see in a photo. With the advancements in computer vision, machines can detect and find objects, people and their faces; track objects in a video stream, estimate the 3D pose of an object etc. These algorithms still fall short when it comes to making sense of a photograph or seeing the symbolic meaning it has! Perhaps one of the nicer things computers can do is to recognise the people, objects or space [symbols] that people see in a photograph and try to augment the static photograph in a meaningful way.

A meaningful way

“We are in a picture together. We are in the same place and time and I am keeping this picture to remember that. We are family, friends. Remember that time when we took the road trip?…”. Photographs more than anything captures a moment in time. Frozen in that moment are our family and friends, things that we care about. And that is so important to think about the time itself, because the meaning of that photograph depends so much on that moment it was captured. People and objects age and what they mean to us may change over time. Moreover a photo can take us instantly back in time, bring back memories and lets us tell the story.

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects?

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects? A meaningful way to explore these questions could be to explore the scenarios assuming such an ageing photograph exists. This explorations could feed itself back into designing something tangible to answer those questions. There are certain innate qualities of a photo that has to be maintained, the context, relationships between the subjects and the aesthetics of the image itself. Also this new medium derived from a static photo will have some new qualities as well, how would the subjects age and change, what are the interactions possible with it when it changes?

One possible way is to think photographs as a collage –a collage of people, things and the space. Then introduce the concept of ageing, changing time. And recreate this collage as there are new photos taken of the same subjects, by recognising them and introducing them into the former picture. Just like making a collage from magazine or newspaper cuttings! The meaning and context of the old picture more or less remains unchanged, but the people and things in the photo is how they are now!

Years later…

“At CIID we are nearing the end of the course and everyone toiling away on their final projects. People are making their prototypes and codifying mobile apps, laser-cutting planks of wood and plastic and making boxes, assembling electronics to fill those boxes, shooting videos and preparing the presentations to explain the concepts.  There is a quiet feeling that lingers. I think we are closer to each other than ever. There will be a photo shoot in a few weeks when all of these end, may be at our graduation ceremony. After the ceremony and party everyone will go home. Months and years would pass and we will get on with our lives and careers.

Our group photo changed over time as we changed. Everyone kept the photograph with them, it is on the school’s website. Occasionally I see someone in a wedding dress or covered in confetti at a birthday party. It brings back so many memories. My friends haven’t changed even a little in my imagination, but they have changed so much, I know, I can see that in the photo. It is now a wonderful collage of everyone’s pictures overlapping each other. I kept it on my phone and running my finger across its screen, I can go through a small slice of everyone’s past all the way to the day the photo was taken in Copenhagen. Everyone’s doing well. Usually coming across this old photo ends up in me texting or calling someone or visiting their Facebook page.”

More than just pixels

Moving away from the technical and implementation details was helpful in developing on the core idea and how it would affect the people and their behaviour. I did various experiments that I imagined would help me in implementing the concept without losing the experience of the basic idea. These experiments included explorations into gestural interfaces, the aesthetics of an ageing photograph, and explorations into various computer vision algorithms for extracting people and objects from photographs and identifying them.

The above image explains the direction I chose towards further technical exploration and implementation. I am using Java with the help of Processing framework and some 3rd party libraries to make a prototype. OpenCV is a great toolkit for computer vision. With quite a bit of help from JavaCV I got C and Java talking to each other. JavaScript could be valid option to code the user interface; Processing.js project would make my life a lot easier. Besides then I can make a web page showcasing some of these ideas and get some feedback.

Few key learnings from the original set of explorations were presented to Intel. And the idea has come a long way ever since I started my solo project, with the kind permission from my team-mate Wan-Ting to take this idea further and explore it.

A few months ago http://1000memories.com published an insightful article titled “How many photos have ever been taken?”. The article collected and presented various statistics related to digital and analog photography, like camera ownership and number of photographs taken per year. Amidst all this digital image influx, the article points out, how certain pictures are special to us –like those which we keep in the shoebox at home.

The technical paradox

The relationship between technology and life has been well discussed topic. Many of the technological artefacts that we use today produce such huge amounts of data, that we need to seek help from the technology itself to consume its outcome. We try to build more intelligent objects and networks to edit, organise and filter the noise out of this data and easier and simpler to understand.

Going back to digital photographs, there are myriad ways that we create, edit and share or store them. Not just as probably the most popular form of social media, we click and keep photographs that are personal to us also. We print them out and frame it, keep it in that shoebox at home. What has changed since humanity started clicking. We don’t have to look that far back to see how it evolved from a hired photographer crafting your family portrait, to you pointing and clicking the Kodak moments and rushing to the photo studio to develop the film-roll, to clicking that funny snap of your cat shredding toilet paper and share it on your favourite cat-meme social[?] network!

These steps itself has not been evolved much from the analog days of photography. Some steps are definitely new but it mostly replaces an analogous process from the old days, such as film development. The result of these are the same still imagery but from chemicals capturing and holding visual information it has evolved into digital bits [mostly Red Green and Blue] with some meta-data embedded in the digital picture file. These meta-data usually contain the technical details of the equipment used to click the photograph and the details of the photograph itself. In case of a Geo-tagged photo, it can contain the latitude and longitude of the place where the photograph was taken. Can it contain symbolic information? Can the technology involved would find or understand the symbolic information contained in a photograph?

The meaning

But can it contain symbolic information? Can the technology involved would identify or understand what the photograph mean to a person?

If technology could understand, the implications could be big. Not only it could completely transform many of the steps involved in the process, also could bring about new ways of capturing and using photographs as well. The simplicity and ease of sharing photographs on various social networks have changed the way we plan, shoot and share. With the advent of digital photography, the cost involved in capturing a photograph went down dramatically; and with cameras having more and more storage capacity, most of us continue to click in large numbers, hoping that some of them might turn out to be worthy of posting on our walls. From being the record of a special moment, a photograph have come to mean a lot more as the process and equipment went cheaper and easier to use.

In order to explore how a machine can find the semantic meaning of a photograph, we might need to understand how do people see photographs? How do we arrange them and tell a story? What do we look for when we want to find a photograph? These questions may seem quite obvious. But since machine intelligence as of today understands mainly if not only quantitative data about the world that surrounds it, we need to investigate the questions mentioned above and identify those quantifiable aspects of the symbols that a photograph might contain.

R G B et cetera

Structurally a photograph [digital] is composed of pixels which in turn is specified by Red, Green and Blue numerical values. This information can be augmented to bring out more detail and make the image richer. Moreover, what beyond the RGB can each pixel contain which are relevant to the symbolic meaning of a photograph? And even if the photographs can contain such extra information, how could a machine make sense of it?

Sense?

 Human intelligence is the result of a multitude of interactions that happen inside our brain. The individual neurones does not contain information or process a piece of information to make sense. However each of them are extensively connected with other neurones, and forms a very large network whose interactions result in consciousness and intelligence. Kevin Kelly in his book What technology wants compares the interplanetary electronic membrane to the human brain in complexity. We share millions of photographs on this electronic network each year. The possibilities of the complex networked information and anecdotes associated with photographs on the web might belie the visual information contained in those photographs itself. These information could be of the curated type or the connections that build the network itself. Tags, names, URL, location data, other meta-data etc. falls into the former classification, where humans themselves largely provide these information. The text that appears along with a picture in a blog or the comments beneath the picture, the tone of those comments, how many have liked or linked to a certain picture, the information about these people, what other pictures did they like etc. falls into the second type where the information is gathered by examining the nodes of the network itself.

Just like a human brain make sense of the world around it by the countless networked interactions, can machines [or the digital network as an entity] make sense of a photograph by collecting and correlating information within the image as well as the way it is networked or shared?

Just like a human brain make sense of the world around it by the countless networked interactions, can machines [or the digital network as an entity] make sense of a photograph by collecting and correlating information within the image as well as the way it is networked or shared? A number of technologies and algorithms that exists together could deal with such a task. However there are certain infrastructural challenges associated with this as well. One of the main complications can be of the availability of the data itself. Not all the social media networking platforms have an open data policy. Privacy concerns may be one of the reasons for this issue.

Living Images

A project done by me and Wan-Ting Liao at Copenhagen Institute of Interaction Design for Intel called Living Images, tried to explore certain aspects of the ideas discussed above. The project explored the idea of adding time as a factor along with RGB for pixels in a photograph. Regions of a photograph ages just as we age and changes in appearance as the people, objects or space in the photograph ages. We explored the aesthetics of such an image and how it imparts the meaning of the original image, scenarios where such a photograph has an impact and the feeling itself when someone navigates through the changes happened to the photograph’s subject in time.

So what?

Our explorations were aimed at provoking a discussion, rather than designing an artefact. My topic for the final project since has changed its direction based on the thoughts presented here. They could be summarised as follows.

• What do we see in a photograph and why are some more special to us? memories and tokens, the people, objects, space moment?
• Why do we share certain photographs? alternate online personality, preserve memory, documentation, history, explanation or annotation?
• Can we distill the answers to the questions above at least to a certain degree so that our digital companions can make sense of the photographs that we click, keep and share?
• –If they can, what new insightful services or digital artefacts can be designed or how the present services or devices can be modified?