This is the second post in the series Pictures, people and machines.

1. Pictures, People and Machines
2. Pictures and their meaning

.

.

For what a picture’s worth

An interesting perspective, to eye catchy colours of an abstract, contrast and the grain of a black and white, to photographs of our friends and family; we have several reasons to click and keep photos. As part of the project we did for Intel, we brainstormed around the various roles a photograph plays in our lives. [the following map tries to map out the objective of photos!] The significance of this exercise was not only to create such a map but to find areas where interventions are possible, –not to mention, Intel was interested in distributing the computing power of [their] processors to creation and consumption part of the spectrum as well, where as now the intermediate editing stage consumes most of the processing power of a computer.

These first thoughts were the reason I started thinking about the difference between what machines see and what humans see in a photo. With the advancements in computer vision, machines can detect and find objects, people and their faces; track objects in a video stream, estimate the 3D pose of an object etc. These algorithms still fall short when it comes to making sense of a photograph or seeing the symbolic meaning it has! Perhaps one of the nicer things computers can do is to recognise the people, objects or space [symbols] that people see in a photograph and try to augment the static photograph in a meaningful way.

A meaningful way

“We are in a picture together. We are in the same place and time and I am keeping this picture to remember that. We are family, friends. Remember that time when we took the road trip?…”. Photographs more than anything captures a moment in time. Frozen in that moment are our family and friends, things that we care about. And that is so important to think about the time itself, because the meaning of that photograph depends so much on that moment it was captured. People and objects age and what they mean to us may change over time. Moreover a photo can take us instantly back in time, bring back memories and lets us tell the story.

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects?

Do meaning and context of a photo change if the photograph aged with its subjects? Can photographs age gracefully with its subjects? A meaningful way to explore these questions could be to explore the scenarios assuming such an ageing photograph exists. This explorations could feed itself back into designing something tangible to answer those questions. There are certain innate qualities of a photo that has to be maintained, the context, relationships between the subjects and the aesthetics of the image itself. Also this new medium derived from a static photo will have some new qualities as well, how would the subjects age and change, what are the interactions possible with it when it changes?

One possible way is to think photographs as a collage –a collage of people, things and the space. Then introduce the concept of ageing, changing time. And recreate this collage as there are new photos taken of the same subjects, by recognising them and introducing them into the former picture. Just like making a collage from magazine or newspaper cuttings! The meaning and context of the old picture more or less remains unchanged, but the people and things in the photo is how they are now!

Years later…

“At CIID we are nearing the end of the course and everyone toiling away on their final projects. People are making their prototypes and codifying mobile apps, laser-cutting planks of wood and plastic and making boxes, assembling electronics to fill those boxes, shooting videos and preparing the presentations to explain the concepts.  There is a quiet feeling that lingers. I think we are closer to each other than ever. There will be a photo shoot in a few weeks when all of these end, may be at our graduation ceremony. After the ceremony and party everyone will go home. Months and years would pass and we will get on with our lives and careers.

Our group photo changed over time as we changed. Everyone kept the photograph with them, it is on the school’s website. Occasionally I see someone in a wedding dress or covered in confetti at a birthday party. It brings back so many memories. My friends haven’t changed even a little in my imagination, but they have changed so much, I know, I can see that in the photo. It is now a wonderful collage of everyone’s pictures overlapping each other. I kept it on my phone and running my finger across its screen, I can go through a small slice of everyone’s past all the way to the day the photo was taken in Copenhagen. Everyone’s doing well. Usually coming across this old photo ends up in me texting or calling someone or visiting their Facebook page.”

More than just pixels

Moving away from the technical and implementation details was helpful in developing on the core idea and how it would affect the people and their behaviour. I did various experiments that I imagined would help me in implementing the concept without losing the experience of the basic idea. These experiments included explorations into gestural interfaces, the aesthetics of an ageing photograph, and explorations into various computer vision algorithms for extracting people and objects from photographs and identifying them.

The above image explains the direction I chose towards further technical exploration and implementation. I am using Java with the help of Processing framework and some 3rd party libraries to make a prototype. OpenCV is a great toolkit for computer vision. With quite a bit of help from JavaCV I got C and Java talking to each other. JavaScript could be valid option to code the user interface; Processing.js project would make my life a lot easier. Besides then I can make a web page showcasing some of these ideas and get some feedback.

Few key learnings from the original set of explorations were presented to Intel. And the idea has come a long way ever since I started my solo project, with the kind permission from my team-mate Wan-Ting to take this idea further and explore it.

The technical paradox

The relationship between technology and life has been well discussed topic. Many of the technological artefacts that we use today produce such huge amounts of data, that we need to seek help from the technology itself to consume its outcome. We try to build more intelligent objects and networks to edit, organise and filter the noise out of this data and easier and simpler to understand.

Going back to digital photographs, there are myriad ways that we create, edit and share or store them. Not just as probably the most popular form of social media, we click and keep photographs that are personal to us also. We print them out and frame it, keep it in that shoebox at home. What has changed since humanity started clicking. We don’t have to look that far back to see how it evolved from a hired photographer crafting your family portrait, to you pointing and clicking the Kodak moments and rushing to the photo studio to develop the film-roll, to clicking that funny snap of your cat shredding toilet paper and share it on your favourite cat-meme social[?] network!

These steps itself has not been evolved much from the analog days of photography. Some steps are definitely new but it mostly replaces an analogous process from the old days, such as film development. The result of these are the same still imagery but from chemicals capturing and holding visual information it has evolved into digital bits [mostly Red Green and Blue] with some meta-data embedded in the digital picture file. These meta-data usually contain the technical details of the equipment used to click the photograph and the details of the photograph itself. In case of a Geo-tagged photo, it can contain the latitude and longitude of the place where the photograph was taken. Can it contain symbolic information? Can the technology involved would find or understand the symbolic information contained in a photograph?

The meaning

But can it contain symbolic information? Can the technology involved would identify or understand what the photograph mean to a person?

If technology could understand, the implications could be big. Not only it could completely transform many of the steps involved in the process, also could bring about new ways of capturing and using photographs as well. The simplicity and ease of sharing photographs on various social networks have changed the way we plan, shoot and share. With the advent of digital photography, the cost involved in capturing a photograph went down dramatically; and with cameras having more and more storage capacity, most of us continue to click in large numbers, hoping that some of them might turn out to be worthy of posting on our walls. From being the record of a special moment, a photograph have come to mean a lot more as the process and equipment went cheaper and easier to use.

In order to explore how a machine can find the semantic meaning of a photograph, we might need to understand how do people see photographs? How do we arrange them and tell a story? What do we look for when we want to find a photograph? These questions may seem quite obvious. But since machine intelligence as of today understands mainly if not only quantitative data about the world that surrounds it, we need to investigate the questions mentioned above and identify those quantifiable aspects of the symbols that a photograph might contain.

R G B et cetera

Structurally a photograph [digital] is composed of pixels which in turn is specified by Red, Green and Blue numerical values. This information can be augmented to bring out more detail and make the image richer. Moreover, what beyond the RGB can each pixel contain which are relevant to the symbolic meaning of a photograph? And even if the photographs can contain such extra information, how could a machine make sense of it?

Sense?

 Human intelligence is the result of a multitude of interactions that happen inside our brain. The individual neurones does not contain information or process a piece of information to make sense. However each of them are extensively connected with other neurones, and forms a very large network whose interactions result in consciousness and intelligence. Kevin Kelly in his book What technology wants compares the interplanetary electronic membrane to the human brain in complexity. We share millions of photographs on this electronic network each year. The possibilities of the complex networked information and anecdotes associated with photographs on the web might belie the visual information contained in those photographs itself. These information could be of the curated type or the connections that build the network itself. Tags, names, URL, location data, other meta-data etc. falls into the former classification, where humans themselves largely provide these information. The text that appears along with a picture in a blog or the comments beneath the picture, the tone of those comments, how many have liked or linked to a certain picture, the information about these people, what other pictures did they like etc. falls into the second type where the information is gathered by examining the nodes of the network itself.

Just like a human brain make sense of the world around it by the countless networked interactions, can machines [or the digital network as an entity] make sense of a photograph by collecting and correlating information within the image as well as the way it is networked or shared?

Just like a human brain make sense of the world around it by the countless networked interactions, can machines [or the digital network as an entity] make sense of a photograph by collecting and correlating information within the image as well as the way it is networked or shared? A number of technologies and algorithms that exists together could deal with such a task. However there are certain infrastructural challenges associated with this as well. One of the main complications can be of the availability of the data itself. Not all the social media networking platforms have an open data policy. Privacy concerns may be one of the reasons for this issue.

Living Images

A project done by me and Wan-Ting Liao at Copenhagen Institute of Interaction Design for Intel called Living Images, tried to explore certain aspects of the ideas discussed above. The project explored the idea of adding time as a factor along with RGB for pixels in a photograph. Regions of a photograph ages just as we age and changes in appearance as the people, objects or space in the photograph ages. We explored the aesthetics of such an image and how it imparts the meaning of the original image, scenarios where such a photograph has an impact and the feeling itself when someone navigates through the changes happened to the photograph’s subject in time.

So what?

Our explorations were aimed at provoking a discussion, rather than designing an artefact. My topic for the final project since has changed its direction based on the thoughts presented here. They could be summarised as follows.

• What do we see in a photograph and why are some more special to us? memories and tokens, the people, objects, space moment?
• Why do we share certain photographs? alternate online personality, preserve memory, documentation, history, explanation or annotation?
• Can we distill the answers to the questions above at least to a certain degree so that our digital companions can make sense of the photographs that we click, keep and share?
• –If they can, what new insightful services or digital artefacts can be designed or how the present services or devices can be modified?

The Hive

The scope that cities have is huge in this respect, especially in the modern context where every piece of public object is networked. These public objects can be considered as the senses of the city, and a cities senses spread across a wider geographic area. ATMs, Parking meters, surveillance cameras, weather sensors, traffic lights and cameras, ticket vending, electric meters, telephone and OFC routers the list of the networked objects existing in the public space is quite long. These information exists in multiple layers, and while certain layers spread across a relatively smaller area of the city or the entire city while others span across different cities across continents. Most of these objects are designed and installed in place usually for singular purposes. Life span of a city is considerably longer compared to us and it would be interesting to know how these networked information could amass from such a large scope and be distilled.

Not just these public objects, the people themselves play roles. The swarm of bicyclists leaving for work every morning and teeming the streets, or pedestrians, buses, cars, delivery trucks, police cars, fire trucks, garbage trucks… There may be several reasons or authorities and rules behind how these seemingly single purposed swarms form in a city; nevertheless it would be quite interesting to investigate what emergent behaviour comes out of these different agents collectively.

The case of the Super-mind

Kevin Kelly in his book “Out of control” discusses different aspects of a collective consciousness, what he calls a hive-mind. While swarm intelligence is well inspired from, especially in fields like simulated intelligence, there are more intriguing behaviours these systems exhibit. In the second chapter of his book Kelly explains this with an ant colony as an example–

“Ants, too, have hive mind. A colony of ants on the move from one nest site to another exhibits the Kafkaesque underside of emergent control. As hordes of ants break camp and head west, hauling eggs, larva, pupae — the crown jewels — in their beaks, other ants of the same colony, patriotic workers, are hauling the trove east again just as fast, while still other workers, perhaps acknowledging conflicting messages, are running one direction and back again completely empty-handed. A typical day at the office. Yet, the ant colony moves. Without any visible decision-making at a higher level, it chooses a new nest site, signals workers to begin building, and governs itself.

The marvel of “hive mind” is that no one is in control, and yet an invisible hand governs, a hand that emerges from very dumb members. The marvel is that more is different. To generate a colony organism from a bug organism requires only that the bugs be multiplied so that there are many, many more of them, and that they communicate with each other. At some stage the level of complexity reaches a point where new categories like “colony” can emerge from simple categories of “bug.” Colony is inherent in bugness, implies this marvel. Thus, there is nothing to be found in a beehive that is not submerged in a bee. And yet you can search a bee forever with cyclotron and fluoroscope, and you will never find the hive.”

Us, human beings are a very social species. In fact we are still searching for our own socialness and the ever more urgent need to interconnect with each other through every mode of communication that exists, using almost every piece of technology that we know of. yet it is quite uncertain that if we posses a hive mind; we definitely do colonise, we have networked systems for communication, but still we trust in hierarchical models for making decisions and foreseeing the future of our hives.

“Wheeler claimed that an insect colony was not merely the analog of an organism, it is indeed an organism, in every important and scientific sense of the word. He wrote: “Like a cell or the person, it behaves as a unitary whole, maintaining its identity in space, resisting dissolution…neither a thing nor a concept, but a continual flux or process.”"

The search for our bugness within

Is the consciousness of the city the hive mind? Do we posses such a hive mind? Are we too complex as an organism to have such an effective super-mind for our own good?! What advantages do we have? What inherent within us might work against such a system? What continual flux or process do human colonies –cities have? examples? If such mindless creatures can do so much collectively, have a collective mind for the entire colony, what can we accomplish if we human beings had such a collective super mind?

While exploring these questions in a more general fashion, may be it is a good idea to focus on one city, one hive at a time. Copenhagen.

electric lantern on a malayalam magazine

brushes app on iPhone.

Brushes app on iPhone.

And here is the video of the process..

Brushes app on iPhone 3G.

Edit: Vimeo seems to be better to upload videos in portrait format. Youtube doesn’t do that!? is a lot neater as well.

Brushes on iPhone 3G.

Few days ago I downloaded the famed Brushes app. It has fewer features than the Sketchbook-mobile app, but it’s responsive and fast on my outdated smart phone. Besides I like the video feature in Brushes app. Later maybe I’ll share my painting process (so that you can tell me how I can improve! )

Brushes app on iPhone 3G (duh!).