Reflections on Click! by James Surowiecki

Much of the critical reception of Click! has focused, understandably, on the artistic quality of the photographs that the crowd liked best, with a number of critics making predictably dismissive comments about the mainstream nature of the favored pictures (and of the show as a whole). This take on Click! fits squarely into a long tradition of art-historical arguments around mass vs. elite taste, connoisseurship, and so on. And these kinds of arguments are inevitable—you can’t (nor would you want to) look at an art show and not make judgments about the quality of the work in it. But I do think that some of the critical reactions to the show have been inflected by a pre-existing assumption that group judgments would necessarily be ordinary. And that assumption has perhaps made it harder to see a couple of the most interesting things about the show. The first thing that I think makes Click! so intriguing for the future is the way the voting system worked. One of the reasons why group judgments are often so mediocre (or so volatile) is that they often rely on very crude evaluation processes, processes that foster bandwagon effects—when things get more popular simply because they’re already popular—and that are often easily gamed. A classic example would be American Idol, where members of the voting crowd generally register their opinion on only one contestant —the one they vote for—and where people can vote as many times as they want for the same person.

Click!’s system, by contrast, was designed to get around these problems. Because people could only rate each picture once, and because they couldn’t go directly to a given photo, the system was hard (if not impossible) to game. Even more important, because people didn’t know how others had voted, each person’s rating reflected his or her own judgment, uninfluenced by the opinion of others. That independence of judgment is key to the wisdom of crowds. And since the contest was run over the Net, it was also able to tap the knowledge of a relatively diverse crowd, both in terms of location and expertise. It’s true that not every person who voted looked at every picture in the contest (in fact, no one outside the museum may have done that). But because the crowd was diverse enough and big enough (so that a sizeable number of people did look at each photo), and because the selection that each person looked at was random, this probably had little or no effect on the final outcome – effectively, the result is similar to what it would have been had everyone looked at every picture. Unlike most attempts to measure popular taste, then, Click! reflects the real collective judgment of the crowd.

But is that collective judgment wise? Well, there’s no real way to answer that, since there’s no objective standard to measure the crowd’s judgment against. (That’s one reason I deliberately avoided writing about art in The Wisdom of Crowds.) But I do think it’s intriguing that there was so much overlap in Click! between the crowd’s judgment and the judgment of the experts. I think if you’d asked most people before the show, they would have said that there would have been a massive difference between the favorite choices of a diverse crowd of people and the favorite choices of people with artistic or art-historical backgrounds. But when you look at the top ten favorites of voters as a whole, and at the favorites of the different subgroups, what you find is that they’re actually not that different. Many photographs show up on all or most of the lists. To me, this is really the most striking result of the show, because it suggests (though it doesn’t prove) that at least in some mediums, the gap between popular and elite taste may be smaller than we think. It also, I think, suggests that the places for tapping the collective intelligence of diverse crowds are wider than we might imagine.

Crowd-Curated or Crowd-Juried?

After I was recently asked to write a blog posting about Click! from my perspective, I spoke with some of my colleagues—Patrick Amsellem, Associate Curator of Photography, and Judy Kim, Curator of Exhibitions--about the exhibition, and we began to discuss the term “curated.” The three of us have all been curators of exhibitions in our careers, but we have also all been on juries for exhibitions, and we think that perhaps what the crowd was asked to do here was to jury the selection—that is, to rank the works that were submitted so that a selection could be made on the basis of that ranking. That is sometimes the first step toward curating an exhibition, but only the first step. Once an initial selection is made, the curator usually begins to refine the idea of the exhibition and to see how the ideas represented by the objects selected best work together, and how placing certain works side by side, or across the room from one another can have an impact on the way we perceive them, and thus help to advance the theme and the learning experience. Further “curating” is done by explaining in written form in the labels some of the ideas the installation conveys visually. So if the crowd juried the images, how was it curated? And what was the idea curated? The theme of the photographs submitted was “The Changing Faces of Brooklyn,” but that is not the theme of the installation that is presented in our galleries. Although the changing faces of Brooklyn is an idea that underlies each of the works of art in the exhibition, the exhibition itself is about the notion of selection, and, specifically, selection by the crowd. Both visually, and in terms of its written didactics, the installation supports that primary idea, put forward—that is, curated—by Shelley Bernstein. And that is what makes it such an unusual and thought provoking exhibition. It seems to me that an exhibition that was only about the changing faces of Brooklyn in photography might be interesting, but an exhibition that is also about the nature of selection, and all the questions it raises about taste, background, interpretation—and a myriad of other issues—creates a richer discourse. In Click! the theme of the work and the selection and installation process complement and mutually reinforce one another, forming a compelling snapshot of who we are and how we chose.

I am interested to know what others think about this, and I will be taking questions on the subject this week.

Information Cascade!

I couldn't help but laugh last Saturday night. Click! was packed—clearly, there was an information cascade happening on the second floor of the Brooklyn Museum. Now that we are open, we are happy to have a cascade like that. Thanks for turning out for such a fun night. It was wonderful for me to get the chance to meet so many of the Click! photographers and evaluators and a lot of folks who were finding Click! for the first time.

Photo by Donna Aceto, who saved the day when my camera battery died. Thanks, Donna.

Photo by Donna Aceto, who saved the day when my camera battery died. Thanks, Donna.

In other Click! news, I wanted to note that we've just posted a podcast from the panel discussion. I couldn't have been happier spending a morning with this group and hope you find it worth a listen.

Lastly, I was poking around Flickr this past weekend and this photo by mercurialn (a.k.a. Nate Dorr) brought a big smile to my face, so I had to share. BTW, Nathan Kensinger (pictured bottom right) has a exhibition up at Brooklyn Public Library. Be sure to catch it before it closes August 30.

Click! is not a contest...

...it is a study in crowds. It should come as no surprise that this title made it into one of my blog posts (there are many colleagues of mine chuckling about this right about now). That said, we are releasing the Click! website today and you'll find it is designed much like the gallery. Images are displayed by size relative to each other given the query you are looking at—lists and scores (not something we equate with subjective subjects) are avoided. Some images fared better than others, but it's all relative depending on what you are looking at and what questions you are asking. All 389 images are on the website. In addition, the 78 images that will be in the gallery can now be viewed—see "In the Gallery" on the Click! menu. Keep in mind, the sizes on the website are more variable than the sizes in the gallery, so if you see your image in that 78, the sizing in the gallery will be slightly different.

Coming up we've got more guests writing for the blog, including two of our consultants, James Surowiecki and Derek Powazek. Our own Chief Curator, Kevin Stayton, will be posting in the coming weeks and we may have a few more surprises along the way. For now, check out the website. The results button on the right side of the page will take you there. Bear with us if you find it slow, we are expecting a bit of traffic today. Need a little introduction? Check out the lo-fi screencast.Click! will be on view beginning this Friday (June 27) and we are making good progress installing the show (see below)!

And now, one quick note of thanks about this website. I count myself lucky that I work with an incredibly dedicated and talented team who loved this project as much as I did and made sure Click! was awesome via the web. Mike, Paul, Jen - I can't say this enough: you are awesome, you rock my world and this site is rockin' because of your dedication. Jessica, Joe - you make life around here fun, easy, and accurate to boot. It's a pleasure to come to work every day—thank you.

Lance and Tomoko hanging the show. We are about half way through.

Preparing to Click!

With the opening of Click! rapidly approaching, I have been asked to describe my approach in designing and mounting this particular exhibition. It is important to note that although this exhibition is comprised entirely of photographs, it is not foremost a photography show, but rather an art installation addressing the conceptual nature of a crowd-curated exhibition. For this reason, the show will not be hung in a traditional manner, but rather laid out in a way that illustrates the diverse, anonymous, web-based crowd selection process. The exhibition is being held in an intimate gallery space to allow the viewer to be immersed in the images of the “changing faces of Brooklyn.” Because of the gallery's space constraints it was determined that of the 389 images submitted, the top-ranked 20% of images would be printed for display. Of this 20% (or roughly 78 photographs) a distinction was required to reflect the crowds selection process; for this reason some photographs were printed as large as 20” x 30” and some as small as 5” x 7”, depending upon the crowd's rankings.

Rough layout of one of the gallery walls.

Rough layout of one of the gallery walls.

When these images of varying size are displayed in a random arrangement it serves to illustrate the crowds' selection process not as linear, but rather a diverse response with certain ideas or, in this case, photographs rising to the front of a collective conscious in much the way a tag cloud uses text to visually illustrate how within many voices certain responses carry varying degrees of impact.

Example of tag cloud via Flickr.

Example of tag cloud via Flickr.

The exhibition promises to be interesting on many levels, and I hope all that participated both in submitting photographs and those involved in the selection process can make it to see the results.

Defining Face, Change, and Brooklyn in Click!

As a visual sociologist looking at the images as to how people define "face," "change," and "brooklyn," I was very impressed with the quality and array of images submitted for the Click! A Crowd-Curated Exhibition. I can see that these photographers at least regard externalities as "face" as though it was "skin" surrounding some other content; e.g., there were few photos inside places and spaces. "Change" seemed to be mostly about physical structures as opposed to people and social activities. It also seemed to me that the photographers were showing something of which they didn't approve. "Brooklyn" was represented geographically in a very limited way with a concentration on some of the most "Brooklyn-branded" of spaces such as Coney Island and Carnival. It might be also that the focus on places like Red Hook, Dumbo, and Williamsburg reflects where artists (photographers in this case) are living or hanging around. An aspect of Brooklyn's growing "Creative Class" perhaps. Due to the choice of venues, it gave the impression that perhaps "groups" of photographers traveled to the same spots. Some of the images were almost identical—see below for three images submitted to Click! along with one of my own from a similar vantage point.

My sociological point is (like crowd theory) that there is something "organized" (structured), predictable, understandable about what people see and how they interpret what it is they are looking at. It is a sort of common visual language which of course varies from culture to culture, education, class, etc...

As an aside, my own orientation is toward people so when evaluating Click! submissions, I gave the highest evaluations to "people images" as well as good evaluations for those not of people but with exceptional (in my estimation) artistic or documentary quality.

Condos and Trolley Cars and Sugar Refinaries, Oh My!

As promised, this week we're writing about the subject matter of the submitted images. If you evaluated all 389 or even a large part of the pool, you know exactly where we are headed with this and may be saying "Ugh, no more!" The image below is from our upcoming advanced search where we've tagged images according to location and basic subject matter. In the search, you'll be able to take any of these tags and run a comparison to see which images from Coney Island or Red Hook happened to fare the best among the evaluators. It should be pretty cool, but for our purposes today, this will give you an overview of what was submitted.

Keep in mind this was a blind open call where photographers couldn't see what others had submitted. I find it fascinating that some subjects seemed to ring true among many photographers. For instance, 43 images of Red Hook alone with 5 shots of the trolley cars behind the Fairway, 4 shots of the Ikea under construction (opening this week), 11 shots of the Revere Sugar Refinery (torn down to make way for Ikea's overflow parking). I realize that many evaluators found it tedious to see these same images pop up over and over again, but you have to wonder—what is it among these particular images or ideas that make them touchstones for the "changing face of Brooklyn?" It's definitely something to think about when you can explore search results in detail and it will be even more interesting to see if the evaluators felt the same was true when selecting images.

Still interested? Check out this analysis from Page 291, this comment from Trish caught my eye and, as always, we'd love to see your own thoughts and impressions in the comments area.

Now I'd like to introduce our first guest blogger for Click!, Jerry Krase. Jerry is a visual sociologist and professor over at Brooklyn College and who better to ask what he saw in the 389 submissions? For those of you who don't know, Jerry runs a bus tour through Brooklyn every year for incoming faculty at Brooklyn College. Rumor has it, this is an awesome tour looking at the communities and cultures throughout Brooklyn which enables new faculty to get to know the incoming student body and surrounding area. I've known people who couldn't stop talking about the tour after taking it and with that, check out Jerry's post up next.

Gaming Click!??!?

Yikes! This week I wanted to take a moment and look at some rather amusing things (or scary things, depending on your perspective) that happened during the evaluation process. We stayed mum on this during the eval period to avoid bringing more attention to it, but it's worth looking at now while revisiting chapter three of The Wisdom of Crowds. In chapter three, Surowiecki discusses information cascades, the effect of one person making a decision and others following suit. He writes, "The fundamental problem with an information cascade is that after a certain point it becomes rational for people to stop paying attention to their own knowledge—their private information—and to start looking instead at the actions of others and imitate them." A bit later in the chapter he concludes with, "Collective decisions are most likely to be good ones when they're made by people with diverse opinions reaching independent conclusions, relying primarily on their private information." Avoiding a cascade was the primary reason for suppressing comments and direct links in the evaluation tool, but, of course, we couldn't think of every situation that might present itself. It's important to keep in mind that there were very few instances of this, but the fact I didn't see this coming makes me wonder about my optimistic state of mind...

The evaluation was being conducted on the web and we were specifically asking for people to help us spread the word. Some of the posts, like this one, were awesome - here's an artist who's spreading the word using our widget which minimizes influence by randomizing images. But, then we noticed posts like this and this popping up (even more here) and, well, I felt a little daft for not instituting a specific rule about it. But, hey, a big part of this project is on the 'net and who wants to post to their blog without a little self-promo? So, I get it. The reality is these posts are fascinating to read and provide a great opportunity to see the process from the point of view of some of the artists - yay! And it's really awesome to come across a post like this, where the artist says, "i can't/won't tell which image i selected, as i believe it's against the rules."—well, it still makes me smile.

All this self-promotion begs the question...did it do any good? We don't think so. Evaluation was not a quick and easy process and you can see how this type of implementation starts to help when issues like this crop up. We carefully looked at the data to see how much of a difference this kind of self-promotion may have made. Luckily for us, we had enough participation (data) to counteract any skewing and the results were fairly balanced (thank you, again). Of course, this could have easily changed had any one instances above (or similar) generated a lot of traffic with very determined users.

Have fun exploring some of these posts, it will be even more interesting to reference them when you see the results on June 27. Next week we've invited a special guest blogger to post about the subject matter in the submitted works...more soon.

Thank you!

Wowzer! If you were one of the 3344 visitors who cast 410,089 evaluations for Click!, you know what a commitment it really was. I can't say this enough: Thank You! This was a very dedicated group of people who gave a lot of time to the process. On average photographs were viewed 22 seconds prior to submitting an evaluation and that's an eternity in web-time. If you calculate the average time on a photo with the 389 images, we are talking a gargantuan effort and this is especially true when you consider that our evaluation tool was almost universally disliked.

We didn't set out to make life difficult, but we were concerned that the evaluation be as fair as we could possibly make it and that meant minimizing influence. There were some encouraging comments here and there about the tool, but for the most part, people seemed to dislike it and they were going to make sure I knew it. "Frustrating," "tedious," "tiring," and "time-consuming" were just some of the words that came into my inbox. There was my personal favorite: "can you fix the website so i can VOTE !!!??! ! it is nearly IMPOSSIBLE to navigate thru your website and see all the pictures quickly and easily. can you fix that ASAP! !?" ...and you know it's really bad when you get a call from you father at 10 p.m. and the conversation goes something like this:

Dad: Shell, you know you can't skip around in there...?

Me: Yes, Dad, I know...it's because of all these reasons (blah, blah, blah and frustration starts to mount)

Dad: I know, but there are close to 400 photographs in there!

So, let's recap. The evaluation tool was designed to be fair. Images were randomized, attribution was withheld, comments (all 3098 that were left during the evaluation period) were withheld, linking was prevented and our algorithm ensured that all images would be seen an equal amount of times (approx 1054 times per image). Sure, we could have provided clickable thumbnails and ways to skip around, but that would have easily skewed results toward images that looked good in thumbnail format and there would have been no way to assure that each image would get equal time. In addition, an interface like that would have made it really easy for people to go vote for a particular work and skew the results — something we were specifically trying to avoid.

While most feedback indicated frustration, I will share one very interesting comment that I think really indicates some success with our methodology: "I have fresh empathy for curators who have to decide what’s in or out. It must be a grueling process". When we first started working on Click! almost every curator who had been a part of an open call or a juried process had much the same story to tell, that it is often a tiring and tedious process. The feedback from the crowd was a decent indicator that the process really was reflecting some of the realities of a juried process.

Thank you for hanging in there. This really was an amazing effort and your dedication and support will shape this exhibition. In the coming weeks we will be discussing more issues and release more data as we go along, so stay tuned and let me know your thoughts in the comments area.

Minimizing Influence

We are launching the evaluation interface for Click! today, so I wanted to take this opportunity to write about some of the choices behind the design. In The Wisdom of Crowds, James Surowiecki asserts that for a crowd to be wise, maintaining diversity and independence are two key factors. Both issues are talked about at length in chapters two and three of the book, but here is a small sampling to give you an idea:

"Diversity helps because it actually adds perspectives that would otherwise be absent and because it takes away, or at least weakens, some of the destructive characteristics of group decision making."

"Independence is important to intelligent decision making for two reasons. First, it keeps the mistakes that people make from becoming correlated. [...] Second, independent individuals are more likely to have new information rather than the same old data everyone is already familiar with. The smartest groups, then, are made up of people with diverse perspectives who are able to stay independent of each other. "

As we started planning for this phase of the exhibition, I started to recognize that many of the features we see on successful websites today are designed to foster community, but they also create a great deal of influence. The view counts, comments, favorites, most e-mailed, and leader boards of sites we all love (Flickr, Digg, StumbleUpon, NetFlix, The New York Times, etc.) are built on the influence of others, so when thinking about the online components of this exhibition, we wanted to minimize influence as much as possible and re-think features that are now commonplace. Of course, there have been plenty of times I've wondered if anyone would take part if we didn't include some of these features that everyone has come to expect, but we'll have to wait and see on that one.

You may have noticed that during the Open Call, we didn't have any preview of the images being submitted. We wanted to hold true to the book as much as possible, so photographers were asked to make a decision for themselves without basing it on what they could see others doing. We selected a theme, "The Changing Faces of Brooklyn", that could have a wide variety of interpretations, but would also keep the amount of submissions to a manageable size so as not to overwhelm the evaluation phase. Both aspects, variety of interpretation and the blind call, were designed to foster diversity and independence in the submitted works.

During the evaluation, images will be presented both without attribution and at random. We've suppressed the permalink in the address bar, to discourage people from sending a link and saying "hey, go vote for this." To keep interest, we've added a preview of the next two images in line, but neither are clickable to prevent skipping around in the interface. You can always see your own stats, but we don't reveal how others have rated a work. You can leave comments, but they won't be revealed until after the exhibition opens.

You can even see these ideas in our widget. Designed to help spread the word about the show, the widget had a very simple informational design during the Open Call period. Beginning with the online evaluation, the widget will pick up a set of images to help pique people's interest in the show, but because we don't want to cause unwanted influence, the widget picks up a different set images displays them at random every time the page is refreshed.

So, now it's up to you! If you don't have an account already, sign up for one and start evaluating! Your participation will shape the exhibition, opening June 27, 2008.

Click! Get the word out...

One of the biggest challenges we face with an exhibition like Click! is getting the word out. Click! depends on two types of participation, initial submissions from the artist community and participation from the online community who will evaluate those submissions. Getting to word out to the artist community is critical and that means we've been doing a lot of leg work.

Physical card distribution is still a major method of communication throughout the Brooklyn communities. Many of the local mom-and-pop establishments like cafes and bookstores have a place for card drops and announcements. As you can see, we printed a lot of cards, so we've been working to get them into the neighborhoods. Happily and with the help of many staff who have taken a bunch to their local hangouts, we are now down to two small stacks!

Since this is a photography show, it seemed like a good idea put announcements in all the Brooklyn-based groups on Flickr. Instead of just posting, we wrote the administrators of each group to ask permission to do so. The admins were great about getting back to us and our postings are now in place. For those of you who have never seen the Brooklyn groups on Flickr, there's a great online community there, so spend some time checking it out.

In addition to Flickr, there's a strong blogging community in Brooklyn and we've been sending information their way. To get an idea of how large and active this community is, take a look at the blogroll at The Gowanus Lounge. Also of interest, the Brooklyn blogging community hosts a monthly blogade.

And that's not all. There are many artist collectives in Brooklyn that we've been contacting and, in turn, they've been sending notification e-mails to their lists of artists. In addition, there are photo documentation projects for certain areas and we've been contacting the people running them. Take a look at What's The Hook? and the Coney Island Documentation Project for great examples.

So, this post is a bit of thanks to all those who have helped us get the word out about our open call. If you've sent out e-mails, allowed us to post to your Flickr group, accepted cards, installed our widget, posted to your blog - thank you! We couldn't do this without your help and we are happy to be neighbors!