|
| |
360-Degree Video and Journalism
From: Columbia University
| By:
John V. Pavlik |
EDITOR'S INTRODUCTION |
Columbia journalism and new-media expert John V. Pavlik outlines the nature of 360-degree video and its implications for the transformation of the news.  Although it is a technology more than two centuries in the making, 360-degree imaging has been slow to make its way into the storytelling of journalism. The reasons are many, including both the skeptical traditions of the newsroom toward anything "new" and the fact that 360-degree images shatter the long-held paradigm of photojournalism: framing an image is the best way to tell a story. But the problem with framing an image is that while it focuses attention, it also removes the object of attention from its surrounding context. 360-degree imaging allows both the continuation of framing and the inclusion of context. |
 n the mid-1990s, a number of researchers began developing alternative omnidirectional, 360-degree imaging systems. Among the first were Lucent scientist Vic Nalwa and Columbia University computer-science professor Shree Nayar. Although Nalwa and Nayar were among the first to develop these technologies in the digital environment, the historical origins go much further back. |
We'll examine these historical origins in just a moment. The year 1996 saw the first commercial products in the omnidirectional market. Omniview introduced a product called PhotoBubbles, which were described as "spherical photographs." The company claimed that "PhotoBubbles capture the entire contents of any location in 180- or 360-degree immersive representations that can be reproduced for viewing on a computer or TV display." PhotoBubbles were used by a number of news organizations, including the New York Times on the Web and CNN interactive. A competing brand has described its product as "surround video." |
In many ways, omnidirectional video represents a changing imaging paradigm, one fundamentally different from that of the Lumiere brothers a century before. PhotoBubbles are now called IPIX, and are now described as an "interactive photography technique that allows the user to be immersed inside a 360-degree digital image representing any environment that can be photographed. The user, via a mouse or keyboard input, is able to navigate in any desired direction in the interactive photograph, magnifying or exploring any part of the image." |
To date, some of the problems leading to relatively limited use of omnidirectional imaging by news organizations have been the mandatory paradigmatic shift in news photography and videography; limitations in bandwidth, making pictures slow to download; and complications in installing the plug-ins that are used to view the pictures. In addition, the early products did not lend themselves to storytelling. Rather, the news organizations provided interesting 360-degree views of news events, frequently for features--but rarely did the images tell a story. Instead they were nice-to-look-at complements to an accompanying text report. Moreover, taking a good omnidirectional image is even more complex than taking a good still image, and a paradigm for omnidirectional storytelling is only beginning to emerge. |
An alternative that may advance the storytelling of omnidirectional imaging is the incorporation of object-oriented video, or "hot spots," as well as the ability to add audio tracks to these omnidirectional still images. In this case, the reporter would report accompanying facts and context for the image in an audio narration and ambient sound while using a Java applet to automatically pan, tilt or zoom throughout the 360-degree view. In this way, one might not only tell a story but create a dynamic, moving picture from a single 360-degree still image, minimizing the bandwidth and processing power for a high-resolution video report. We'll return to this later and take a look at experimental approaches to this technique. |
It wasn't long until the panorama made its way to the New World. Steamboat skipper Robert Fulton introduced the panorama to Paris in 1799, and shortly thereafter to the United States. John Vanderlyn's panoramic 1819 depiction of the Versailles gardens is on exhibit at the Metropolitan Museum of Art in New York. The digital panorama debuted almost two centuries later on the "Star Trek" CD-ROM, which allowed circular tours of the chambers of the Starship Enterprise. |
Unintended effects
Typically, technologies are invented for explicit purposes and applications, such as to improve the efficiency, accuracy and speed of some aspect of communication or, in our case, journalism. As important as these intended effects are, the most significant effects are often the unintended or unexpected consequences. Consider the case of the panorama. When King George III visited one of London's first panoramas in 1794, he was intrigued by the imagery surrounding him. He saw a detailed painting of a naval fleet moored near the Isle of Wight. The reaction of his wife, Queen Charlotte, was somewhat more visceral: She became mildly seasick, which may explain why the work was retitled "The Nausorama" when it was displayed in Hamburg, Germany, a few years later. |
Current omnidirectional imaging
Now let's consider in more detail the invention of Shree Nayar. In 1996 Nayar, a professor at Columbia, developed what we've been calling the "omnicam," an imaging technology that can be used for "interactive visualization of a remote scene without any moving parts, unlike any standardized motorized pan or tilt system," reports the Remote Reality website. The omnidirectional camera works by pointing a specially designed lens at a specially crafted parabolic mirror. Because of its curved surface, a parabolic mirror is able to take an entire 360-degree field of view. The light reflected off of this mirrored surface is captured through the lens on the CCD (charge-coupled device). Since the omnicam produces images from which perspective images can be generated via software for any desired viewing direction and zoom, multiple remote users can "roam" around the visual space without any conflicts. There are no mechanical or moving parts, and the images can be streamed live or from stored files. |
One of the problems is that the traditional CCD chip is square. But the camera records a round view. Therefore, not all of the CCD surface is used to record image information. As a result, the camera does not maximize the efficiency of the CCD chip, nor does it obtain the highest possible resolution. An alternative chip, known as CMOS (complementary metal oxide semiconductor), has been developed, and Prof. Nayar and his colleagues are in negotiation with the developers of the CMOS chip to design a round chip that would work with the omnicam. An interesting historical footnote is that Nayar's invention is based in part on astronomical telescope designs using mirrors to view the entire night sky. |
CMOS
The CMOS and CCD image sensors are both constructed from silicon, which gives them similar properties of sensitivity over the visible light spectrum. Thus, both devices convert incident light (photons) into electronic charge (electrons) by the same photoconversion process. Although CMOS chips sense light in the same way that a CCD does, CMOS devices use sensor amplifiers, which are made from CMOS transistors. CMOS detectors can be better integrated than a CCD, and thus can be manufactured more cheaply, resulting in less expensive digital cameras. |
Omniview
Although the technology started out as the omnicam and was designed for scientific purposes, Prof. Nayar and Columbia University, as well as a number of investors, saw the commercial potential of this technology. Through the Columbia University Innovation Enterprise (CIE), a Silicon Alley start-up company was launched in 1997. The company was called Remote Reality Technologies, but in 2000 changed its name to Remote Reality. |
Omniview to ParaMax
 | |
| The Parashot strutless system uses a standard digital camera that attaches to the lens mount. | |
Remote Reality has produced its first commercial product, called the ParaShot, which produces still 360-degree images and has forthcoming a full-motion video commercial product called the ParaMax. Although there are a number of technical limitations to the ParaShot technology, there are also a number of important advantages. The ParaShot strutless system (left) includes an off-the-shelf Agfa e-photo 1680 megapixel camera, as well as a specially designed lens and parabolic mirror. The lens and mirror are attached via a rod placed in the center of the mirror, which is a natural but small blind spot, so there are no struts to remove from the photo. The lens cap can actually be easily removed so the Agfa camera can be used for standard photos as well. Also, there's no need for someone to buy the complete system if they already have the Agfa camera; they need only the lens cap attachment and software to start producing 360-degree images. The photographer uses a timer to make sure she or he is not in the picture when it is taken. Colleagues from the New York Times reported that one of their problems was that their photographers' feet kept getting in the picture. |
One of the most important advantages is that the ParaShot/ParaMax camera takes in the entire field of view through a single-lens camera, whereas most alternative products use multiple cameras, multiple mirrors and multiple CCDs requiring the user or photographer to stitch together the images. (Both SmoothMove's Imove and FullView use multiple cameras on a single mount to simultaneously observe a 360-degree view; these solve the problem of seeing an entire scene in a single moment, but they are expensive, both in terms of cost and bandwidth.) Consider the case of the IPIX system, which requires two pictures taken with a fish-eye lens to be stitched together, or QuickTime VR, which requires 12 images to be stitched together. Imagine photographing a breaking news story and having to take two or more different pictures of a single event--a gunshot, say, or a blow from a police baton on a slumping motorist named Rodney King. |
Another advantage the Remote Reality approach has over IPIX is that it does not require photographers or their organizations to pay a license fee every time the picture is published. Furthermore, the Remote Reality approach is easier to use and is more portable because of its much more compact size and light weight. The system also offers an advantage to the user or audience member. The ParaShot (and the resulting 360-degree images) uses server-based software and does not require the user to download and install a plug-in. |
360-degree imaging is also being used for real-estate sales. But the most intriguing applications are yet to come: the implications for journalism. Of course, some interesting journalism implications arise from the use of 360-degree views in real-estate sales. Imagine you're planning to move to a new community, and you want to not only view your potential home but also take a tour of a community school--without actually having to travel there during normal visiting hours. |
Times Square: implications for journalism
Consider the implications for covering a news story such as the economic rejuvenation of Times Square using a combination of traditional reporting techniques and new-media tools, including an omnidirectional imaging system. Such reporting was done by students in my 1997 summer online course, Exploring New Media, with funding from CNM partner Viacom and using an early full-motion 360 video camera developed by Prof. Nayar. |
Viacom itself is an interesting story in new media. In the late 1940s, cable television pioneer Ralph Baruch was walking one evening on a midtown Manhattan street when he saw a crowd gathered in front of storefront. After dinner, he noticed the crowd was still in front of the window. Curious, he took a closer look to see what caught their fascination. They were watching a new device called a television; the program was an engineering test pattern. It was then that Baruch knew that there was a future in the TV business. |
Baruch went on to found Viacom as a CBS spin-off (today this spin-off is now acquiring its parent). He pronounced it "vee-a-com," which is correct according to the Latin root. When Viacom was bought out and sold to Sumner Redstone, the transformation of his company was completed by changing the pronunciation to today's Viacom ("vie-a-com"). Viacom today is much more than a cable television programmer. Its wide interests range from Paramount Pictures to MTV to Blockbuster Video. CNM has a continuing relationship with Viacom to explore emerging new-media trends. |
The Kennedy assassination
Consider the implications of omnidirectional imaging in the context of the assassination of John F. Kennedy in 1963, almost exactly a century after the assassination of Abraham Lincoln. (The application of 360-degree imaging to the Kennedy assassination was first suggested to me in 1996 by Don Bates, the managing director of marketing and new media at Media Distribution Services (MDS) and an adjunct professor of communications at the New York Institute of Technology and at the New School University.) Kennedy was the 35th president of the United States. He was a decorated military veteran and a recognized statesman. On June 26, 1963, he visited the city hall in West Berlin at the height of the Cold War and made one of his most famous speeches. "All free men, wherever they may live, are citizens of Berlin. And therefore, as a free man, I take pride in the words Ich bin ein Berliner." Five months later he would go to Dallas, Texas. |
In this case, a very famous 5.6 seconds of film actually documented the assassination of President Kennedy. |
What if Dallas resident Abraham Zapruder had a 360-degree video camera, circa 1999? Rather than holding a Bell and Howell 8 mm film camera with a narrow field of view that could record only Kennedy's motorcade and the fusillade of bullets that killed Kennedy, what if Zapruder had held an omnidirectional full-motion video camera to record the assassination? We might not only have been able to view the shots that killed Kennedy but also to record the entire scene. We would have been able to pan around the scene and view the School Book Depository building where Lee Harvey Oswald, the alleged assassin, fired the fatal shots from a sixth-floor window, according to the Warren Report. The Sixth Floor Museum now located in the School Book Depository building commemorates the assassination. |
The motorcade
We would have been able to view the entire route of the Kennedy motorcade. |
Consider this ParaShot of Dealey Plaza. This omnidirectional image represents the view that Zapruder would have had back in 1963. |
 | |
| ParaShot of Dealey Plaza. | |
|
I have extracted four views of Dealey Plaza (below). Included are the views of a route of the motorcade, both where it came from and where it was heading. We can also see the School Book Depository building where Oswald crouched on the sixth floor, and panning about the scene we can take in the so-called "grassy knoll." We could have zoomed in and seen whether Oswald actually crouched in the sixth-floor window or we could have examined the grassy knoll, where conspiracy theorists have proposed a second gunman may have fired the shot that killed Kennedy. In other words, we could have examined the entire scene for details that we may have missed during the trauma of the moment. We might not only have transformed news about the event but perhaps influenced the entire course of history. We might not have had 35 years of lingering doubt, 35 years of conspiracy theories, a controversial Warren Report and more. Perhaps Oliver Stone might not have had to make his controversial film JFK. |
 | |
| Four extracted views of Dealey Plaza. | |
|
Consider how you could embed digital objects, also called hot spots (marked with an X in the image), for each of the objects of interest in the video, such as the spot where Kennedy was shot, the grassy knoll, and the School Book Depository building's sixth-floor window. By using these tools, one can tell an entire story interactively. We can use these hot spots to embed high-resolution close-ups of important parts of the assassination site. We could provide an interactive tour, or news report, about what happened and what the nature of the controversy is. We could also provide links to related websites for additional context. |
Consider the social implications involved in journalistic uses of omnidirectional imaging, especially privacy concerns. |
Finally, consider the use of omnidirectional imaging in coverage of two breaking news stories. In the spring of 1997, an interdisciplinary team of students in The News Laboratory, a class I taught at Columbia University, created the first omnidirectional documentary. They used an early omnidirectional video camera to shoot a three-and-a-half-minute 360 report about the protest and arrest of members of the Irish Lesbian and Gay Organization (ILGO) while they were trying to march in New York's annual St. Patrick's Day Parade. The organizers of the parade had banned the ILGO group and the student team used the omnidirectional camera to capture the dynamic scene. The 360-degree video report permitted the viewer to pan, tilt or zoom anywhere in the scene, observing the police, protesters and parade all converging in New York's Times Square. |
Two years later, another group of students in the author's class teamed with professional journalists from apbnews.com, a news site devoted to coverage of crime and criminal justice. Their collaborative project used a newer version of 360 photography to report the story of Amadou Diallo, a resident of the Bronx, New York, and a recent immigrant from West Africa who had been shot and killed late one night in February 1999 in a fusillade of 41 bullets fired by four members of the New York Police Department's street-crime unit. At their trial, the officers maintained that they had apparently thought Diallo was a suspect in a rape case and opened fire on him when they thought the wallet he was holding was actually a gun. |
Covered widely by the media, the apbnews-Columbia collaboration provided a unique perspective on the case by using 360 photography to provide a navigable look inside the vestibule where Diallo was killed. Visitors to apbnews.com could pan, tilt or zoom anywhere in the vestibule and examine the scene of the shooting, the bullet holes and notes written by police inspectors and other visitors to the vestibule. Another 360-degree photo provided a panoramic view of the street in front of Diallo's building. This view was especially useful in providing much-needed context for the shooting. |
 | |
| Panoramic view of the street where Amadou Diallo lived. | |
|
Many people outside New York no doubt envision either of two images of the Bronx: Yankee Stadium or Tom Wolfe's book The Bonfire of the Vanities. A navigable view of the street where Diallo lived could help visitors better understand the context of the police firing upon an unarmed man: Was his neighborhood one filled with crack houses and criminals, or was it something else, perhaps a nice, middle-class community? 360-degree imagery, whether still or motion video, can be effectively used to both engage an online audience and provide much-needed context in news stories. |
Are these 360 video reports a better form of journalism? Do they help to place events in a more complete context? These are the questions journalists and new consumers should be asking about all new forms of storytelling beginning to emerge in online journalism. |
|
| |
