Autonomy

One thing they didn’t do much of was bathe. The wastewater tank in their rented RV filled up, and by the time they got around to driving it to the nearest town to empty it, the vibrations from the washboard dirt road into town splashed sewage all over the RV’s interior. Cleaning the mess was so traumatizing that the team outlawed use of the RV’s bathroom. While there were bathrooms available in the mechanics shop, no other showers were available, so the guys went without washing for about six weeks. Then, in mid-February, one of their computer sponsors, Intel, invited the Nevada members of Red Team to San Francisco, where the computer chip manufacturer wanted to show off Sandstorm at the Intel Developer Forum.

By that time, Sandstorm had managed a speed of 49 mph and an autonomous run of a hundred miles. The guys were excited about the progress they’d made. But the robot still had its mechanical idiosyncrasies. It was apt to see obstacles that weren’t there, or miss obstacles that were, or even misinterpret pre-programmed commands. What if something like that happened while Sandstorm was onstage at the conference?

The following morning, an audience of hundreds watched the autonomous vehicle creep out onto the stage, apparently thanks to the benefit of high-tech sensors, engineering and computers powered by “Intel inside.” The crowd cheered in response. The applause felt good to the Red Team members present. Here they were at a Silicon Valley event being treated like celebrities. The recognition validated their sacrifices and the worth of the project. It also made the team thankful that no one realized that during the onstage demonstration, a Red Team member had hidden in the space under Sandstorm’s steering wheel, prepared on a moment’s notice to slam his hand on the brake pedal if the massive robot threatened to roll off the stage into the crowd.

On Friday, March 5, 2004—eight days before the race and just three days to go before the qualifying events—Chris Urmson rose early in the morning, put on his usual uniform of a mud-spattered baseball cap, fleece sweater and worn jeans, laced up his running shoes and decided that today would be the day to stage Sandstorm’s culminating test challenge.

Urmson, Peterson, Spiker and the rest of the Nevada squad tested Sandstorm in the worst conditions they could imagine—frequently, along sections of the trail the old Pony Express had followed more than a hundred years earlier. “Red is really gung-ho about testing hard,” explains Peterson. DARPA had said its route would be about 150 miles. The longest run Sandstorm had made was a hundred miles. But with the race a little more than a week away, everyone on the team was hoping for a longer run to boost their confidence.

The goal was just like the race: 150 miles in ten hours. The route amounted to a flat oval, about two miles around. While they prepared Sandstorm, Urmson and Peterson tinkered with a new part of the software: a component of the speed-setting module designed to slow down the robot when it approached a curve. The new code was designed to allow Sandstorm to drive more quickly on straightaways.

The code worked wonderfully. During a few warm-up laps, Sandstorm managed to get up to 49 mph along the straightaways and then the new algorithm slowed it down as the robot headed into the curves. In fact, as Urmson and Peterson watched the robot, they wondered whether it slowed Sandstorm too much. An adjustment to the algorithm during a refueling break seemed to improve things. On the first lap they watched as Sandstorm cruised into a curve, slowed a little bit and then accelerated through the curve’s exit. At the end of the second lap, Sandstorm was heading fast into what Urmson would later describe in his field test report as a “soft S-curve” to the left. The right-side tires drifted off the road into deep sand, and when Sandstorm tried to correct things, to get back on the track’s packed-down dirt, it steered too hard to the left. The right-side tires bit into the soft sand. The left-side tires came up off the road. Behind, in the chase car, Urmson watched, horrified, as Sandstorm tipped up and over, and came to rest upside down—right on top of the e-box and the gimbal housing all the vehicle’s most sensitive equipment.

The robot had been designed to insulate the box’s components from being damaged in all sorts of accidents. Front-end collisions, rear-end collisions—pretty much any collision that happened on the ground plane, Sandstorm would be able to withstand just fine. But the robot had one fatal weakness: a rollover. Because Humvees sat comparatively low and flat, their geometry made rollover accidents almost impossible.

Unless you were testing a robot Humvee in the Mojave Desert, apparently.

A History Channel crew had come out to film the test run. They rushed out onto the track with their cameras and shoved one into Urmson’s face, asking him to list the damage. Urmson looked at the wrecked robot the team had spent the better part of a year engineering: at the crushed gimbal, the compacted GPS antennae, the flattened e-box and the connecting rods bent out of shape. And he let fly with the expletives that made him one of the few people to ever have to have been bleeped by the History Channel. “Shock and disbelief,” Urmson says when asked to describe his reaction, more than ten years later. “But mostly disbelief.”

Disbelief, because they had felt like they’d been making such great progress. Disbelief because they were just days from the qualifiers. Disbelief because this had happened on the second lap of a two-mile track that Sandstorm was supposed to drive for seventy-three more laps.

Most of the crew figured Red Team was over. That they’d never be able to repair the robot in time. Somebody called Pittsburgh to inform everyone else about the accident. Red’s assistant Michele Gittleman took the call. She recalls sobbing when she processed the news.

Maybe a crew led by someone other than Red Whittaker would have given up. But Whittaker didn’t even consider it.

At the Nevada Automotive Test Center, Urmson, Peterson, Spiker and the rest of the team attached the four-by-four chase vehicle to Sandstorm with the help of some nylon webbing and managed to flip over the robot. Spiker, the most mechanically minded among them, went over the engine to look for problems. The engine was flooded with diesel fuel but aside from that, everything looked okay.

The other guys assessed the electronics equipment. The GPS units were toast. The gimbal suffered the worst impact and would need to be completely rebuilt. The main LIDAR unit was irreparable. Luckily, an extra gimbal and LIDAR sat in storage back in Pittsburgh. They towed Sandstorm to the mechanics shop and for three nearly sleepless days and nights they worked to fix everything they could. And they nearly did it.

The race was March 13, 2004. Heading into the qualifiers the week before, at the California speedway in Fontana, the GPS system worked, which meant the robot could locate itself in the world. The sensors were active, which enabled Sandstorm to perceive obstacles. The computers could calculate the trajectories required to follow the path set out by the Red Team mapping crew. The only problem? “We had no time to calibrate,” recalls Whittaker. Which meant Sandstorm viewed the world through a distorted lens.

Think of each sensor as its own individual eyeball. You are able to see one version of reality because your brain is able to amalgamate the view from your two eyes into a single picture of the world. Sandstorm amalgamated the information from four different LIDAR units plus the stereo-camera system. Ensuring that the robot’s sense of the world resembled the actual world required calibrating the individual sensors—a time-intensive process of trial and error. “Think of calibration as alignment,” Whittaker explains. “Even in the car shops they align your headlights, right? And when there are multiple sensors that will fuse data into a common model, it’s important that they’re all aligned. If you just bolt it back together you’re creating a kind of Frankenstein, and maybe it’s a little cross-eyed.”

And so the cross-eyed Frankenstein’s monster limped into race week, belching diesel exhaust, dented and scratched, but otherwise intact. Red Team would compete against twenty other entries from across the United States. In the qualifiers each of the twenty-one robots would have to navigate a mile-long obstacle course to progress to the main event.

Soon after his arrival at the California Speedway, Urmson wandered around to see what he could learn about his competition. He saw Doom Buggy, created by the only high school admitted to the competition, Palos Verdes, near Los Angeles. An undergraduate from UC Berkeley, Anthony Levandowski, led the team behind the only two-wheeled entry, a robot motorcycle that was able to balance itself with the help of a gyroscope. UCLA’s entry, the Golem Group, was led by a guy named Richard Mason, who had seeded his project with $28,000 he’d won on Jeopardy!

On the other end of the spectrum were the professional teams, which were affiliated with various engineering-focused corporations. An inventor named Dave Hall had created an autonomous Toyota Tundra pickup truck that was notable for driving smoothly with a stereoscopic camera setup—it didn’t use any LIDAR at all. From Wisconsin, the makers of Oshkosh Trucks entered a six-wheel-drive, 32,000-pound fluorescent yellow behemoth with the imposing name of TerraMax. Louisiana’s Team CajunBot also used a six-wheeled vehicle. A fraction the size of the Oshkosh entry, it was based on an all-terrain vehicle more commonly used by the state’s hunters to navigate the bayou.

Urmson, Peterson, Whittaker—all of them wandered the event, talking to people just as technically minded as they were. It quickly became apparent that Red Team was among the biggest of the teams. Popular Mechanics gave them seven-to-one odds to win, highest of all their competitors’. The front-runner status positioned Red Team as the entry everyone else wanted to beat. Urmson had posted on the Red Team website a photo of Sandstorm after the rollover. Now, as he wandered the raceway, talking to the leaders of other teams, Urmson spied the photo on numerous computer monitors. Some other teams had made it their wallpaper—as motivation.

Adding to the excitement was the fact that DARPA’s public relations team had arranged for reporters and television producers from across the nation to visit the raceway. Urmson and his teammates had toiled for months in obscurity. The accolades they received at the Intel event had been nice, but the more common reaction to their work was incredulity. “A car that drives itself?” people would scoff. To many, it sounded ridiculous. The presence of the reporters going around interviewing anyone available reminded the competitors that their work was important. Important enough for the U.S. government to put up a million-dollar prize. Important enough, possibly, that it might save the lives of U.S. soldiers fighting in distant theaters of war.

On the morning of March 13, 2004, the start of the race was one of the most exciting moments Chris Urmson had ever experienced. The robots were lined up in their starting chutes. Media and military helicopters hovered in the sky. Grandstands supported hundreds of spectators, each of them getting whipped by the desert sand, and over it all, Tony Tether’s amplified voice marked the momentous event.

“We’re thirty seconds from history,” shouted the DARPA director into the microphone. “All right, ladies and gentlemen, boys and girls, the bot has been ordered to run, the green flag waves, the strobe-light is on, the command from the tower is to move!”

Because Sandstorm had performed best in the qualifiers, it had the honor of starting first. The big Humvee rolled slowly out of its chute. “Ladies and genetlemen, Sandstorm!” Tether cried. “[An] autonomous vehicle traversing the desert with the goal of keeping our young military personnel out of harm’s way.”

The first complication in the race course was a leftward turn. Its inside edge was marked with some scrubby vegetation, and its outside edge, with concrete jersey barriers protecting spectators from the robots. Sandstorm followed the road perfectly throughout the curve and accelerated once it headed out on its straightaway.

While it was still in view, Sandstorm ran over a hay bale. Urmson winced. But the big off-road vehicle just kept on going. Soon, the Red Team couldn’t see their robot at all. No one had thought to provide the teams with a video feed of their vehicle’s progress. All they could do was settle in and wait to hear reports issued back to the start from helicopter-borne observers and other officials set along the course.

Soon, the other entries headed out: A team called SciAutonics II. Then CajunBot rolled its six wheels from the starting chute and drove straight into a jersey barrier. Team ENSCO’s robot, based on a Honda ATV, wandered from the road just past the turn, flipped over on its side and was out of the race just two hundred yards into the event.

Palos Verdes High School’s autonomous SUV also ran into a jersey barrier. And then came the most curious of the entries: Anthony Levandowski’s autonomous motorcycle. Levandowski pulled it up to the starting line, activated the gas-powered motor, stepped away—and watched, brokenhearted, as the motorcycle immediately tipped over. As Levandowski would discover later, he’d forgotten to activate the gyroscope that kept the motorbike balanced. His race was over.

Minutes later, Red Team heard from a race organizer that something was wrong with Sandstorm. The hay bale the robot had run over just after the start turned out to reflect an ongoing problem. Perhaps because its sensors hadn’t been calibrated properly, perhaps because the main LIDAR’s replacement unit scanned at a much slower rate than the original, Sandstorm consistently appeared to think that it was a foot or two to the left or right of where it actually was. The Humvee drove over a fencepost, then another and a third. Some miles later, the vehicle swung itself into a curve, a particularly tricky one given the inside edge was separated from a steep drop-off by only a knee-high berm. As Urmson and Peterson intended, Sandstorm slowed down as the road turned. But the robot was a foot or two to the left from where it should have been. As a result, the left-most tires climbed up the berm, then dropped down the steep inside ledge. Sandstorm was now stuck on its belly—what Urmson called “high-centering.”

Things quickly grew worse. Sensing Sandstorm wasn’t moving, the speed control system kicked in, directing more power to the engine. One of the tires hanging over the other side of the berm was situated just high enough off the ground that it could still touch the Mojave sand. The friction heated up the rubber until it smoked and eventually burst into flames. The robot’s progress was over 7.3 miles from its start.

The media used Sandstorm’s flame-out as a metaphor for the entire event. The number-two entrant, SciAutonics II, also got stuck on a low hill of earth. Dave Hall’s Toyota Tundra became confused by a small rock. The UCLA entry, Golem Group, stalled out when a safety device prevented its engine from accelerating enough to get up an incline. And TerraMax, the 32,000-pound monster truck known for its brute force approach, ended up halted when a pair of tumbleweeds it incorrectly considered immovable obstacles blew ahead and behind it. And those were the best-performing vehicles.

The result put DARPA director Tony Tether in a tough spot. At the other end of the race course, in Primm, Nevada, was a tent full of reporters who had traveled across the country to file stories on the race winner. Tether figured he was going to get killed by the press—an expectation that proved right. “DARPA’s Debacle in the Desert,” went one headline. The gist of the stories portrayed DARPA as an out-of-touch government bureaucracy that had wasted money staging a fool’s errand. So to distract them, Tether took the stage and announced a second race, to be held in a year or so, with a doubling of the 2004 race’s purse, to $2 million.

Chapter Two

A SECOND CHANCE

The only way to prove you’re a good sport is to lose.

—ERNIE BANKS

Red Whittaker started planning for the second race even before Sandstorm returned to Pittsburgh from the first. Through his repeated entreaties for sponsorship, Whittaker had developed a relationship with AM General, the company that manufactured the Humvee. Now Whittaker thought he could convince the executives to donate an additional vehicle for Red Team to use in the next challenge—if the executive team would only witness a demonstration of Sandstorm’s capabilities.

Several days after the first challenge, Whittaker, Spiker and Peterson arrived with Sandstorm at the AM General campus in South Bend, Indiana, to conduct that demonstration. Spiker and Peterson stayed outside and set up the robot on an obstacle course the Humvee manufacturer maintained to educate new owners on the capabilities of their vehicles.

One element of the obstacle course was a concrete tabletop structure, maybe eighteen inches off the ground. Peterson and Spiker wondered whether Sandstorm could drive itself up and onto the obstacle. Moments later, rather than creeping toward the tabletop, as Spiker and Peterson had intended, Sandstorm took off toward it at high speed.

A kill switch was designed to deactivate Sandstorm if it ever did anything unpredictable. Trouble was, the kill switch had about a two-second delay. Spiker pressed the switch, but Sandstorm hit the tabletop before the command took effect. The front wheels bounced the front end into the air. The rear wheels hit the tabletop and bounced up the Humvee’s back end. For a moment the entire vehicle was airborne. Then the front end nose-dived with a violent slam against the concrete.

That’s when the kill switch disabled the vehicle.

Spiker and Peterson rushed to assess the damage. Whittaker was in a nearby building conducting his presentation for AM General executives on Red Team, and the wonderful capabilities of the robot they’d developed. Outside, Spiker and Peterson discovered the impact of Sandstorm on the tabletop had crushed an engine-compartment coolant tank. Once that was repaired, they set up Sandstorm on a section of clear road and activated the giant robot to test it. Immediately the front wheels turned to the right. That shouldn’t have happened. “Kill kill kill!” Spiker shouted to Peterson. With a snort of exhaust, Sandstorm accelerated right off the road and straight into the building where Whittaker was talking to the AM General executives. The impact of the Humvee against the wall shook the entire structure.

Later, Spiker figured out that the tabletop collision had detached a steering position sensor from its mooring—which, in turn, caused the second accident. But it turned out not to have mattered. Whittaker and the AM General executives rushed from the building to investigate the source of the impact. Spiker figured the sponsorship bid was toast. But as the execs surveyed the scene of the accident, Spiker realized his fears were groundless.

“Unflinching grace” is the way Whittaker characterizes the AM General execs’ reactions, portraying them as “great hosts who don’t fuss over a dropped fork or spilled water.” The executives saw themselves as manufacturing a vehicle designed to push the bounds of what an automobile could do—and so, in its own way, did the Red Team. Of course they would sponsor Whittaker’s team. “We’ll give you two Humvees,” one of the AM General execs proclaimed. “Just be careful.”

Some months later, in the summer of 2004, a computer scientist named Sebastian Thrun listened to a presentation about the first DARPA Grand Challenge in a seminar room at Stanford University. Thrun had recently moved from a faculty position at Carnegie Mellon’s Robotics Institute, where he’d been working on a project with Red Whittaker—a robot called Groundhog that was designed to map Pennsylvania’s abandoned coal mines. His new job was in Palo Alto, California, leading the Stanford Artificial Intelligence Laboratory, a once-respected research facility established by AI pioneer John McCarthy in 1963, which had been dormant since it had been rolled into the greater computer science faculty in 1980. To reincarnate the facility, Thrun brought nine Carnegie Mellon academics with him. Having left behind all his projects at his old school, Thrun was looking for a quick way to reestablish the AI lab’s reputation.

Thrun had attended the first Grand Challenge as a spectator, and was intrigued by the prospect of entering the second, as the rebooted Stanford AI lab’s first major feat. So Thrun asked one of his fellow CMU transplants, who had also attended the first challenge, to conduct a presentation to the rest of the group.

The presenter was Mike Montemerlo, a soft-spoken engineer who had a reputation as a software whiz known for his ability to program robots to conduct the simultaneous localization and mapping that had so bedeviled Sandstorm in the first race. Montemerlo’s father, Melvin Montemerlo, was a program executive at NASA and had worked closely with Whittaker on numerous projects. When Mike had been in high school, his dad had taken him on a pre-college trip to experience firsthand candidate campuses. One evening in Pittsburgh, the pair of them threw pebbles up at Whittaker’s window to convince the robotics legend to give the teenager a tour of the Field Robotics Center. That experience was the reason Montemerlo attended CMU. Years later, Whittaker would become Montemerlo’s PhD adviser; in the same period, Montemerlo also happened to be Chris Urmson’s officemate.

At Stanford, Montemerlo’s presentation amounted to a travelogue of his experiences at the California Speedway. Full of photos of the various robots, the seminar highlighted the problems and foibles that each team experienced. He spent a lot of time on the work that had almost been destroyed by Sandstorm’s rollover accident. The penultimate slide asked whether the Stanford AI lab should compete in the second DARPA Grand Challenge. The final slide featured the answer: “No,” in bold and all caps.

Thrun is a slim man who communicates in perfectly enunciated, precisely formed sentences colored with a German accent; he was born in the small Rhineland city of Solingen and raised in north Germany. “Why not?” he asked softly.

“It’s hard,” said Montemerlo, whose side-parted brown hair and wire-framed circular glasses made him resemble the Hollywood stereotype of a software engineer. “It’s all encompassing,” he followed up, perhaps thinking of the experience of Urmson and the rest of the CMU team. “People have to work all day and all night. They lose their social life. And—it can’t be done!”

Somewhere, somehow, Montemerlo must have known that telling Thrun that something couldn’t be done was the quickest way to entice him to try it. “I’m a rule breaker,” Thrun says, a character trait he shares with Whittaker. “A rebel—I like to do crazy things.”

Thrun was the third of three children. “I was the one the parents didn’t have the energy and time to pay attention to,” he told one reporter, years later. “I remember a beautiful childhood—but pretty much on my own.” Left to his own devices, he developed various obsessions with intellectual projects. At the age of twelve, in 1980, the obsession involved a Texas Instruments pocket calculator that could be programmed to solve various equations. Thrun delighted himself using it to create little video games. Next, he happened upon a Commodore 64 personal computer on display in a local department store. The computer was too expensive for his middle-class family, so Thrun returned to the store display to program on it, day after day, week after week. Each day he tried bigger and bigger programming challenges. He grew adept at efficient coding; because the staff turned off the computer each night, he had to execute each challenge he set himself in the two and a half hours that passed between the end of the school day and the store’s closing time.