Part IV: Semi-Autonomous Control Framework: Present Performance and Future Work

Please welcome back Sterling Anderson, a Ph.D. candidate at MIT, for the final post in his series on semi-autonomous driver assistance systems.

We’ve made it! Congratulations to all those who hung on through the first three posts in this series. Having done so, you are better prepared to understand and appreciate what I’m about to show you. For those tuning in for the first time (or you who decided to skip straight to the good stuff), welcome! The demonstration that follows should be sufficiently accessible that you’ll be able to appreciate, at least in part, what we’ve done here. If at any point you find yourself asking the question “wait a minute, don’t some cars already do this?” I would suggest you go back and read Parts 2 and 3 to understand the fundamental advances this framework provides when compared to the existing state of the art.


Vehicular accidents are costly. Not only do they end lives, injure travelers, and destroy assets, but they also inspire excessively large, heavy, and inefficient vehicles. Active safety systems can assist error-prone human drivers in avoiding accidents and thereby improve safety, efficiency, and cost. Active safety systems existing today are fundamentally limited in their inability to accurately quantify threat and intervene in more than one dimension to assist the human driver in avoiding it. As such, these systems must be implemented in an ad-hoc fashion, requiring significant fine-tuning to avoid conflicts in their sometimes-competing objectives.

What we have created is an integrated (read: ‘all-in-one’) planning and control framework that performs all of the functions of existing safety systems, in addition to predictively avoiding future hazards. This framework uses a fundamentally-new and incredibly-useful threat assessment method to predict the danger or ‘threat’ posed to the vehicle given its current state and the state of its surroundings. Based on this threat assessment, it then determines when, how, and to what degree it must intervene to ensure that the vehicle does not crash, lose control, or otherwise endanger its occupants. The controller is designed to allow the human driver as much control as possible in low threat scenarios and intervene only as necessary to keep the vehicle safe in high-threat scenarios. In the figures and videos that follow, I’d like to demonstrate a subset of the framework’s capabilities using figures and videos selected from the thousands of simulations and over 800 experimental trials that weíve used to vet it. Note that due to proprietary controls at Ford’s proving grounds, we were unable to record video of our Jaguar S-Type performing these maneuvers. Instead, we recorded telemetry data from each experiment and re-produced the results in high-fidelity simulation software (ADAMS/car).

Each of the videos below overlays the results from two simulations: the gray vehicle is controlled solely by a human driver model whereas the blue vehicle is also fitted with the semi-autonomous controller. In experimental trials, 8 different human drivers, each with different driving styles, were tested.


The experiments shown in the figure below illustrate the semi-autonomous controller’s ability to adjust its behavior to the preference and/or performance of the human driver. The upper plot shows the vehicle path as the driver drifted laterally in the lane (edges shown in gray). The lower subplot shows the proportion of available steering control assumed by the controller.

Note that by simply changing the threshold threat at which the controller intervenes, we can allow the human driver more or less control in low-threat scenarios (between X = 0 and 100 meters) without adversely affecting the controller’s ability to keep the vehicle safely within the lane in high-threat situations. Thus, an inexperienced or cautious driver might prefer more controller intervention all the time in order to smooth out mistakes, while a seasoned or more adventurous driver would prefer that the controller not intervene until this intervention was absolutely necessary. In the figure above, the red solid line represents an intervention function tuned to the more cautious driver while the magenta dash-dotted line shows the results of tuning the controller to more experienced driver. Notice that in both cases, the controller allowed the human to wander freely within the lane, while intervening as necessary to prevent unsafe lane departure. The black dashed line shows what happens when the controller is turned off.


The video below demonstrates the navigation framework’s performance in the presence of stationary hazards such as road edges, roadway obstacles (not shown), etc. In this simulation, the driver of both vehicles actively seeks to remain on the road surface — a difficult feat at 20 m/s (~44 mph).

Notice that including the semi-autonomous controller in the control loop not only keeps the vehicle stable, but also moderates the driver’s inputs in the process. Whereas the unassisted driver oversteers and loses control of the vehicle, the assisted driver notices that the vehicle is responding as desired and is thus more moderate in his steer commands. This allows him to maintain control of the vehicle. Moreover, allocating less than 50% of the available control authority to the controller (see green bar on the right) is sufficient to keep the vehicle on the navigable roadway and within 0.4 meters of the (invisible) line on the center of the roadway that the driver model is trying to track. The combined effect of both inputs (driver and controller) is a vehicle trajectory that more closely tracks the path the driver is trying to follow than the driver could accomplish on his own.

In scenarios where a drowsy, inattentive, or otherwise-impaired driver fails to steer around an impending threat, the semi-autonomous controller foresees the threat, gauges the control action necessary to avoid it, and if the driver does not respond appropriately, takes the necessary control to keep the vehicle safe. Once the threat has been reduced, it returns control to the driver. The video below demonstrates one such case.

In order to avoid moving hazards, the semi-autonomous framework predicts their future position and pre-emptively assists the driver in avoiding those regions of the environment. In both of the videos below, the human driver acts as though he doesnít see the vehicles up ahead (no steering input). In the first video, the controller recognizes that a passing opportunity is available and takes only as much control as necessary to execute that maneuver. The second video illustrates a slightly different case in which the yellow vehicle accelerates once the blue vehicle initiates a passing maneuver (weíve all known one). In this case, the controller behaves much like an alert driver would ñ seeking first to pass, then pulling back in behind the yellow vehicle as it accelerates.


I hope that the ideas discussed in this mini-series have provided a glimpse into the unique challenges and opportunities facing the emerging science of semi-autonomous control. While the issues and potential solutions weíve discussed in these four articles might seem a bit long-winded for a blog, they only scratch the surface of the technology, user studies, and legal infrastructure requirements that must be satisfied before these systems can be commercially implemented. Not the least of these considerations are driver acceptance issues. Almost everywhere I go to present this technology, one of the first questions I am asked is whether our system will come with an ‘OFF’ switch. Many people distrust the invisible face of automation and prefer to feel like they are in complete control. While we cannot completely concede the latter without sacrificing safety, we can certainly improve drivers’ perception and acceptance of autonomy by creating reliable, non-intrusive systems that modify driver inputs as little as possible while avoiding hazards. Significant work remains to be conducted in both human factors and usability studies before this research is road ready (my standard legal disclaimer), but I believe that at some time in the near future, it will be. Here’s to smaller, lighter, safer, and more efficient automobiles!


I’d like to thank Dr. James Allison for his invitation to contribute these articles. Writing them has been an exercise in making my research more understandable to non-technical readers. For those of you who would like more details (and believe me there are many), I would invite you to read any of the applicable papers/theses listed on my website. If you have further questions, or would like to continue the conversation offline, I would be more than happy to visit with you. Please feel free to send me an email and/or leave comments below.

Posted: February 12th, 2011 | Filed under: Design, Modeling, Sustainability, Transportation | No Comments »

Part III: Fundamental Challenges Facing Existing ADAS Systems and Description of The Alternative

Sterling Anderson, a Ph.D. candidate at MIT, continues his discussion of semi-autonomous driver assistance systems in this post. If you haven’t read his previous posts (Part 1, Part 2) you might want to start with them.

In my last article, I described existing driver assistance systems and introduced the notion of threat assessment. Near the end of the post, I described how the choice of threat assessment metrics is critical to the performance of any advanced driver assistance or semi-autonomous system. This article explains how existing systems assess threat and describes two significant problems caused by their approach. It then describes our alternative approach to threat assessment and semi-autonomous control. My next and final post will then demonstrate our system’s performance in both simulated and experimental results.

Significant Challenges Facing the Current State-of-the-Art

The first challenge facing existing systems is their inability to accurately capture the threat inherent to a particular scenario in a meaningful way. Because assimilating multiple sources of threat into a single, actionable metric is difficult, the majority of ADAS systems in existence today consider only one hazard at a time and limit their assistance to one dimension or another. For example, adaptive cruise controllers focus on the closest obstacle within the ego vehicle’s line of sight (longitudinal dimension) and apply the brakes as necessary to avoid rear-end collisions. Lane keeping controllers focus instead on the lateral dimension, neglecting longitudinal vehicle dynamics and collision hazards in order to maintain a desired position and/or heading within the lane. Like the child playing checkers for the first time, the tactical, near-term focus of these systems often fails to consider the effect of current evasive actions on future threat scenarios. For example, Figure 1 illustrates the predictions of an emergency braking system (solid green), along with those of a lateral collision avoidance controller (dashed blue). Without considering Obstacle B or the maneuver required to straighten out after it has passed Obstacle C, the obstacle avoidance system might consider a pass to the left of Obstacle C preferable to an emergency braking maneuver. Such a decision could be disastrous if either 1) Vehicle A cannot straighten out in time to avoid running off the road or 2) Vehicle A cannot avoid an oncoming Vehicle B once it has passed Vehicle C.

Figure 1. Illustration of tactical avoidance maneuvers considered by existing driver assistance systems

Figure 1. Illustration of tactical avoidance maneuvers considered by existing driver assistance systems

The second major problem facing existing driver assistance systems is closely related to the first: the myopic focus of existing systems on a single source of threat requires that a vehicle be equipped with multiple safety systems. For example, comprehensive assistance might require that the same vehicle be equipped with warning devices to alert the driver, anti-lock brakes to prevent skidding, yaw stability control to prevent loss of control, adaptive cruise control to prevent rear-ending the vehicle ahead, lane-keeping control to prevent wandering out of one’s lane, etc. While each of these systems may perform adequately by itself, their combined output when placed together on the same vehicle can lead to vehicle performance that is suboptimal at best and unpredictable or inconsistent at worst.

For instance, imagine a vehicle traveling quickly along a curve in road (illustrated in Figure 2). Wanting to avoid the large truck that is passing you on the inside of the turn, you drift toward the outer edge of your lane. As the lane departure warning begins to sound (adding to your already-heightened apprehension), the lane assist system takes control of the steering wheel, turning it hard left. As you swerve left, the car begins to roll right, whereupon the rollover alarm lights begin to flash and the roll stability controller engages, turning you back toward the outer edge of the road and again engaging the lane keeping controller. You can imagine how this scenario might continue. The point is that when distinct systems with disparate goals try to control the same vehicle, conflicts in their warnings and their steering, acceleration, and braking commands are bound to result. Figure 3 shows a (simplistic) schematic of how multiple systems arranged on the same vehicle can sometimes interact.

Figure 2. Illustration of a passing maneuver (see text for narrative/context)

Figure 2. Illustration of a passing maneuver (see text for narrative/context)

Figure 2: Illustration showing how using multiple heterogeneous driver assistance systems in the same vehicle can have detrimental consequences.

Figure 3: Illustration showing how using multiple heterogeneous driver assistance systems in the same vehicle can have detrimental consequences.

The Alternative: An Integrated, Semi-Autonomous Active Safety Framework

To summarize the above, existing active safety systems are both incomplete in their assessment of threat and ineffective at sharing control with the human driver. This is where we come in.

When I began working with Karl Iagnemma and Steve Peters at MIT’s Robotic Mobility Group a couple of years ago, we began exploring a promising new idea for semi-autonomous control. We based our system on the simple observation that most human drivers tend to operate within a field of safe travel as opposed to along a predetermined path. Thus, instead of starting with a simple avoidance path, we chose to define a corridor through the environment that avoids obstacles, road edges, and other hazards. Then rather than selectively replace the driver when s/he strayed from the automation-desired path, we chose to gradually blend the driver’s inputs with the controller’s – giving the driver free reign while s/he remained within the safe corridor and only intervening as the likelihood of his or her leaving that safe corridor or losing control, increased.

The framework we developed (illustrated in Figure 4) storyboards as follows:

As a human driver navigates the vehicle, forward-looking sensors detect road edges, identify and terrain features (slopes, holes, etc.), and localize obstacles. Based on this information, a safe corridor that avoids these hazards is defined. A mathematical model of the vehicle is then forward-simulated to determine the safest or most stable path through the corridor given the vehicle’s current state (position, velocity, roll angle, etc.) and the current state of the environment. Because this optimal trajectory can be considered the best maneuver that can possibly be performed given the current circumstance, it is then used to assess the (best case) threat posed to the vehicle. In other words, no matter how skilled the human driver is, s/he will be unable to perform any better than the “best case” avoidance maneuver. As this “best case” maneuver becomes more dangerous, so does any maneuver that the human might attempt. In low-threat scenarios, the human retains full control of the vehicle (K=0 for you sharp-eyed readers). As threat increases owing, for example, to the driver’s failing to make adequate preparations to avoid an obstacle, so does the level of control authority given to the autonomous controller (gain K in the Figure 4 below). In extreme scenarios, when the avoidance maneuver required to keep the vehicle safe becomes so severe that only an optimal maneuver can be expected to pull it off, gain K becomes one and the system effectively acts as an autonomous controller until threat is reduced to human-manageable levels.

Figure 3: Block diagram illustrating basic framework operation

Figure 4: Block diagram illustrating basic framework operation

Figure 5 illustrates what the predicted avoidance maneuver might look when the vehicle is at position 1 (low threat, full human control), as well as how that prediction might appear when the vehicle reaches position 2 (high threat, nearly-autonomous operation).

Figure 5. Illustration of how corridor boundaries affect the optimal trajectory prediction and how the threat assessment based on that prediction affects the control allocation (level of autonomy)

Figure 5. Illustration of how corridor boundaries affect the optimal trajectory prediction and how the threat assessment based on that prediction affects the control allocation (level of autonomy)

The video below illustrates the semi-autonomous controller in action. Corridor constraints are shown in black and green, with the vehicle’s trail in blue and the predicted trajectory in red. In this simulation, the human driver fails to see or respond to a hazard. As the system’s prediction begins to predict a more and more severe avoidance maneuver, it gradually asserts only enough control necessary to avoid the hazard before giving control back to the human driver.

To conclude, the system we’ve developed combines the environment’s many hazards into a single, safe corridor and moderates the driver’s steering and braking inputs as necessary to keep the vehicle within that corridor. In low threat scenarios, the driver maintains full control of the vehicle. As threat increases, the controller shares control with the human driver to ensure that the vehicle does not leave the safe corridor. By inherently considering multiple sources of threat in a single, unified framework, this approach provides a significant advantage over existing driver assistance systems.

In my next post, I’ll show how this framework performed in the over 800 experimental trials with 8 different human drivers on a Jaguar S-Type. Stay tuned.

Posted: February 2nd, 2011 | Filed under: Design, Optimization, Transportation | 2 Comments »

Part II: Driver Assistance Systems and the need for Threat Assessment

Please welcome back Sterling Anderson, a continuing guest blogger and Ph.D. student at MIT working in the Robotic Mobility Group. In today’s article, Sterling tells us about further developments regarding his work in semi-autonomous vehicle control. In his first post, Sterling discussed how this work relates to sustainability by helping reduce the vehicle mass required to keep its occupants safe. Here he describes an important aspect of these systems: threat assessment.

Hello again! With a few patents in the pipeline, and having given you a chance to think about what a semi-autonomous hazard avoidance system might look like, we’re ready to discuss the system we’ve developed. In this post, I briefly (re)introduce the vehicle safety problem and describe the current state of the art. I then highlight significant challenges inherent to the design of a comprehensive driver assistance system and describe the need for improved threat assessment techniques. In my next post (Part III of this series), I will point out two fundamental challenges facing existing ADAS systems and introduce our approach. Finally, Part IV will demonstrate the performance of our system and outline the challenges it faces going forward. Each part in this series will close with a thought question for the interested reader, which you are free to discuss either in the comments section below.


Vehicular safety is a problem that, I think, needs little motivation. Recent traffic safety reports from the National Highway Traffic and Safety Administration show that in 2008 alone, over 37,000 people were killed and another 2.3 million injured in motor vehicle accidents in the United States. While the longstanding presence of collision mitigation systems (seat belts, roll cages, crumple zones, etc.) has contributed to a decline in these numbers from previous years, it has failed to eliminate collisions altogether and has limited engineers’ ability to create smaller, lighter, and more energy-efficient vehicles. This is where Advanced Driver Assistance Systems (ADAS) – systems designed to avoid hazards altogether – come in.


Advanced Driver Assistance Systems (ADAS) in use today can be (roughly) placed into four classes. The first of these classes can be said to be perhaps the most hands-off of the driver assistance techniques. Systems that fall into this class are broadly known as driver-warning systems and include lane departure warning systems, lane change assistance systems, and collision warning systems (sometimes imprecisely referred to as “collision avoidance systems”), among others. Driver warning systems typically provide feedback to the driver on visual, audible, or in some cases haptic (touch) channels. Research in this area is both active and complex owing largely to the immense variability that exists between human drivers and the consequent difficulty of predicting exactly how each will respond to various warning cues. For example, where a flashing light or audible tone may be helpful to some, it may unnerve, confuse, or even annoy others.

Stepping up the level of autonomy (or “hands-on’ness”) a notch brings us to collision preparation systems. This class of ADAS seeks to prepare for or help the driver avoid accidents. Examples include systems that pretension the seatbelts, prime the brakes, reduce the speed, or even adjust the suspension stiffness when a collision or threat appears imminent or when various sensors on the vehicle sense excessive wheel skid or roll angle.

One step closer to full vehicle autonomy lie Electronic stability control (ESC) systems, which help the driver avoid skidding and loss of control by selectively applying the brakes. This incredibly-useful (and increasingly-popular) ADAS class includes anti-lock brakes, yaw stability controllers, and roll stability controllers, among others.

Still greater levels of vehicle autonomy are found in what can be called “semi-autonomous” systems. The primary difference between these systems and the more passive stability control systems just described is that semi-autonomous hazard avoidance systems actively determine a course of action that may differ from the driver’s intended maneuver and may, when necessary, cause the vehicle to deviate from the course or speed that the driver’s commands prescribe. For example, the adaptive cruise control systems that since their 2006 debut in the United States have become an icon for intelligent (and expensive) high-end vehicles, determine based on the relative speed between the host vehicle (itself) and a hazard vehicle (the guy in front of you) whether, when, and how much to adjust velocity to avoid a collision. Another semi-autonomous ADAS that exists in very limited form today is the lane-keeping system. As its name implies, a lane-keeping system actively seeks to keep the vehicle within its current lane by applying anything from steering torque overlays to differential braking commands.

At the far end of the vehicle autonomy spectrum lie the autonomous systems. The more technical/nerdy among you may have heard or read about these during the DARPA Urban Challenge, and DARPA Grand Challenge competitions. Or, you might remember Nightrider. Autonomous systems are designed to navigate a vehicle without any human input. The common approach is to plan a path through the environment given sensory information about the location and velocity of obstacles, then track that path using a suitable controller.

Figure 1: ADAS system classes arranged in order of increasing vehicle autonomy

ADAS system classes arranged in order of increasing vehicle autonomy


At the heart of each of these sytems lies the need to determine, based on the current state of the vehicle, the environment, and (optionally) the driver, the level of threat the current situation poses to the vehicle. The algorithms used to make this evaluation are known as threat assessors and can be argued to be perhaps the single most important component of any active driver assistance system. Without an accurate assessment of threat, these systems can be ineffective at best (imagine a collision warning sound incessantly and unnecessarily dinging as you drive) and downright deadly at worst (as when a lane keeping controller misreads a lane marking and sends the vehicle careening off the road). Figure 1 illustrates the relationship between threat assessors and ADAS systems.

Figure 2. Venn diagram illustrating the relationship between threat assessment and various modes of driver assistance.

Figure 2. Venn diagram illustrating the relationship between threat assessment and various modes of driver assistance.

Given the paramount importance of an accurate threat assessment, it becomes the task of any designer to determine what exactly constitutes “threat” and how to combine various sources of threat in some meaningful way. For example, imagine a vehicle traveling down an urban road amidst other vehicles. Potentially, any vehicle up ahead could pose a threat to the host vehicle were it to slow, stop, or lose control. Similar arguments can be made for other obstacles such as pedestrians, bikers, pets, curbs, etc. How, then, does one evaluate the threat each of these obstacles poses to the vehicle? Furthermore, how does s/he combine the threat posed by various (and often very distinct) sources into a single metric upon which s/he can base the decision as to how to best assist the driver? Further complicating this task is the requirement that not only must the vehicle avoid colliding with obstacles, but it must also stay within its own lane (or whichever lane the driver chooses), remain on the road, and avoid skidding, rolling, or otherwise losing control.

Though these questions may sound philosophical, they have significant practical implications for an intelligent assistance system, which must determine whether, when, to what degree, and in exactly what manner to intervene to help the driver simultaneously avoid collisions, instability (skidding, rollover, etc.), and loss of control. In my next post, I will discuss how existing ADAS systems assess threat and seek to assist the human driver and describe how these approaches fall short of the ultimate goal of comprehensive advanced driver assistance. I will then describe the alternative: our threat assessment and semi-autonomous control system.

Posted: September 29th, 2010 | Filed under: Design, Transportation | 3 Comments »

Next Generation Vehicle Control, Collision Avoidance, and Link to Sustainability

Please welcome the newest Design Impact guest blogger: Sterling Anderson. Sterling is a Ph.D. student at MIT working in the Robotic Mobility Group. In today’s article, Sterling writes about his work in the next generation of vehicle stability and hazard avoidance control, and how it relates to vehicle sustainability.

Today I’d like to briefly discuss exciting new developments in a field not commonly associated with or considered a critical component of vehicle sustainability. That field is vehicle safety. The connection I’d like to draw between safety and sustainability goes as follows: no matter what its energy source (gas, hybrid, electric, etc), a vehicle may be made more efficient by removing or otherwise lightening its structural elements. Many of these elements, however, such as secure seat belt harnesses, large airbag systems, sturdy roll cages, and large crumple zones, cannot be removed without increasing the risk of injury to vehicle occupants in the event of a collision. This limits the degree to which vehicles can be made smaller (which reduces drag), and lighter (less mass) without forfeiting the structural protection provided by larger and more massive vehicles.

Enter driver assistance systems. In recent years, the historical focus on passenger safety in human-controlled motor vehicles has shifted from collision mitigation systems such as seat belts, airbags, roll cages, and crumple zones to collision avoidance systems, which include anti-lock brakes, yaw stability control, roll stability control, and traction control. Whereas collision mitigation systems seek to reduce the effects of collisions on passengers, active collision avoidance systems seek to prepare for and avoid accidents altogether. This accident avoidance reduces – and may one day eliminate – the additional mass and design constraints required by passive safety systems.

But while existing collision avoidance systems are effective at reducing accident frequency, they are still limited in one respect: their avoidance methods are fundamentally “reactive” in nature. In the majority of these systems, controller intervention is based solely on current vehicle conditions, and thus cannot anticipate and prepare for future threats. For example, an anti-lock brake system seeks to help the driver avoid accidents by more intelligently applying his intended braking command – it does not preview the road ahead and decide to apply the brakes of its own accord. Ditto with stability and traction controllers; neither preemptively seeks to avoid hazards – each simply responds to the driver’s command. Thus, a drowsy, distracted, or otherwise inattentive driver receives very little benefit from such a system as it does not engage until he begins his own evasive maneuver.

Recent developments in onboard sensing (cameras, radar, laser-based sensing, vehicle-to-vehicle communication, etc.) and drive-by-wire technology have facilitated the development of collision avoidance systems that use information about the vehicle’s surroundings, along with predictive computer models to determine the best course of action to avoid an accident. If needed, such systems intervene and share steering and/or braking control with the driver. These “predictive” systems generally attempt to honor driver intentions, opposing them only when doing otherwise would lead to a collision or loss of control. By constantly monitoring a vehicle’s surroundings and predicting a safe path through them, they may warn the driver and take control of the vehicle steering and/or braking to avoid accidents before it is too late. Much like a copilot or driving instructor, this controller intervention should strike a necessary balance between the level and frequency of intervention: not altering the driver’s steering and braking inputs “too much”, “too soon”, or “too often” while still guaranteeing that the vehicle avoid hazards independent of that driver input.

In my work with MIT’s Robotic Mobility Group, we are currently developing a predictive active safety system that predicts the “best-case” trajectory through the environment, assesses the threat this trajectory poses, and intervenes as necessary to avoid accidents. We’ve tested this system in both simulation and experiment with excellent results. As the patent is still pending, I’ll defer details until my next post. Until then, you can see a demonstration of its performance in a few simulation videos posted on the research page of my website. In the mean time, and before I’ve biased your creativity with our solution, please brainstorm your own possible solutions. We have the technology to identify hazardous conditions and help the driver avoid collisions. What would you think of driving a car with a system like this? How do we know when intervention is “too much” or “too soon”? Feel free to discuss these ideas with others via the comment section below.

Sterling Anderson
MIT Robotic Mobility Group
Sterling’s Website

Posted: June 23rd, 2009 | Filed under: Design, Sustainability, Transportation | 3 Comments »