Girish Venkataramani, Senior Director, ML Accelerators at Cruise, gave this presentation at the Computer Vision Summit.
Hello, everyone. I'm really glad I made it here. I think that a really good topic to talk about today is transportation. But before we get to that, let's all acknowledge where we are in this AI revolution.
Whether you like it or not, AI is upon us. I remember about 10 or 11 years back when we had the first deep learning models. Everything changed from there. We sold vision with ResNet, and so on, and then we went on to solve speech with transformers and BERT. We didn't really solve it, I'm just joking.
The point is that AI is constantly evolving. The pace of innovation in AI is tremendous, from the first deep learning model that solved image classification problems to today, where we have ChatGPT and generative models taking us to another dimension.
The question is, how are you going to adopt AI into your business? What's the right model? What's the right way to integrate it into your systems? What's the cost of AI?
These are all very pertinent questions you should be asking yourself in terms of how it affects your business.
But one thing is clear - we have to adopt and adapt to AI. So part of the accelerated growth of AI also means the accelerated growth of adoption, integration, and execution of AI systems in our businesses.
Today, I'm going to talk about how we do that for our AVs or self-driving cars, and give you a little bit of a preview of what's coming down the line.
Improving transportation in our cities: The mobility revolution
Needless to say, I think we can all agree that transportation, or mobility, as we like to call it, is not in the most desirable state in our current societies.
Cruise’s mission is to drive us safely and connect us to places as efficiently as possible. But it’s also a very personal mission for me.
Going back to when I wasn’t even 10 years old, I remember my next-door neighbors deciding that they were going to move. I was growing up in Bangalore in India.
They bought a house about 10 miles away, and I really wanted to see my friend. So I said to my mom, “We’ve got to go and visit.”
We didn't have a car or any transportation of our own, so we had to take public transport. And it took us about two-plus hours on public transport to cover those 10 miles. This was back in the 80s. Oh, wait a minute, did I just date myself there?
So, here we are today in the 2020s in the United States. But guess what? We haven't solved that problem. Even in the US today, going 10 miles using public transport will probably take you around two-plus hours. So that's a problem I want to solve.
When I moved to the US, I decided to use public transport as much as possible. I was working in Boston and my workplace was about 15 miles away. It took me an hour and a half to get there. Something wasn’t working well.
Apart from transportation as it is today, there are a number of other reasons why mobility needs some help. Safety, the environment, the amount of pollution that we're creating, carbon emissions, and the amount of space we allocate for cars.
In some cities, there’s 20x more parking real estate than household real estate. This is true for certain cities in the US, and most of the time, these cars aren’t being used. Our utilization of our personal vehicles is probably around 10% of the time.
We can do better, and we can invest in technology that can improve the state of mobility. In fact, McKinsey did a thorough report on this. They believe that we’re at the beginning of a mobility revolution. So when we talk about a mobility revolution, we're thinking about EVs, EV infrastructure, and AVs.
Together, the ecosystem can get us to a better mobility future. And they believe that by 2040, about two-thirds of all miles driven in the US will be autonomous. We’re at the beginning of this revolution.
At Cruise, we want to lead the way through to this future. And I'm going to talk about how we got started and where we’re at in our investment in this domain.
How Cruise is disrupting the status quo of transportation
I want to start off with this example, which is a great representation of what we can do today. It’s one of our driverless cars. There’s no driver in the front seat, and it was navigating its way through St. Patrick's Day in San Francisco last month.
On the front cameras, you can see how crowded the scene is. The top is what the software sees. You can see the main vehicle, which is our Cruise vehicle, all the little circles are pedestrians, and the boxes are other cars. You can see what our software is thinking in terms of the detection and tracking of these objects.
You can also see a faint green patch in front of our AV; that's the path planning. That’s where we’re trying to go.
But when we saw this and we realized what our cars could do, it was phenomenal.
If you stay focused on the bottom frames below, you wouldn’t want to be driving in this mess. I know that I wouldn’t want to. This is some of the messiest street traffic that we’ve seen.
And believe it or not, our AV was able to navigate it just perfectly. I was able to go around pedestrians, go through the traffic, across all the bars, through narrow streets and double-parked vehicles, and figure out a way to get out of there.
We were amazed when we saw this video. It was really satisfying to see all the work we’d put in to get us to this stage.
The evolution of Cruise
So, where are we today? We launched our first driverless vehicles back in November 2021. And today, we’ve clocked more than 1.5 million miles in SF without a driver in the front seat. This is incredible by any standards.
We haven't had any severe incidents, and so we’ve built enough confidence and trust in the safety of our vehicles.
But how did we get here?
Cruise was founded back in 2013, so about 10 years back. And if you look at this timeline below, there's a lull in the first eight or so years. That's actually intentional because that's when we were investing in the technology. We were in an R&D mode. We were actually figuring out how to drive these cars. We were testing and iterating.
And as we were going, we started to build confidence in our investors. We got a lot of big investments from very prominent investors. And it was in November 2021 when we launched our first driverless rides in SF.
And you can see the scaling which has happened in the last 17-18 months to today, where we’re now operating in all of SF. In fact, we just announced yesterday that Cruise AVs are available 24/7 in the entire seven-by-seven ODD of San Francisco. We’re also launching the service in Austin and Phoenix.
We’ve got a great roadmap ahead, and I'm going to talk to you a little bit about that.
The self-driving problem
If you think about the technical problem and what we’re trying to solve, you can see that there’s a very standard way of defining autonomous driving.
You start with sensors, you get sensor data, cameras, radars, whatever you have. And then you have perception as a subsystem, which is making sense of this data. Who's around me? What's the environment like? Where's my car? Who's coming at me? Where are others moving to? So that's perception.
And then comes tracking. Tracking is trying to figure out what the objects are around me. Pedestrians, cyclists, other cars, and other vehicles.
And then prediction tries to predict their path. Where are they going? Where are they headed?
And then on top, you have localization mapping, which tells us where our AV is, and free space tells us where we can drive. So all this is input to our planning stack, which takes our input and figures out where to drive the car. And this needs to run in real-time on our AVs, within the hardware that's in our AVs. That’s the problem we've been solving.
Now remember, this problem has been solved in a way. We've all figured this out. Companies all around the world have figured out how to implement the system. The difficult part is to make sure that this system can work reliably 100% of the time.
If you were working on an L3 or an ADAS system, a lot of your cars probably have it. You have your safety packages, you have some self-driving Teslas, but also other cars like Mercedes and BMW have it. You have the expectation of grabbing the wheel when something goes wrong.
In that context, if you can drive autonomously for 90% of the time, it's really good. The software’s doing well. But for a fully self-driving, driverless system, it’s not good enough. We've got to burn down that 10%. We've got to be able to drive autonomously all the time.
So, what’s the problem there?
Obviously, driving autonomously all the time means being able to adapt to any situation that's thrown at the AV.
Below are just a few examples of the types of scenes that our AVs have seen. All of this is camera footage from our AVs, and you can see the craziness that happens in SF. These are things that we learn and navigate and figure out how to drive through.
This is your long-tail. This is that 1% of scenarios that are really difficult to navigate through. And we've got to solve this problem if we want to get to a fully driverless mode.
Safety is our North Star at Cruise. It always has been. We won’t put a product on the road if we can’t put confidence in our safety metrics behind it. That’s always been our first priority: safety.
To summarise, our AVS must be capable of driving autonomously 100% of the time. So we've solved this problem 99% of the time, but it’s not good enough. Burning down that one last person. Usually, if you've worked with long-tail problems, you know that's where 99% of the effort goes. That's the long-tail investment.
And it's an engineering problem here. You've got to test, iterate, come back, see what the results look like, send your cars into all these kinds of weird situations. Where are you performing poorly? Send your fleet out to test, bring back the results, analyze it, and start the process again.
This is about engineering velocity. If you want to make it, we've got to accelerate the cycle of innovation and engineering to actually deploy autonomously.
The AI lifecycle at Cruise
At Cruise, this is what our AI lifecycle looks like:
We have a testing fleet that’s on-road, driving around, and gathering data. But we’re also starting to build our simulation system so we don't have to test on the road. Instead, we can get the signals from our simulation systems, and that's going to be the future of scaling too.
Whatever method we have, we test and we get some data. There’s a lot of data. There are petabytes of data that come in every day and we’ve got to manage that. That's part of our data platform.
And then from that data, we get two things. One is insights into what we're doing well and what we're doing poorly, and then we analyze that and figure out what our strategy is, what we need to change, and what models we need to update.
And at the same time, take the data, mine it, ground truth it, label it, and make it available for training in the future. So this cycle is always continuing.
Then when our ML engineers are building new models, they use newly labeled data sets, train the model, and then we have to optimize them.
We’re going to run this in the car; it’s not running in the data center, it’s not running in the cloud, so it needs to run on its own in the hardware in the car. And it poses its own challenges, which I won’t get into.
So, real-time integration, and then you start the cycle again.
Now, once you deploy AI into our cars, the mistake that a lot of people make is they think that once you deploy, you're done. Let me tell you, in AI, there’s no such thing as ‘done.’ Your model is never done being deployed. You're always going to improve it with data. Data is what makes AI great.
We’re getting new data all the time, so we need to update and upgrade our models with this new data. And this is where the acceleration comes in. We want to automate this whole process of getting data, ground-truthing it, mining it, improving our existing models, and pushing it back into the car.
We’re looking to automate the whole lifecycle without a human in the loop. We call it the continuous learning machine.
In each of your enterprise systems, you might have a similar problem; you’ve got to figure out how you're going to keep your models up to date with new data.
Cruise leads the pack in driverless miles
Below is a chart of how we've been doing. What I'm showing you is public data. The California DMV collects this data, and it's been collecting it for about seven years now from all AV companies which are testing on the road.
What they collect is the number of autonomous miles that the fleet has driven, and what they call disengagement. Disengagement is when the safety driver needs to take over. Something went wrong, the software disengages, and the safety driver takes over.
The y-axis is miles per disengagement. We want that number to be high. And I'm showing you the curves. This is all public data of all the different companies that are testing in this market.
I hope this is pretty obvious, but I see a big orange hockey stick out there. I'm not going to go into detail as to why we're doing so well, but we drove more than 10x driverless miles than any other company or all the companies put together. We’ve had a quantum shift in the last year, and this is where Cruise is today.
How did we get here? Through our guiding principles in the early days for safety.
As I mentioned, we've got to put a safe product on the road. That’s achieved through engineering velocity. But let's not forget about the need to work with the government and with the regulatory frameworks to actually get the permits and get us going on the road. So this is a huge investment, and we have a great team that’s got us to where we are today.
Launching autonomous vehicles commercially
So that's how we built this technology. We launched driverless cars. But what are we doing next?
Well, now we’re at the scaling phase. We've got a fleet of hundreds of cars driving on the road, and we’re looking to scale this by orders of magnitude. Tens of thousands of cars, production-intent vehicles, across multiple cities, and in any kind of conditions. That’s the inflection point that Cruise and many other AV companies are at today.
And the challenges we have to solve, in addition to the three previous challenges, safety, engineering velocity, and government regulations, is that we have to think about rider experience, our infrastructure and ability to scale reliably, and most importantly, product-market fit. We oftentimes forget about that.
We’ve got great technology, but how are we going to leverage it from a business perspective?
From a product perspective, we have two products: ride-hail, which is similar to your Lyft and Uber kind of service, and delivery. We’re partnering with Walmart to deliver groceries for them. So these are the two products that we have.
The next question is, how do we define the product-market fit? This is a very challenging problem. I can give you a little insight into how we did it in SF. We didn't go from zero to one overnight.
When we launched, it was a small fleet. We had three dimensions that we were scaling on: the size of the fleet, the time of day we drive, and the geofence, or what we call the ODD, which is the operational design domain in which the cars drive.
And then we slowly expanded that out. It was a small fleet in a small part of San Francisco, only at night.
Today, we have a large fleet in all of San Francisco, 24/7. That’s an iterative improvement. That's your ops team figuring out the product-market fit.
And now we're launching in Austin and Phoenix as well. We’re scaling this to multiple cities, and we want to show that we can do this effectively. It’s the same stack running in different cities. So that's where we're driving today. We started small in SF, and now we’ve covered all of SF.
Right now in Austin and Phoenix, again, we’re starting small, and we expect to cover the ODD of these cities in a short amount of time.
Reducing costs is critical to successful commercialization
Another part of commercialization, when you think about product-market fit, is the cost of the platform. You need to make money. You're generating revenue, which is great, but what's the cost of the platform? Can you generate profits?
Cost is a huge contributor. The cost of an AV is the vehicle cost, the sensors, the compute, networking, and the hardware in the car.
This is where we've had a lot of public debates about which sensors are expensive and which sensors are needed. I won't get into that, but they all factor into this equation.
We’ve got to bring the cost down, which means we’re trying to run more complex AI models on a much more compressed hardware system. That’s a challenge that we have to solve in order to make this a reality.
I'm going to point you to a blog article that I published recently, where we talk about this problem and why it’s a difficult problem to solve. The article is called, ‘AV Compute: Deploying to an edge supercomputer.’
I'm going to pick one thread here, which is the compute.
If you think about AI hardware today, there are two ends of the spectrum. You have the data center hardware, like your NVIDIAs, TPUs, and whatnot, which are big, honking, high-capacity hardware systems, but are also power-consuming and very costly.
On the other end of the spectrum, you have your tinyML, like your mobile and IoT hardware. They’re very power efficient, but not scalable when it comes to capacity.
We fall in the middle of the spectrum, so there’s not much of an ecosystem that’s there to support us. So what we've realized at Cruise is that the only way to continue reducing cost is vertical integration, which means the entire platform, from the vehicle, up to the sensors, to the compute, to the platform, and the software and AI models as well. So that’s one of our driving principles as well.
Now, I'm happy to say that we’ve invested in building our own hardware which will go into the cars. We have silicon that we've created, which is going into our next-generation platforms, and we’ve seen a tremendous reduction in cost because of that.
To summarise, we have our first-generation AV that you'll see in SF. If you go there, you'll see our cars around. And we’re very excited about our next-gen-AV. It’s often called the Origin, and this is a production intent AV with no steering wheel or gas pedals, which is going to ferry passengers around.
What does the future hold for Cruise?
So, where do we go from here?
Think about that McKinsey report I mentioned earlier. In a decade or two decades, we want to cover the entire world. We’re already starting on this path. We already have partnerships in Dubai and Japan to start testing and deploying our fleet out there.
Additionally, we’re already starting to solve more complex problems, like how to drive in weird weather conditions like fog, rain, and snow. These are tough problems that are yet to be conquered, and that are on our radar.
So, in order to achieve this, we've got to accelerate our AI technology. I talked about the lifecycle, continuous learning, etc. Innovation has to be accelerated. But we also need to accelerate our operations.
As I said, commercial operations, business operations, product-market fit, getting the fleet reliably on the street, all of these things require data analytics. It doesn't just come from our gut. We have to be data-driven.
There's data and AI in this whole equation, all over the place, and we're only going to succeed if we get to that level of acceleration.
