Scaling of industrial computer vision within Volkswagen Group

Jakob Engelmann, Product Owner Industrial Computer Vision at Volkswagen Group, gave this talk at the Computer Vision Festival 2021. Catch up on the presentations from this event using our OnDemand service.

In this article, I’m going to talk to you about the industrial Computer Vision technology at Volkswagen. It’s been quite the journey, and I can’t wait to share my experiences with you! You probably already know, but Volkswagen is one of the biggest car brands in the world.

More specifically, we’re talking about applications of Computer Vision in the production of cars. That’s right, we’re not just going to talk about autonomous driving, we’re also going to dig deep into the internal processes, and really look at how this technology can make the production of cars more efficient and exciting!

Let’s take a look at our main talking points:

Interested in the whole presentation? Download Jakob's slides below:

Initial Challenges

Industrial Computer Vision focuses primarily on how we're applying Computer Vision to the production context. This is radically different from the classical end-user applications that spring to mind when we think of Computer Vision. The good news is, it’s no less exciting!

Firstly, you have to establish the starting point. Ask yourself: what's the situation in our company? That's exactly what we did! We discovered that as far back as 2019, we'd been really struggling to tap the full potential of Computer Vision within the production area.

So much potential, so little action

On a positive note, we found over 100 use cases that were already pursued across all brands and all locations. We've more than 100 production sites, so the number isn’t that surprising, but it’s still very high. Of these 100 use cases at least 60 were concrete projects with tangible activities.

Imagine our response when we realized that we had so much great potential and yet so much of it was going unused! So many of our use cases were never making it to production.

How we formed a strategy

So, we started interviewing all these different project leads we found across the brands. We wanted to identify the pain points. As you can see, we managed to identify quite a few challenges. 👇

We were pretty precise about this. The pain points are placed on a sliding graph, with the larger challenges placed closer to the right-hand side.

In a nutshell, the major ones were really missing a concrete plan for operations. It's really easy to do a pilot program, but it's often unclear how to operate a use case sustainably. Once you get to the production context, you cannot fail. There's zero tolerance for any miscalculation with things like that.

Old systems

So you really need a good operations concept for ICV and we found that classical IT admin processes were a complete mismatch at Volkswagen because they are designed for classical monolithic applications. They weren’t really designed for an agile project with a lot of releases.

If you're really forced to do an IT security approval for each of your minor releases, then that's really gonna kill you. 😩 We’re very proud of our legacy, but ironically, it was the connection to legacy that was really one of the biggest challenges we were facing.

We've a lot of old legacy systems operating in our plants, and connecting them to a state of the art technology was a real headache.

The dreaded blackbox

We have excellent suppliers at Volkswagen obviously, but they're usually operating in blackbox setups where you don't really know what they are doing or what the transparent costs are. This was a major challenge.

IT infrastructure

What we also found: IT infrastructures that the projects used were not really consistent. This meant that the potential for scaling wasn't really there at that point in time. This is where we started off. Some very promising technology, a lot of potential, but a lot of it sadly going to waste

This was our jumping-off point into formulating some kind of vision going forward. Volkswagen is very cost-focused. So, as a target for motivation, we felt that double-digit million-dollar savings should be in reach for Computer Vision.

Develop and scale group-wide use cases

Here we were really looking at identifying and implementing use cases that were impactful and would save money. Obviously, with this, there’s a huge implementation effort. This is what we're really driving on the central level where I'm most active.

Use case transparency

Okay, so someone bring out the violin. 🎻 We had the luxury problem of having so many ongoing use cases across the company, but we were struggling to identify which were really profitable applications.

A classic situation that you'll face in any big company is that there's a lot of pilots, but no one really thinks about what the business case is behind them. Are we really pursuing the most profitable application here?

Screening of production sites

One of the first things that we did was to take a couple of production sites, talk to all the employees on-site and really identify a long list of ideas. The two most crucial elements to identify were: what would be feasible and attractive applications for Computer Vision.

Now, I'm not lying, there were at least 50 use cases on every site. That’s why it's super important to look at.

Prioritization of use cases

Now, admittedly, this was very time-consuming. The production workers weren’t really used to the technology, and they didn’t always know how to assess how expensive/profitable each application was. So that's something that took quite some time.

Most of the 50 applications didn’t have a positive business case. This meant that we usually ended up with like 2-4 applications that were really profitable.

So, to sum up, here’s a step by step breakdown of the process:

We assessed a couple of production sites.
We identified the various use cases that could be found.
Finally, we looked at whether they were profitable and scalable.

In case you’re curious, here are some examples of some of the use cases that we looked at.

Use case examples

Type label inspection

You might have noticed this type of label in your own car, the little black label. The location of it actually varies from country to country.

Sometimes workers will inadvertently damage it as they’re putting it on. There are a lot of things that can go wrong. Also, if you export your cars, then you will have big trouble if your type labels are not correct.

If you have a German plant, for example, but there's a Chinese type label in it, the German worker will struggle to check if it’s the right label.

Crack detection

At Volkswagen, we are always pressing metal parts on a large scale. Every couple of 1000 parts there might be cracks in it. It’s essential that you identify this because, if it goes unidentified, you could end up crashing the whole car.

Traditional Computer Vision didn't really work out here because it had too many false positives and false negatives. With modern Computer Vision, we found that the precision and recall were far more accurate.

Assembly check

This is a really straightforward use case. We’re simply checking the underbody of cars. We’re doing it now with cameras, segmenting the underbody of the car, and then checking against the production IT systems. This obviously saves us a lot of time on manual visual inspections and ensures that we're always legally compliant.

Implementation

The first use case I showed you has been a major success for us in terms of our Computer Vision implementation. There’s a fixed camera installed here. Whenever there's a car crossing by, there's a trigger from the IT system that triggers a photo of the type label.

This is then stored in the cloud, where we check whether the label is in the right position. This is a very positive use case that we can roll out across a lot of areas.

Democratize industrial Computer Vision into plants

The second pillar is democratization. This is about how we can bring Computer Vision into our individual plants. It’s about how we can enable these plants to carry out their own pilots at high speeds.

Democratization really means that we're really focusing on smaller projects.

👆 On the upper part of the slide here, you can see that there are four major steps in the use case pipeline.

If all four steps are successful, then you can scale it across all other brands. As I said, we’re really focusing on scaling, so we’re really relying on the use case working nicely.

When we have enough pilots and enough use cases going on across different plants then we can take them and scale across the various plants. We found that with our central approach, where we take major use cases and develop them together with relatively costly VW developers, that's not an approach to take when developing all the smaller use cases.

That's why we were thinking about how we can bring the technology into the plan so that everyone can target their local problems.

Democratization really has two major aspects to it.

The technical aspect

You need to enable plants to do their own CV models and develop them on their own. If the model works, you need some kind of feedback into the system. For us, the PLC is the main computer setup that we're working within the production context.

We’re working on a very simple self-service PLC interface so that once you have a nicely working CV model, you can detect the defects you want to see. Then, once you have detected the objects, you want to check that you have an easy interface back into the PLC.

You need to be able to tell the system that there's an error, or you need to be able to tell them to stop the production.

Organizational democratization

This is a pretty major challenge. You need to find a really lean way to deal with IT security and open-source approvals. If you’re going to have small pilot programs in the plants, you need to make it easy for the plants to handle that kind of complexity.

So far, this is working really nicely. We have a great developer guide for any new plant that joins us, which shows them how to set up our platform and how to connect cameras, etc. and the results are really promising.

We’re really looking forward to having hundreds of local pilots in our plants by next year.

IT foundations

It’s not just about IT, it’s also about organizational foundations. Do we have the foundation to make sure the use cases we're implementing are on the same page? Do they have the same prerequisites to be successful?

Now, we have our own IT infrastructure that we set up for Computer Vision. Of course, there's a lot of external suppliers where you can buy CV platforms and startups that have easy-to-use products. But nevertheless, we found that, with our very specific requirements as a car producer, we needed to define our own platform. We called this Vision Workbench.

This is an open-source software. The platform can handle most of the classic CV use cases such as object detection, classification, segmentation, and so on. This is stuff that's pretty familiar but still covers 95% of all the use cases we found in our plans.

Feature list

What’s more exciting to us is the feature list on the right-hand side of the slide. This defines why our own platform is better for us than any external BI software you can get on the market. We are really pursuing an end-to-end approach from VW’s perspective.

Camera integration

Since we have a lot of different cameras working in our systems, as of today, we need to cover all the different plants, so that they can continue to use their own technology. Thankfully, the camera implementation is pretty straightforward now. It’s pretty much ‘click and connect.”

Interface to production It

This was actually our main challenge. We have a lot of standard interfaces now communicating with standard production IT systems. Those production IT systems are quite old. It's actually not that straightforward to have a working interface that connects to these.

The trick is, now that the more use cases developed on the platform, there are more interfaces that already exist. So, we've increased the speed by every use case, because at some point in time, we’ll have connected all their major IT systems.

Flexible deployment

This is a major challenge for us. Due to latency challenges, we also offer edge deployment or data center deployment options, because our production side likes to have full control over their own technology. They're still a bit hesitant to put critical infrastructure into the cloud, so it's often easier to start with some kind of data center deployment.

If that works out nicely, you can move to the cloud later. It's a nice door opener if you can offer local deployment as well.

Anonymization tool

This is very important in Europe in general. Data protection laws in Europe are really quite severe. You need to have good answers if the worker’s council or employees are asking you why you’re putting cameras in certain places. With this, you can give them the answers that we are anonymizing and there's no personal data. This is a major door opener here.

Team

We decided to go with internal resources as a start. We're working with open source software, so it's more about cleverly putting together everything that's there in the market. We now have a team of about 10 people working on Computer Vision full-time, both on a platform and a use case level.

Sustainable integration

The major advantage is that we have a sustainable integration. We’re not buying from an external provider. The problem with this is that if an external provider leaves, we won’t have a clue what we’re doing. This way, we still have full control over what we're doing.

The industrial could

I want to draw your attention to a really nice piece of technology we’ve implemented. Recently, we’ve partnered up with AWS to build what we call an ‘Industrial cloud.’

This is mostly a pretty standard AWS cloud, but it’s really tailored to the needs of production and industry needs. And by now, many of our VW plans are already connected to that cloud. This is really a game-changer for us and Computer Vision.

If you look at the use case that I talked about earlier, if you really want to scale those, you need some kind of foundation. You need to gather data and centralize it. You need to be able to train your models and then deploy your models, even if that’s only a local deployment.

The industrial cloud is our major vehicle for this. We're also planning to have a public marketplace on the cloud. This is so that our major use cases are available publicly. We’re really hoping that this is gonna provide extra additional revenue in the coming years.

Summing up: our timeline

1. In 2019, we defined our strategy. We put things in place for starting.

2. In 2020, we focused on building the platform. Not only the technical platform, but also the organizational platform.

3. In 2021, we really started to ramp up the technology. We started working on the first use cases, and we really started scaling.

4. This year, we plan to industrialize ICV, so that we have many more use cases and many more plans. Via the industrial cloud, we’re really hoping to make this externally available.

Want to see the full presentation? Download Jakob's slides below: