Mark Brinkmann, Head of AI Non-Retail at Schwarz Group, gave this presentation at the Computer Vision Summit.
Check out the highlights below, or watch the full presentation on demand here.
Hi, everyone, I’m Mark Brinkmann, and I’ve been with Schwartz Group for the last seven years. For the first four years, I was responsible for the manufacturing division for the whole application management's support development operation.
Three years ago, I moved to the headquarters and started the AI industry team, where we focus on AI topics throughout our industry divisions. I was employee number one within this team and started many, many prototypes. And so far, we’ve been quite lucky.
Today, I want to talk about one prototype which is currently being rolled out. At the beginning of next year, this prototype will be a working product.
Who is the Schwartz Group?
One of the divisions within the Schwartz Group is retailing, which includes Lidl and Kaufland. And this is one of the main parts of the Schwartz Group.
However, there are two more large divisions. One division is PreZero, which is a waste management recycling company. In the last few years, they’ve become a major player in the European waste management area.
I started my journey at Schwartz Group within the Schwartz Production division. In Germany, we currently have 15 factories, and one of these is one of the largest ice cream factories, also where we fill up water and other beverages.
Our first factory outside of Europe is going to be built in England, and it will be in production in the next few months.
The latest factory was one of the largest coffee roasteries, where we have one of our first use cases in the testing system.
But today, I want to talk about the bakery process, which is also one of the largest factories in Germany, if not in Europe.
So, as you can see, we’re living in this circular economy. We’re in a very unique position where we have everything beginning from the production and manufacturing, through to the retailing, and back to the recycling management processes.
This is quite interesting for us as AI because we get the data from manufacturing, retailing and recycling, so we really don't have a data problem. We have more of a resource problem, but more on this later.
So, a few numbers for our company. We have around 550,000 employees, and over 13,000 stores throughout the world. We also achieved over 133 billion euros in sales in 2021.
One thing that makes me really happy is that we really think about sustainability within our company and its importance.
As you can see below, we have four pillars of sustainability: people, product quality, circulatory systems, and ecosystems.
You can see that even in our strategy around retailing, recycling, and manufacturing, we’re living it and not just talking about it.
And today, the main thing for me is this cautious use of materials, especially in bakery goods.
Reducing food waste in the bakery process
The case study today is some ideas about how we tackle the problem of waste management within the bakery process.
So how do you reduce food waste in a bakery process? We use milk and dough which are very precious goods, so we need to make sure not just from a financial point, but also from a sustainable point that we’re able to get the most out of this process.
In order to start this, let's talk a little bit about the manufacturing process. As you can see in the picture below, this is the whole process from the beginning where you have the dough, you mix the dough, and then afterward, you form the bread. Then it’s transported through a pre-baked process environment and it gets shipped to the retail stores.
We really need to measure this process. But why do we need to measure it? Let's assume we put 100 kilograms of dough in the front of the machine. In the end, we only get 90 loaves of bread, but we should be getting 100 loaves of bread out. Sure, we have some water loss within this process, but in order to not make it too complex, let’s just assume that we have 100 loaves of bread as a goal, but we only get 90.
So, what do you do? Where do you lose this bread? There are many reasons for this, mostly mechanical, where something might’ve fallen down or gotten stuck, so you can't use the bread anymore.
How do we count this bread? It's very easy. You just need a laser counter. You build it in, you create a metal frame, and every time the laser’s interrupted, you count it.
It was very cheap; the counter cost was under 100€. If you have a very good purchase manager, you can get one for maybe 50€.
You plug it into a PLC, and this connects to the line management system. It’ll operate for many years and will require very little support and maintenance. You just have to clean it sometimes. If you do encounter any issues, you can just send in an engineer or somebody who runs the line, and they can easily repair it.
But what happens at the other places of the system? You still have 90 loaves of bread instead of 100 loaves of bread. As you can see below, the view is different and you can't use any laser counters anymore.
So how do we account for this? The only solution is to use computer vision. So what does this look like?
Streamlining the bread counting process with AI
We have a different setup here. You need to build a whole metal frame around the line and record this.
It's not easy because you're working in a very clean environment. It's not ship manufacturing where you don't have any air distraction. However, it's still food, and food needs to be preserved and kept clean.
Therefore, we built a custom metal frame throughout our lines. And one important thing is also the lighting structure. At some points, you just put the camera in there and it works very well. But at other locations, you really need the lighting in order to count correctly.
The second part is that once you have some metal frames set up with lighting and cameras, you need to take care of the data connection.
You could send it to the data center on the factory level, but you have a huge problem because if you put in new network cables, you have a fire hazard problem. Every connection through the wall is certified by the fire department, so you can’t just tear things open. You need to have the compute power directly at the line.
Therefore, we built a custom PC with our hardware colleagues.
Setting this up and getting approval from everyone was quite difficult. But we got the approval from the relevant bodies, and we’re now in the process of rolling the system out.
But the main thing which took us the most time was the setup of the line. We needed to get into the switch cabinet, which was very close to the line. And you can see in the picture below that the switch cabinet isn’t very big. You only have a few spaces left in it.
So we had our network cable and our camera was powered over Ethernet. But then we had to be able to operate this 24/7 throughout the whole year. The factory is only closed on Christmas Day.
We thought about it, and we realized that we had to have a very flexible system. So we started with our hardware. The hardware is standard, however, custom built. And on top of this custom build, we looked into our setup and decided to use the Linux distribution.
On top, we implemented our NVIDIA drivers and libraries. Then we got our Docker runtime. And together this is our “golden image,” which we can use and send it throughout our devices.
However, in order to be very flexible, we decided that all the applications we wanted to send out should also be Docker-based and container-based. Therefore, we decided to use a Kubernetes agent.
We didn't use the large Kubernetes. We instead decided to use the K3s, which is much smaller for the deployment process. And the main reason for that decision was that we don't have this large data internet connection into the cloud.
We needed to look at the size of our container as well. The NVIDIA libraries, as you might know, are almost eight gigabytes, so we needed to look into how to scale the size of our images down.
On top of this are the containers that we built. We have the ingest container, the interference container, and the communication container.
In the end, we linked our libraries into our container, and on top of it was our application.
So that's the whole system. This is our AI@Edge stack. We can run this on every device where we deploy it.
Also, our support process is quite unique. If this system breaks down, we just tear out the hardware and plug in new hardware. The first-level support can identify it right away.
And the process starts automatically. Everything from the cloud down to the system is automatically deployed. It’ll run as soon as all the containers are up there and start automatically.
In order to get this down, we also looked into our deployment process. We did train in the cloud, and we also started our own hyperscaler at the beginning of this year called Stack It.
The vision is, of course, to compete with the big two or three. However, we’re in the very early process here, and right now, we’re still using one of the three major hyperscalers.
Once we built our process in the cloud or deployment pipeline or container, we automatically sent it to the factory data center, and from the factory data center, it goes out to all industrial PCs on the line.
This process not only enables us to do this in the manufacturing plants, but also for all the other divisions. We can use the same architecture, the same deployment processes, and the same support processes, which provide high synergies throughout the company.
We’re now in the process of rolling this out. And at the beginning of next year, we’ll actually have the first lines of the product.
Taking AI beyond the factory
So what does this look like? We have the bread, and not only are we detecting the bread, but we’re also classifying it. We track the bread throughout and then we have our counter.
And the possibilities for this aren’t just limited to counting. You also could look into this solution for quality issues. Quality issues could include bread sticking together, or if there’s just half of a bread, we can easily recognize this and have live, real-time monitoring.
So our colleagues are very happy, and they’re really looking forward to this tool. They’re able to track what we’re building throughout the manufacturing process.
We now have a combination from the beginning, where we have the dough and the laser pointers, which are installed by our colleagues in the factory. And then at the end, we have multiple computer vision stations where we record it. And the product is therefore very well received by our colleagues at the factory level.
We do think it's a whole process. As I said in the beginning, we not only look at the factory, but we also look at the store itself. We use demand forecasting for our bakery goods in our Lidl stores, as well as store optimization for product availability. That’s an AI process and has AI technology behind it.
