Physical AI on the BIM construction site

Artificial intelligence is leaving the computer and beginning to act in the physical world. For the construction industry, this development becomes particularly interesting when robots learn in virtual construction sites and later transfer their knowledge to reality. In a B4T interview, Thomas Riedel, Head of BIM Strategy at OTTO WULFF, explains how building models could serve as training environments for learning robots in the future, what is crucial when transitioning to a real construction site, and where the approach still encounters limitations today.

B4T: You are concerned with the question of what happens when AI leaves the digital world and begins to act in our physical environment, for example, in the form of robots on a construction site. We then speak of Physical AI. What exactly does this mean - and why is this development relevant for the construction industry?

Thomas Riedel: Physical AI refers to AI that no longer just works on a screen but perceives, decides, and acts in the real world through sensors and actuators. Simply put: machines gain an intelligence that allows them to orient themselves in a real environment and autonomously take on tasks there. Early humanoid robots are already showing that they can perform household tasks such as loading a dishwasher or folding laundry. If that works in the home, the question arises why not on a construction site. For the construction industry, this is relevant for three reasons. First, the shortage of skilled workers: robots can fill gaps in physically demanding or repetitive tasks. Second, occupational safety: dangerous work at heights, with heavy loads, or in contaminated environments can be shifted to machines. Third, the data basis: with BIM, we already have a digital 3D representation of the building. This model can serve as a training environment for robots in virtual environments to prepare them for use on the construction site.

B4T: Why do we need learning systems? Why don't robots on a construction site simply follow predefined rules?

Thomas Riedel: Rule-based systems solve tasks based on a fixed, programmed if-then logic. This works well as long as all situations can be described in advance, such as route planning from A to B based on a known map. A construction site, however, is the opposite of a controlled environment. It changes daily, material is not where it should be according to the plan, components differ from specified dimensions, and light, dust, weather, and people in the work area are added. This diversity cannot be fully captured in rules. Therefore, learning systems based on neural networks are needed. They recognize patterns in large amounts of sensor data, generalize from experience, and can also deal with situations that no one has explicitly programmed. In practice, it is usually a combination of both. Learning systems take over perception and adaptation. Conventional, transparent logic ensures planning and, above all, safety.

OTTO WULFF Bauunternehmung GmbH

Robots on a construction site - an AI-generated vision1 / 4

OTTO WULFF Bauunternehmung GmbH

Robots on a construction site - an AI-generated vision2 / 4

OTTO WULFF Bauunternehmung GmbH

Robots on a construction site - an AI-generated vision3 / 4

OTTO WULFF Bauunternehmung GmbH

Robots on a construction site - an AI-generated vision4 / 4

B4T: What role can a digital model play in this?

Thomas Riedel: Today, robots often learn in virtual environments, i.e., in simulations, before they are deployed in the real world. My idea is: why shouldn't this environment be a virtual construction site? With BIM, we already create and optimize a building completely digitally during planning so that it can later be built as smoothly as possible. This virtual world already exists and could serve as a training environment for a robot. An example is a virtual shell, where a robot learns to move or to attach a cable to a wall. If the shell later stands on the real construction site, the machine enters this environment with the virtually acquired knowledge. The principle is the same as with a robot training to load a dishwasher: instead of a virtual kitchen, it has a virtual building available. One challenge remains that reality never exactly matches the model. The robot must therefore be able to transfer what it has learned to deviations. This is where the learning systems come into play again.

„The principle is the same as with a robot training to load a dishwasher: instead of a virtual kitchen, it has a virtual building available. “

B4T: What is the significance of IFC, the open standard for exchanging BIM data, in this context? Does a learning robot necessarily need an IFC model?

Thomas Riedel: No, IFC is not mandatory. A robot initially doesn't care what file format its training world originated from. In a production hall, the simulation can be based on a completely different 3D model. In construction, however, the situation is special: many parties work together on a virtual world. Specialist planners and contractors generate different data that must be merged via a common, vendor-neutral schema. IFC is the central basis for exactly this. The model is supplemented by further data, for example from laser scanning, which captures the actual construction status. An IFC model also contains more than pure geometry. It describes what a component is, such as a load-bearing concrete wall or a plasterboard partition, and what properties it has. For a robot that is supposed to attach a cable, precisely this information is valuable. The biggest advantage, however, is that we create these models anyway. It is crucial, however, that they are available in sufficient quality and detail. An incomplete or inaccurate model is only of limited use as a training environment.

B4T: Can machines already derive reliable actions from current models, or do we need a standard in the sense of a Robot-Ready BIM?

Thomas Riedel: Honestly, this cannot yet be conclusively answered today. There is a lack of practical testing in corresponding training and simulation environments. At least none that I know of, but development is rapid! My assessment is that it depends on the task. If a robot is to learn to move or perform simple tasks in a virtual building shell, current models might already be sufficient. The finer the task, the higher the requirements. For example, if a robot is to learn to install a window, it needs information that is still missing in today's IFC models: exact fastening points, tolerances, or the sequence of assembly steps. Whether a separate standard is needed for this will be shown in practice. It would be sensible to first build on existing standards like IFC and define what additional information a model must contain for specific robotics tasks.

B4T: You illustrated learning in your presentation with a ball on a balance board. How does a digital building model become a "gym" where a robot learns through many repetitions?

Thomas Riedel: The principle is called Reinforcement Learning. The robot receives positive or negative feedback for its actions. Like a ball on a balance board, the system tries things out, gets a signal whether it has gotten better or worse, and adjusts its behavior. In a virtual 3D construction site, this would mean: If the robot stacks a scaffolding component and it falls down, the action is negatively evaluated. If it succeeds, it receives positive feedback. Through many repetitions, its behavior improves. The crucial point is that no one has to specify every single action. The machine finds out for itself how to achieve its goal. Here, the virtual building model becomes a training gym. In it, the robot can train for as long as it needs, and also make mistakes, until it reaches a well-trained state. In addition, the training can take place much faster than in real time and in many parallel runs.

„If the robot stacks a scaffolding component and it falls down, the action is negatively evaluated. If it succeeds, it receives positive feedback. Through many repetitions, its behavior improves.“

B4T: A real construction site is constantly changing. How does the system deal with the difference between the perfect digital model (planned condition) and the constantly changing reality (actual condition)?

Thomas Riedel: The robot comes to the construction site with knowledge from the virtual world. It knows the planned building and basically what is supposed to be created there. The model is a guide for it, not a hundred percent exact representation of reality. Because it was confronted with many different situations during training, it can also react to changes. It has learned to generalize. Through its sensors, it constantly compares what it actually finds. If a scaffolding component is not in the planned location, it recognizes that it is missing, searches for it elsewhere, and adjusts its path. Because it learned in the simulation what a scaffolding component looks like, it also recognizes it in another location.

B4T: What role do sensors and real-time data play in this?

Thomas Riedel: A central role. The model tells the robot what should be, the sensors tell it what is. Much will work through cameras and the evaluation of image data. In addition, there are laser scanners that capture rooms three-dimensionally, as well as sensors for position, movement, and force, for example, to safely grasp a component. It is crucial that this data is evaluated in real time. If a person enters the work area, the robot must be able to react immediately. Many prerequisites are already met on assembly lines in production halls. However, how a robotic system on a real, complex construction site also deals with rain, snow, dust, or wind is a further development question. Nevertheless, such conditions can also be simulated in the virtual world to optimize training.

B4T: What information beyond geometry must models contain for an AI to derive actions from them?

Thomas Riedel: Many of these requirements already exist today, independently of robotics. If we want to work effectively on digital construction sites, we need classified components, clear data structures, and defined information about walls, ceilings, or building services routes. We already define the content a model should have on a project-specific basis. For a robot, information that tells it what it can and may do with a component is particularly important. This includes the material, for example, whether a wall can be drilled into or if it is load-bearing, as well as attachment points and tolerances. In addition, there is the temporal dimension: In what order will it be built, and what state is to be expected at what time? Areas that are blocked for safety reasons should also be recognizable in the model. For initial tests in a building, however, I suspect that not much more would be needed than what a consistent BIM application already requires. However, this would have to be tested in practice.

B4T: Beyond the construction phase, where do you see potential for robotics in the life cycle of a property, for example in facility management or maintenance?

Thomas Riedel: I even see the greatest potential there, because a building is operated for much longer than it is planned and built. Clients are already ordering digital twins today to use building information during operation. If a malfunction occurs, a person is still sent to fix it. In the future, a robot could take over some of these tasks, such as regular inspection tours, condition checks, or simple maintenance work. The data basis is the same for humans and machines: reliable information about the building. Today, this information helps to send a person specifically to a malfunction. In the future, it could be a humanoid robot.

„In the physical world, another dimension is added. Robots can not only cause property damage but also endanger people. “

B4T: The more autonomously such systems act, the more important safety and responsibility become. Who bears the responsibility if a learning system causes damage?

Thomas Riedel: There is no definitive answer to this yet. Basically, responsibility will be distributed among several parties. The manufacturer is liable for ensuring that its system is safely designed and sufficiently tested. The construction company as the operator is responsible for ensuring that the robot is used as intended and that occupational safety on the construction site is guaranteed. What is new is that a learning system changes its behavior and not every decision is traceable in advance. In the physical world, the question also takes on another dimension. Robots can not only cause property damage but also endanger people. We know similar discussions from autonomous driving. Therefore, I consider it crucial that safety is considered from the outset: clear protection zones, the ability to intervene at any time, and traceable records of what a system has done. Beyond that, ethical questions remain that go beyond pure technology.

B4T: Will the robot be an autonomous actor in the future or rather an intelligent tool for humans?

Thomas Riedel: Both, but on different levels. The robot will be autonomous in execution, but it remains a tool in terms of objective setting. Humans specify what is to be achieved, but no longer describe every single step to get there. The system finds the way itself. For this, clear guardrails are needed. Humans set the goal and the framework within which the robot acts independently. And they retain control: They can intervene at any time and ultimately bear the responsibility for its deployment. At least in the foreseeable future, I therefore see the robot as an autonomously acting tool in collaboration with humans.

B4T: If we have this conversation again in five years, what will we then be talking about as reality on construction sites, no longer just as a vision?

Thomas Riedel: Development is progressing so rapidly that a reliable prediction is difficult. However, I can well imagine that the first physical AI systems will then take over fixed tasks on construction sites. In five years, we will hopefully no longer ask whether robots can work on construction sites, but what tasks we should assign them next. And we may no longer create our BIM models only for planning, construction, and operation, but also as a training environment for machines. I would leave it open whether Germany will be among the first markets to adopt this.