MADDOX System Architecture
MADDOX as an assistance system for machine operators is used in industrial production environments. The diverse boundary conditions require a wide range of solutions, for example, for machine interfaces. On the other hand, the software should be able to be integrated as flexibly as possible into the customer’s existing operational processes and system landscape. Our assistance system is therefore not just one product that is simply rebuilt after each change and made available to all customers for download. At Peerox, we have recognized that special solutions are needed in a system’s architecture to meet these special requirements.
We thus adapt MADDOX to the conditions of our customers. When we start working with a new customer, we first have to connect to the machines on site. If interfaces in the form of common industry standards (e.g. OPC UA) are available for this, it is often done with a one-time configuration. In other cases, we have to reprogram the interface. In addition, there are customer-specific features that we create as plugins and can switch on and off with a simple switch in the configuration.
Last but not least, MADDOX is the product of a spin-off from the Fraunhofer IVV in Dresden. It continues to be developed in cooperation with the researchers of the Digitization and Assistance Systems working group, as well as in association with external service providers.
Executing customer-specific requirements and coordinating development with our partners forces us to have a good grip on our processes, but also to modularize the architecture accordingly and design the interfaces well so that each team can work effectively.
To avoid losing sight of the big picture, we set a few important goals early on to keep technological issues out of our way as much as possible in order to focus on the challenging tasks. In this section, we highlight some of the techniques and technologies we use:
Let’s go.
Python Projects That Build On Each Other
MADDOX is implemented in Python. The source code is not managed in a single repository. Core components are developed modularly so that they can be reused. In fact, there are two of these core components—our own core component forms, among other things, a backend with an API gateway based on the well-known web framework Flask, as well as the user administration and our knowledge base. The other core component is being developed at the Fraunhofer IVV Dresden, with whom, as mentioned, we are in close cooperation. There, functionalities for machine data acquisition and processing, as well as algorithms for machine learning, are continuously being further developed as the subject of research projects.
The components we use in a modular way can be created, managed, and used with the home remedies of the Python community. With set-up tools we create a library archive, which we upload to an internal Python Package Index (PyPI), and then we incorporate dependent projects, just like external libraries with the Python Package Manager (pip). Deploying a MADDOX installation generates a nested project structure.
The splitting into several projects is a natural consequence of the interplay between the partly individual requirements of the customer projects, as well as our organizational structure. This results in a certain inherent complexity, which nevertheless enables us to create customized, yet scalable solutions. It also enables us to seamlessly integrate experts for web front-ends or machine learning directly into the development process.
Service Architecture And Message Broker
Due to the different environments at our customers, we are forced to keep our software architecture as flexible as possible. A single component for pre-processing machine sensor data may have hardly tasks to accomplish for one customer, and be exposed to very high loads for the next. While in the first case, it makes no difference whether it runs in the main process, in the other it can potentially become a bottleneck and can ideally run in parallel to its own process or even on its own machine.
In early development versions, we had initially created a monolithic system that managed multiple processes using its own process runtime. However, we quickly realized that we could accomplish the same thing with less effort by using containers. After all, containers are processes in their own right and provide some additional features that we like to take advantage of (see section “isomorphic builds and infrastructure as code”).
As a result, we switched our entire software to containers and ended up with a modular service architecture. So there are several modules for reading and translating sensor values and machine states, one for managing knowledge modules, several machine learning modules, and so on. We also use this strategy in customer projects to add customer-specific features as needed in the form of our own plugin modules.
The modules communicate with each other via a message broker, i.e. they send and receive messages on a common channel without knowing the dialogue partner. Except for the common message types and the common database, the modules are therefore completely decoupled. In contrast to a point-to-point topology, the additional work involved in distributing the various modules to different systems is thus limited to providing each module with the address of the message broker as a configuration variable.
With the service architecture, we use a bit of the best of both worlds: service-oriented and monolithic architecture. The architecture is very malleable, but that only comes at the price of as much complexity as is absolutely necessary.
Automated Tests
There is certainly no need to say too many words about the advantages of automated tests.
In our case, the potential damage caused by tests is significantly higher due to the project landscape described in the previous sections than if we only had a central mono-repo. Efforts and costs to fix bugs are much higher in production operations than they are during the development process. The same applies to a bug that is not found until the start of a dependent project (e.g., in a customer’s project) but has its origin in an underlying internal library. From this point of view, we benefit several times over from every test that catches bugs on the spot.
Nevertheless, we are not dogmatic here. One hundred percent code coverage and a test-first dogma has advantages, but in view of frequently changing requirements, it naturally also entails corresponding effort, which must be mapped in the economic -time-frame. Currently, we therefore tend to use integration testing for the expected path through the code as well as for some special cases, catching most of the bugs.
The tests accompany us during development. However, they are also permanently executed by our pipelines as part of the build process. In this way, we ensure that a pipeline also catches us if, for example, we forget to adhere to the convention or if an unforeseen regression occurs.
We also work on a high standard for code design with code reviews and mentoring. Many errors occur in the first place because developers don’t understand exactly what their colleagues (or they themselves) implemented a few weeks ago. With appropriate style tools, source code can be made more readable and malleable. It is important to us to continuously improve ourselves and our work through appropriate refactoring.
Isomorphic Builds thanks to Docker
When dealing with multiple projects, it can quickly become configuration hell if every developer environment, every test instance, and every customer installation is unique. The differences between operating systems alone can cause all sorts of problems, as can different versions of program libraries on the host machine, and other configurations.
Instead, we use Docker as an established technology for virtualization at the operating system level to establish the same boundary conditions on each machine. It allows us to work with a unified environment (containers) whose state we can precisely control.
CI/CD and Configuration Management
Containers are created from container images. We create these images in a CI pipeline as part of a build process and can then add them to a specially operated image registry. We use Gitlab for this purpose. In addition to the source code repositories, it also provides us with the registry for container images and is also functional for executing CI/CD pipelines.
The containers of a MADDOX instance are each created from an image and can be parameterized using environment variables. In addition, customer-specific initialization data can be used in the containers. We use these mechanisms for our purposes when operating the MADDOX instances: individual customer configurations are maintained separately and are available in a dedicated project. As a technology from the configuration management environment, Ansible allows us to centrally control the operation of a MADDOX instance (startup, maintenance, update). It thus brings together images, initialization data, and environment variables, and is our control instrument.
The trade-off at this point is a definitely increased number of moving parts and technologies. To some extent, the complexity of managing operating systems, packages, and environment variables is replaced by the complexity of configuration management and pipelines. Added to this is the expertise required to work with the chosen tools, as well as with containers and images. The advantage is that the complexity does not increase linearly with each additional instance. With each additional customer, we therefore benefit from the path we have chosen–a special solution for a special challenge.