Customer success story 1. FSV Mainz05 & PRTG

About 1. FSV Mainz05 e.V.

Founded in 1905, 1. FSV Mainz05 e.V. currently has 1,100 employees and more than 20,000 members. The professional soccer team has been a fixture in Germany’s top league – the “Bundesliga” – for the past 20 years.

Since 2011, the club has been playing its home games in the MEWA ARENA, a modern soccer stadium that is also used for major events. Even a “normal” company of this size needs an extensive IT team. A modern professional soccer club has special needs that require even more technology, personnel, and expertise.

Video: See how PRTG makes Mainz05 match days run smoothly

Karsten Lippert, Head of ICT & Digitalization at Mainz05, is in charge of the club’s stadium technology and IT. His title alone is a sign that digitalization has become a part of everyday life in the world of professional soccer. In short, a game would not even be possible today without properly functioning IT. IT controls and monitors just about everything at the stadium, from advance ticket sales to admission at the gates, video analysis systems for halftime talks in locker rooms, cash registers at bratwurst stands, floodlights, Wi-Fi, the club’s radio, and much more.

Lippert and his team are of course also responsible for the traditional IT environment, i.e., the network and the infrastructure with its devices, applications, VMware, and storage systems.

“From the outset, our objective was to become proficient in PRTG so we could maintain and develop our monitoring environment autonomously. Our service provider trained our employees on the ins-and-outs of our solution during the implementation of the tool. In no time at all, we were able to develop our own dashboards and extensions for PRTG. This wouldn't have been possible without the outstanding usability of the monitoring tool.”

Karsten Lippert, Head of ICT and Digitalization, 1. FSV Mainz05 e.V.

The requirement: A comprehensive monitoring system

In Germany, professional soccer clubs work with the German Football League (DFL), which imposes extremely strict regulations on the operations at stadiums. The first requirements take effect four days before the day of a game. In total, the DFL requires Mainz05 to successfully complete around 100 checks. These include the ISMA network for providing video analysts with camera feeds, the turnstiles for controlling access to the stadium, and the telephones used on the day of the game.

After a game, the systems required for the game are switched off and the stadium runs in “normal” operation mode. To meet these specific requirements, the IT team set up two separate systems: the game operations system and the “normal” IT. The traditional IT systems must also be available at all times, especially on the day of – and the days leading up to – a game. Without reliable and comprehensive monitoring, the IT team at Mainz05 would be putting the club’s entire operations at risk.

Originally, the team used three different systems: HPE’s Intelligent Management Center, Woodstone Servers Alive, and ONE. The first two are classic IT monitoring and management systems, while Kentix ONE is an IoT platform. Kentix is a manufacturer of environmental sensors and door locking (and other security) systems. “ONE” is the main software component of its devices.

The IT team at Mainz05 found it extremely difficult to integrate these three systems with each other. What’s more, the systems were unable to keep the team in the know and quickly inform them in the event of a disruption. The club therefore needed a system that could incorporate the IT network and infrastructure – as well as the entire stadium technology – to immediately identify and resolve issues.

Planning and design

Karsten Lippert had already worked with Paessler PRTG Network Monitor during a previous job, and was confident the tool would meet all the club’s needs. As mentioned above, Mainz05 uses systems from the manufacturer Kentix. Paessler and Kentix have been partners for many years, and Kentix sensors and systems can be easily integrated into PRTG.

In addition, the club was impressed that Paessler – a company from Germany – was the creator of the tool. In 2020, a regional service provider helped the Mainz05 IT team implement PRTG. During the implementation, the goal was to make the soccer club’s employees familiar with the tool, such that the use of the centralized monitoring system could be expanded in future.

Karsten Lippert explains: “From the outset, our objective was to become proficient in PRTG so we could maintain and develop our monitoring environment autonomously. Our service provider trained our employees on the ins-and-outs of our solution during the implementation of the tool. In no time at all, we were able to develop our own dashboards and extensions for PRTG. This wouldn't have been possible without the outstanding usability of the monitoring tool.”

The first step was to take stock and plan the structure of the monitoring solution. Extremely quickly, three main focus areas emerged:

1. Monitoring the traditional IT environment

PRTG comes with numerous ready-made “sensors” and predefined device templates for use with most devices and systems from the best-known manufacturers. These sensors make the monitoring of the IT environment both quick and easy. The default PRTG dashboard displays all sensors in a hierarchical tree. For larger configurations (in particular), this structure must be developed and maintained from the beginning.

Although PRTG offers the flexibility to make necessary adjustments, an efficient system makes day-to-day monitoring with the tool that much easier. The Mainz05 monitoring system was jointly developed by the club’s service provider and Lippert’s IT team, and employees were simultaneously trained in the use of PRTG.

2. Monitoring the stadium technology

Ticket sales, admission gates, arena services, floodlight control, radio, contactless payments, video analysis systems – a modern soccer stadium is much more than just a field, two goalposts, and bleachers. In Germany, clubs must also meet the strict requirements of the DFL in terms of broadcasting technology and security.

All of this surpasses the scope of most monitoring tools, which only cover IT standards. But thanks to an API and numerous custom sensors, PRTG makes it easy to integrate a wide variety of systems using PowerShell or protocols.

3. Visualzation

A turnstile at the entrance is jammed, an access point for the 05 app is down, a payment system at a bratwurst stand is acting up... The Mainz05 monitoring system quickly detects the cause of the problem, thereby allowing a service technician to immediately get down to troubleshooting.

PRTG is designed with “maps” (HTML-based dashboards created via drag & drop in an editor) for this purpose. Example: a stadium map can be integrated as a background image, upon which individual devices are positioned in their exact locations. If a problem requires an on-site intervention, the technician can immediately see where he needs to go.

Karsten Lippert explains the importance of comprehensive monitoring for his role as Head of ICT and Digitalization: “With digitalization, IT is increasingly becoming the basis for components and departments that were previously analog. For us as an IT team, this means more and more responsibility. To best fulfill our role, we must always be informed of the status and performance of all IT and stadium technology, and be promptly notified in the event of disruptions – not only about the type, but also the cause and location. Such notifications are only possible if digitalized environments and IT are monitored with a single monitoring system, and if this system is able to maintain a comprehensive overview. This is exactly what we get with PRTG.”

IT monitoring: The basis

A traditional IT environment composed of servers, databases, storage systems, firewalls, switches, backup systems, Wi-Fi access points, cloud services, and many other IT components – all of which are integrated into the centralized PRTG monitoring environment – serves as the basis for Mainz05’s stadium technology and everyday operational procedures.

Many of these components have been integrated into PRTG automatically via SNMP. For others, such as Fortigate firewalls or the Veeam backup system, Karsten Lippert and his team use PowerShell scripts to query data via the device application programming interface (API).

Mainz05 uses Büro+ from Microtech as its ERP solution. Büro+ generates an XML file that is read via a PowerShell script and integrated into PRTG. With this tool, the team can keep a constant eye on the number of users and the cash flow in the inventory management system.

But monitoring with PRTG doesn’t stop there: servers, services, the internal databases of the ERP system, the use of licenses... If a service fails, PRTG restarts it automatically. As a result, many IT problems are resolved immediately – without Lippert and his team having to intervene.

The Mainz05 website is one of the club’s most important communication tools for fans and the press. Lippert and his team use PRTG remote probes in various subnets to monitor load times from different perspectives and therefore optimize performance for all users.

PRTG remote probes are polling engines that use queries (sensors) to collect monitoring data via common protocols, before sending this data to the main PRTG instance (with TSL encryption) for evaluation. With remote probes, multiple locations can be monitored by a single instance of PRTG. Among other things, they can also be used (as in the current example) to monitor website load times from different perspectives.

“Of course, a monitoring tool with a high degree of automation is extremely beneficial. On the other hand, this can also be extremely limiting as the creative solutions (which are always needed in practice) are often difficult to implement. PRTG offers the perfect compromise here: the software comes with numerous predefined sensors and features that function out of the box, which makes our day-to-day work much easier. In addition, its custom sensor templates and API allow us to easily integrate our own scripts to meet our own individual needs.”

Karsten Lippert, Head of ICT and Digitalization, 1. FSV Mainz05 e.V.

How digitalization benefits fans

Without highly complex and networked stadium technology, games would not even be possible today. The MEWA ARENA in Mainz is a state-of-the-art stadium that is entirely up to speed. At the stadium, a comprehensive monitoring system ensures everything runs smoothly. This centralized system is able to incorporate all the stadium technology and IT (systems, components, etc.) – one of the decisive features that led to Mainz05 choosing PRTG. But how does it benefit fans of the soccer team?

Got your ticket yet?

You have to buy a ticket to attend a game. Today, tickets are mainly purchased online via the Mainz05 website. Naturally, the Mainz IT team monitors the online ticket shop with PRTG, as well as the backend connected to the ERP system.

How about the new jersey?

Fans can show off their team pride by purchasing the new Mainz05 jersey in the online store. PRTG monitors the store pages, and ensures the purchase process runs smoothly.

Can I try the jersey on first?

Fans who would rather try on the jersey first can do so in one of the Mainz05 fan stores. The IT team uses PRTG to make sure the checkout systems and the connection to the ERP system are always online and available.

Arriving at the stadium

PRTG monitors the admission gate systems (including the turnstiles) to ensure all fans make it into the stadium before kickoff. The Mainz05 IT team uses PRTG to monitor the floodlight control system.

Where’s my seat and the food?

PRTG also monitors the stadium’s sign system. This not only provides for extra convenience, but also security. IN addition, PRTG helps ensure that cash register systems and printers at stands are working properly.

How can I pay?

At the MEWA ARENA, by card. Or with your cell phone. Real fans, however, pay with their fan ticket. PRTG monitors the cash register systems, printers, network, power supplies, and much more to ensure all this runs smoothly.

Let's hope they play better later on!

PRTG is also used to monitor the video analysis systems, which allow the coaches to analyze the game in progress and show their players how to turn the game around in the second half.

What does the radio have to say?

Tim Klotz, the voice of 05ER.fm (the Mainz05 online radio station), provides the live report as well as news from all Bundesliga games. The IT team makes sure the Wi-Fi is working in the MEWA ARENA, and monitors the equipment in the 05ER TV and radio studio.

Everything at a glance

One of the original goals of the IT team was to find a monitoring tool that would provide them with a centralized overview of all stadium technology and IT. Lippert wanted to snuff out disruptions before they led to serious problems, something which can only be done by quickly and conveniently identifying the root causes.

Right from the start, the team therefore learned how to create PRTG maps. These HTML dashboards are compiled via drag & drop with an easy-to-use editor. A graphic (such as the stadium floor plan) can be integrated as a background image.

The monitored components are then positioned over the graphic. Example: if a turnstile at the entrance is stuck, PRTG promptly recognizes the problem and marks the turnstile on the map.

As a result, the technician in charge immediately sees where he needs to go to fix the problem.

The Mainz05 IT team proceeded to set up maps for all important IT and stadium technology systems. The club’s media department (for example) can view the status of the radio broadcast and studio equipment on its own dashboard, as well as the current number of listeners.

One map clearly displays the access to the stadium (i.e., the admission system), the security cameras, and the sign system, while another map with cash register systems, telephones, and printers provides relevant information for the catering team.

Lippert explains: “If a turnstile gets stuck, a cryptic IT message with a few IP addresses won’t help our technicians resolve the issue. With PRTG, our team receives a push message in plain text on their mobile device, allowing them to locate the cause directly in the PRTG app via the corresponding map and fix the problem as quickly as possible.”

Before and after the game

Monitoring the games of a Bundesliga club is not always easy. Many systems are switched off between game days, only to be gradually brought back online in the days leading up to the next home game. Creative measures are required to incorporate these factors into an IT monitoring solution. After all, if a system is switched off, an error message will automatically be generated in the monitoring system.

Although error messages can be avoided by having sensors paused automatically in PRTG, the Bundesliga game schedule is unfortunately too random for such automations to be effective. Mainz05 plays a game every week in the Bundesliga, but usually only has one home game every two weeks. Normally – after all, there are also games during the week and breaks for cup games and international competitions – Bundesliga games take place on Fridays, Saturdays, and Sundays.

As a consequence, the Mainz IT team wrote a PowerShell script that incorporates the official DFL game schedule into PRTG. When the systems start back up, they are therefore automatically included in the monitoring.

Conclusion

For Mainz05, PRTG by Paessler has become an essential part of all the club’s operations. Karsten Lippert and his ICT & digitalization team use the monitoring solution to keep a constant eye on their entire IT environment, including the stadium technology, fan radio, and security systems.

Disruptions are detected immediately, with the causes clearly visible on the software’s maps. With PRTG, errors are identified quickly, allowing for targeted troubleshooting measures to be implemented before more serious problems occur. Mainz05 can only meet the strict requirements of the DFL with such a monitoring tool.

And PRTG has already helped in other ways as well: “Continuous monitoring with PRTG allowed us to meet all the data protection management requirements of ISO 27701,” declares Lippert.

Paessler, the monitoring experts, are now certified with ISO 9001, the globally recognized standard for quality management

“At Paessler, it has always been a key objective to provide customers with the best possible products and services. We are proud that our efforts are recognized with the ISO 9001 certification”, says Evren Erisen, Team Lead Quality at Paessler, who led the preparation process.

View press release