Sheldon Anthony IO

A telemetry pipeline product is a powerful software solution that can help simplify and streamline the process of log telemetry. With its comprehensive data ingestion, processing, storage, analysis, and visualization capabilities, it can significantly reduce the complexity and costs associated with managing multiple tools and platforms. Based on Gartner's projection, the adoption of telemetry pipeline products will increase from less than 10% in 2022 to 40% by 2026, indicating its importance in today's data-driven landscape.^[1] It enables faster and easier access to actionable insights, allowing businesses to make informed decisions and stay ahead of the competition.

When log data is not centralized, it can be scattered across different systems and applications, making it difficult to access and analyze. As systems and applications grow, the volume of log data increases exponentially, which can exceed the capabilities of traditional tools and manual processes. Manually managing log data can increase the risk of unauthorized access, data breaches, and non-compliance with regulations. The shift towards cloud-based applications has further exacerbated the data challenge cloud applications are often distributed across multiple environments, making it difficult to collect and analyze logs in a centralized manner. According to Gartner, the cost and complexity of managing this data can be more than $10 million/year in large enterprises.^[2] In today's fast-paced business environment, organizations need to be able to make decisions in real-time as data volumes grow, organizations are increasingly looking for ways to optimize their data management costs.

Over the last five years Employment Hero has made the migration to microservices supported by Kubernetes. This has provided faster time-tomarket, greater resiliency and release flexibility, and autonomy to Engineering teams working on different system components. During this process they faced a challenge familiar to many of today’s businesses—they had to find a way to maintain visibility into their applications as they were broken out into multiple services spread across distributed architectures, while also increasing speed to market. Employment Hero helps over 6,000 businesses and 250,000 users manage people, payroll and productivity. Their suite of fully integrated cloud-based software takes the hard work out of people management so employers can grow their teams and businesses with confidence. By offering robust support for collecting logs from across its distributed, microservices-based environment, while at the same time keeping costs in check, Mezmo helped Employment Hero modernize its log aggregation and analytics strategy alongside its applications so that the business can continue to scale. Employment Hero, which was founded in 2014, originally deployed a software stack composed of monolithic applications. It started refactoring those applications to fit a microservices architecture around 2017. At the same time, it made the move toward a Kubernetes-centric environment for hosting its applications. The company’s engineers quickly discovered, however, that the logging and monitoring solution they had in place at the time was not sufficient for managing microservices applications. The platform did not offer an easy way to collect logs within a distributed environment where each service stores logs in a different place, let alone analyze the logs efficiently. Tracing user requests across the logs of multiple services was difficult, and there was no support for live tailing logs to track the latest activity. As Luong Vo, Platform Engineering Manager at Employment Hero, explained, “We needed the logging system to be scalable, easy to search, accurate in time, and support live tailing,” and the existing logging solution didn’t do that. At first, Luong and his team experimented with using self-hosted Elasticsearch, Logstash and Kibana—the ELK stack—to meet their need for microservices friendly log aggregation and analytics. However, “we found that we were spending too much time scaling Elasticsearch and maintaining the whole stack,” he said, prompting them to search for a more userfriendly alternative. After evaluating other logging platforms, the Employment Hero team settled on Mezmo. Among the biggest factors in their decision was Mezmo’s out-of-the-box support for collecting and analyzing logs from Kubernetes clusters. That solved one of the key pain points engineers faced as they sought to manage logs for the applications that they had refactored into microservices. With the Mezmo platform, they were able to deploy a single tool that unified log management across their entire environment. The fact that Mezmo collects logs using agents was an attractive feature, too, because it made logging easy to orchestrate. With Mezmo’s language-agnostic log aggregation, there was no need to configure application-dependent logging services. “When we started using microservices,” Luong said, “Mezmo provided a solution that we could deploy right away and have it just work, without tinkering with each service in our system.” And, because Mezmo offers fast Live Tail and Search, Luong and his team are able to use the product to gain real-time visibility into all of their services, and to trace live transactions as they flow between services. After making the migration to Mezmo, Employment Hero has enjoyed several concrete positive business impacts. Developers are able to build new features more quickly and with confidence, thanks to their ability to use Mezmo to troubleshoot applications in staging as well as production. The company’s Mean-Time-to-Resolve (MTTR) for support tickets has also remained steady even as its application environment has grown much more complex due to the switch to microservices, another advantage that Luong attributes to Mezmo.^[3] 

The world of data analysis is evolving rapidly, and telemetry pipeline products are poised to play a vital role in this transformation. The exponential growth of data generated by various sources such as cloud, edge, IoT, and mobile necessitates more efficient and scalable solutions for log telemetry. By utilizing a telemetry pipeline product, businesses can harness the power of data to gain a competitive advantage and stay ahead of the curve. Telemetry pipelines provide a unified platform for managing data from all cloud sources, irrespective of their location or complexity. It allows streaming analytics, enabling companies to identify and address issues as they arise, minimizing downtime, and optimizing operational efficiency. The OpenTelemetry standard is driving widespread acceptance, providing a vendor-neutral approach to data collection and instrumentation. This standardization empowers developers to easily integrate telemetry pipelines into their applications, regardless of the underlying technology stack. Telemetry pipelines offer an affordable solution by streamlining data processing and reducing storage requirements. This transformative change brings significant benefits, including improved operational efficiency, real-time insights, and enhanced security and compliance. As a result, telemetry pipelines are set to become an essential tool for modern businesses, driving growth and innovation across industries.

According to Spherical Insights, the global telemetry market is anticipated to exceed USD 512.25 billion by 2032, growing at a CAGR of 8.9% from 2022 to 2032. ^[4] The segment of wireless telemetry systems is expected to register the highest Compound Annual Growth Rate (CAGR) in the global telemetry market during the forecast period. The increased use of portable devices in the defense and commercial sectors for wireless data collection, transport, and storage is expected to fuel the growth of the telemetry market for wireless telemetry systems. Market leaders gain a significant advantage in their respective industries, driving innovation, enhancing customer experiences, and ultimately achieving long-term success. Telemetry pipeline technology has matured significantly in recent years. Today, numerous vendors are offering robust and feature-rich solutions. This increased competition is driving innovation and making telemetry pipelines more accessible.