(2 min read)
In our 2019 survey, Streaming Data And The Future Tech Stack, over 800 software engineers from around the world shared their experiences. When asked in which use cases real-time stream processing was most used, we see while unflashy app monitoring, log aggregation and ETL top the current use cases, new capabilities in Artificial Intelligence (AI) & Machine Learning (ML) and integration of multiple data streams are starting to rival these leaders.
These results indicate that companies have embraced real-time processing of data as a way to handle machine generated data and to more efficiently manage existing data environments. Other notable findings include:
Application monitoring and log aggregation are the top use cases because it is essential to detect problems quickly instead of waiting for offline analysis. Instead of storing all of the raw data generated by modern applications and systems, the data is often aggregated and stored into time- series databases that only store metrics that can be easily analyzed.
Based on a question only asked of the developers, 45% of developers surveyed have experience working with at least one streaming data application that has been deployed into production or will be within the next six months.
ETL, data warehousing, and recommendation and decision engines use cases are more than twice as likely to be deployed at organizations where developers have hands-on experience incorporating data streams into a production-ready application. ETL and data warehousing are old problems for which streaming is now being applied. Although recommendation engines are used less often, developers are more likely to be involved with these types of applications.
Viktor Klang, Deputy-CTO at Lightbend
"While these results show some interesting trends, the results for these top use cases are to be expected—shifts will tend to occur where they are most needed. This includes modernizing existing systems and workloads for streaming first, then filter down to lesser-priority cases as time progresses and needs change.
I think the areas that we’ll see significantly rise in adoption in the next few years are real-time personalization and IoT pipelines. Real-time personalization makes sense because user demands are even now disrupting markets with expectations of highly relevant content and services delivered instantly. With IoT, as we’ve seen, there is a deluge of millions of new devices hitting the market–enterprises will need an intelligent, safe, and highly visible way to manage these IoT pipelines.
Most of the use-cases listed are great candidates for the Lightbend Pipelines technology, both as a way to migrate workloads, but also to create more opportunities for different components within the system can be connected—for instance tying in ETL with ML, or collecting metrics from log aggregation."
