New serverless time series database for IoT and operational applications can scale to process trillions of time series events per day up to 1,000 times faster than relational databases, and at as low as 1/10th the cost
Amazon Web Services, Inc. (AWS), an Amazon.com company, announced the general availability of Amazon Timestream, a new time series database for IoT and operational applications that can scale to process trillions of time series events per day up to 1,000 times faster than relational databases, and at as low as 1/10th the cost.
Amazon Timestream saves customers effort and expense by keeping recent data in-memory and moving historical data to a cost-optimized storage tier based upon user-defined policies, while its query processing gives customers the ability to access and combine recent and historical data transparently across tiers with a single query, without needing to specify explicitly in the query whether the data resides in the in-memory or cost-optimized tier.
Amazon Timestream’s analytics features provide time series-specific functionality to help customers identify trends and patterns in data in near real time. Because Amazon Timestream is serverless, it automatically scales up or down to adjust capacity based on load, without customers needing to manage the underlying infrastructure. There are no upfront costs or commitments required to use Amazon Timestream, and customers pay only for the data they write, store, or query.
Today’s customers want to build IoT, edge, and operational applications that collect, synthesize, and derive insights from enormous amounts of data that change over time (known as time series data). For example, manufacturers might want to track IoT sensor data that measure changes in equipment across a facility, online marketers might want to analyze clickstream data that capture how a user navigates a website over time, and data center operators might want to view data that measure changes in infrastructure performance metrics.
This type of time series data can be generated from multiple sources in extremely high volumes, needs to be cost-effectively collected in near real time, and requires efficient storage that helps customers organize and analyze the data. To do this today, customers can either use existing relational databases or self-managed time series databases. Neither of these options are attractive.
Relational databases have rigid schemas that need to be predefined and are inflexible if new attributes of an application need to be tracked. For example, when new devices come online and start emitting time series data, rigid schemas mean that customers either have to discard the new data or redesign their tables to support the new devices, which can be costly and time-consuming. In addition to rigid schemas, relational databases also require multiple tables and indexes that need to be updated as new data arrives and lead to complex and inefficient queries as the data grows over time.
Additionally, relational databases lack the required time series analytical functions like smoothing, approximation, and interpolation that help customers identify trends and patterns in near real time.
Alternatively, time series database solutions that customers build and manage themselves have limited data processing and storage capacity, making them difficult to scale. Many of the existing time series database solutions fail to support data retention policies, creating storage complexity as data grows over time.
To access the data, customers must build custom query engines and tools, which are difficult to configure and maintain, and can require complicated, multi-year engineering initiatives. Furthermore, these solutions do not integrate with the data collection, visualization, and machine learning tools customers are already using today. The result is that many customers just don’t bother saving or analyzing time series data, missing out on the valuable insights it can provide.
Amazon Timestream addresses these challenges by giving customers a purpose-built, serverless time series database for collecting, storing, and processing time series data. Amazon Timestream automatically detects the attributes of the data, so customers no longer need to predefine a schema. Amazon Timestream simplifies the complex process of data lifecycle management with automated storage tiering that stores recent data in memory and automatically moves historical data to a cost-optimized storage tier based on predefined user policies.
Amazon Timestream also uses a purpose-built adaptive query engine to transparently access and combine recent and historical data across tiers with a single SQL statement, without having to specify which storage tier houses the data. This enables customers to query all of their data using a single query without requiring them to write complicated application logic that looks up where their data is stored, queries each tier independently, and then combines the results into a complete view.
Amazon Timestream provides built-in time series analytics, with functions for smoothing, approximation, and interpolation, so customers don’t have to extract raw data from their databases and then perform their time series analytics with external tools and libraries or write complex stored procedures that not all databases support. Amazon Timestream’s serverless architecture is built with fully decoupled data ingestion and query processing systems, giving customers virtually infinite scale and the ability to grow storage and query processing independently and automatically, without requiring customers to manage the underlying infrastructure.
In addition, Amazon Timestream integrates with popular data collection, visualization, and machine learning tools that customers use today, including services like AWS IoT Core (for IoT data collection), Amazon Kinesis and Amazon MSK (for streaming data), Amazon QuickSight (for serverless Business Intelligence), and Amazon SageMaker (for building, training, and deploying machine learning models quickly), as well as open source, third-party tools like Grafana (for observability dashboards) and Telegraf (for metrics collection).
“What we hear from customers is that they have a lot of insightful data buried in their industrial equipment, website clickstream logs, data center infrastructure, and many other places, but managing time series data at scale is too complex, expensive, and slow,” said Shawn Bice, VP, Databases, AWS. “Solving this problem required us to build something entirely new. Amazon Timestream provides a serverless database service that is purpose-built to manage the scale and complexity of time series data in the cloud, so customers can store more data more easily and cost effectively, giving them the ability to derive additional insights and drive better business decisions from their IoT and operational monitoring applications.”
Autodesk is a global leader in software for architecture, engineering, construction, media and entertainment, and manufacturing industries. “At Autodesk, we make software for people who make things. This includes everything from buildings, bridges, roads, cars, medical devices, and consumer electronics, to the movies and video games that we all know and love,” said Scott Reese, SVP of Manufacturing, Cloud, and Production Products, Autodesk. “We see that Amazon Timestream has the potential to help deliver new workflows by providing a cloud-hosted, scalable time series database. We anticipate that this will improve product performance and reduce waste in manufacturing. The key differentiator that excites us is the promise that this value will come without adding a data management burden for the customers nor Autodesk.”
PubNub’s Realtime Communication Platform processes trillions of messages per month on behalf of thousands of customers and millions of end users. “To effectively operate the PubNub platform it is essential to monitor the enormous number of high-cardinality metrics that this traffic generates. As our traffic volumes and the number of tracked metrics have grown over time the challenges of scaling our self-managed monitoring solution have grown as well, and it is prohibitively expensive for us to use a SaaS monitoring solution for this data. Amazon Timestream has helped address both of these needs perfectly,” said Dan Genzale, Director of Operations, PubNub. “We’ve been working with AWS as a Timestream preview customer, providing feedback throughout the preview process. AWS has built an amazing product in Timestream, in part by incorporating PubNub’s feedback. We truly appreciate the fully-managed and autoscaling aspects that we have come to expect of AWS services, and we’re delighted that we can use our existing visualization tools with Amazon Timestream.”
Since 1998, Rackspace Technology has delivered enterprise-class hosting, professional services, and managed public cloud for businesses of all sizes and kinds around the world. “At Rackspace, we believe Amazon Timestream fills a longstanding need for a fully managed service to capture time series data in a cloud native way. In our work with Amazon Timestream we’ve observed the platform to be performant and easy to use, with a developer experience that is familiar and consistent with other AWS services,” said Eric Miller, Senior Director of Technical Strategy, Rackspace Technology. “Cloud Native and IoT are both core competencies for us, so we’re very pleased to see that Amazon Timestream is 100% serverless, and that it has tight integration with AWS IoT Core rule actions to easily ingest data without any custom code. Organizations who have a use case to capture and process time series data should consider using AWS Timestream as a scalable and reliable solution.”
Cake is a performance marketing software company that stores and analyzes billions of clickstream events. “Previously we used a DIY time series solution that was cumbersome to manage and was starting to tip over at scale,” said Tyler Agee, Principal Architect, Cake Software. “When we heard AWS was building a time series database service—Amazon Timestream—we signed up for the preview and started testing our workloads. We’ve worked very closely with the AWS service team, giving them feedback and data on our use case to help ensure Amazon Timestream really excels in production for the size and scale of time series data we’re dealing with. The result is phenomenal—a highly scalable and fully serverless database. It’s the first time we’ve had a single solution for our time series data. We’re looking forward to continuing our close work with AWS and cannot wait to see what’s in store for Amazon Timestream.”
Trimble Inc., is a leading technology provider of productivity solutions for the construction, resources, geospatial, and transportation industries. “Whenever possible, we leverage AWS’s managed service offerings. We are excited to now use Amazon Timestream as a serverless time series database supporting our IoT monitoring solution,” said David Kohler, Engineering Director, Trimble. “Timestream is purpose-built for our IoT-generated time series data, and will allow us to reduce management overhead, improve performance, and reduce costs of our existing monitoring system.”
With over 60 years of fashion retailing experience, River Island is one of the most well known and loved brands with over 350 stores across Europe, Asia, and the Middle East, and six dedicated online sites operating in four currencies. “The Cloud Engineering team have been excited about the release of Amazon Timestream for some time. We’ve struggled to find a time series data store that is simple, easy, and affordable,” said Tonino Greco, Head of Cloud and Infrastructure, River Island. “With Amazon Timestream we get that and more. Amazon Timestream will enable us to build a central monitoring capability across all of our heritage systems, as well as our AWS hosted microservices. Interesting times!”
