I am thrilled to announce that Timeplus Enterprise v2.5 is now Generally Available! This milestone marks a significant leap forward for our Timeplus Enterprise v2 which was released earlier this year. In this release, we pushed our unparalleled performance to a new level, natively integrated with Redpanda Connect and Apache Pulsar to access a rich ecosystem of enterprise and AI applications.
Powered by our lightweight, powerful, and efficient single-binary platform, Timeplus Enterprise offers flexible deployment options. Whether fully managed in the public cloud, self-hosted in your data center, running on your laptop, or even on edge devices, our platform adapts to diverse user needs and unblocks massive use cases, from handling large-scale workloads to operating in resource-contained edge environments, capturing real-time insights where data resides. Most importantly, Timeplus Enterprise drives down operational efforts and costs down with simplicity.
Our software has been rigorously tested and proven in a variety of industries and mission-critical use cases, including cybersecurity, algorithm-based trading, and real-time streaming pipeline and analytics.
For example, a Fortune 1000 company has successfully deployed Timeplus Enterprise v2 in production since the beginning of 2024, addressing their critical trading use cases that demand efficient, reliable and scalable streaming analytics. The use case runs 300+ continuous queries on raw incoming data of 100,000 messages per second with peaks lasting 5 hours.
Today, we are excited to upgrade our offerings with the release of Timeplus Enterprise v2.5. This unified, enterprise-grade product is available through our fully managed cloud service or can be installed on various infrastructures, including bare metal, Docker, and Kubernetes.
Key Breakthroughs in Timeplus Enterprise v2.5
The v2.5 release of Timeplus Enterprise introduces several groundbreaking features:
Materialized Views Auto-Rebalancing
Performance Improvements
Enterprise-Grade Real-Time Data Integration with 200+ Connectors from Redpanda Connect
Pulsar External Stream to query or process data in Pulsar with SQL
Materialized Views Auto-Rebalancing
Materialized views are key building blocks in Timeplus Enterprise, which continuously execute the streaming SQL queries in the background and “materialize” the results physically to the internal storage of the materialized view. These materialized views can be queried as tables via other SQL queries or serve as data sources for other materialized views.
Alternatively, you can configure a target stream for the materialized view. The target can be an append-only stream in Timeplus, or a Mutable Stream for UPSERT and fast OLAP queries, or an External Stream for writing data to Apache Kafka, Apache Pulsar, or an External Table to write data to ClickHouse. By utilizing this feature, materialized views act as intermediaries between upstream sources and downstream sources.
To learn more about the difference between Timeplus materialized views and ClickHouse materialized views, please refer to https://www.timeplus.com/timeplus-and-clickhouse.
It's common practice to define numerous materialized views in Timeplus for various computation and analysis. Some materialized views can be resource-intensive, requiring significant memory or CPU resources. In a cluster of Timeplus Enterprise with multiple nodes, each materialized view is executed on a single node, as all data is replicated across the cluster. Ideally, the workload distribution among nodes should be balanced, based on the CPU or memory consumption rather than the number of materialized views.
Manual Load Balancing
Since the release of Timeplus Enterprise v2.3, it has been possible to configure a materialized view with the optional memory_weight setting. This setting allows for memory-intensive materialized views to be prioritized on specific nodes within a cluster. For example:
CREATE MATERIALIZED VIEW my_mv
SETTINGS memory_weight = 10
AS SELECT ..When a Timeplus cluster is initiated, materialized views are automatically scheduled to different nodes based on the user-defined memory_weight. This ensures that each node in the cluster experiences similar memory pressures, regardless of the number of materialized views running on it.
This mechanism effectively addresses customer requirements. However, setting the appropriate memory_weight can be challenging due to the complexity of materialized view queries, which may contain multiple subqueries and various SQL functions. Accurately estimating memory usage for each materialized view can be difficult. Consequently, most customers categorize their materialized views into light queries and heavy queries. They then assign the same memory_weight to materialized views within the same category.
Auto Load Balancing
Now in the Timeplus Enterprise v2.5, we have introduced auto-load-balancing for materialized views. This feature eliminates the guess work and ensures optimal resource utilization.
Within a cluster, Timeplus Enterprise monitors the memory consumption for each materialized view. Every 30 seconds (configurable via workload_rebalance_check_interval), the system checks whether there are any nodes with memory over 50% full. If such a node exists, Timeplus further checks if any materialized view on that node consumes 10% or more of its memory (configurable). When these conditions are met, Timeplus reschedules the materialized views to less busy nodes. During the rescheduling process, the materialized view on the previous node is paused, and its checkpoint is transferred to the target node. Subsequently, the materialized view on the target node resumes streaming SQL based on the checkpoint.
By default, auto-load balancing is enabled in Timeplus Enterprise, which significantly reduces maintenance efforts and enhances overall system efficiency. For further details, please refer to the documentation.
Performance Improvements
In this release, we pushed our unparalleled performance to a new level.
For the Ingest REST API, the new version of Timeplus Enterprise is up to 2x faster than the previous version.
Ingestion Throughput in MB/s: The Higher, the Better
Another example is the EMIT ON UPDATE query processing, which generates stream processing results whenever the aggregation values are modified. In Timeplus Enterprise v2.5, we introduced a novel data structure and algorithm to attain a 0.1 millisecond end-to-end latency, even with 100 million unique keys. Previously, this process could take 8 milliseconds (for 1 million keys), 95 milliseconds (for 10 million keys), or 0.9 second (for 100 million keys).
E2E Latency in μs: The Shorter, the Better
Enterprise-Grade Real-Time Data Integration with 200+ Connectors from Redpanda
Timeplus Enterprise now embeds Redpanda Connect with over 200+ connectors into its platform, dramatically expanding our customers' ability to integrate streaming data across their entire technology stack and extract valuable insight while the data is fresh.
The partnership of Timeplus and Redpanda addresses the challenges of real-time data integration by:
Streamlining the development process with Redpanda Connect, a battle tested standardized connectivity framework while letting Timeplus Connector manage the lifecycle, recovery, and checkpointing of Redpanda Connect pipelines progress along with Timeplus streams
Providing instant access to 200+ pre-built connectors for numerous data sources and destinations (e.g. Websocket, PostgreSQL, NATS, Amazon S3, Amazon SQS, Azure Blob Storage)
Exposing flexible connector configuration APIs (via the versatile Bloblang DSL) allowing for customization of things like headers, authentication, and lightweight transforms
Enabling rapid transformation (stateless or stateful) of data via Timeplus for consumption by popular enterprise systems, databases, cloud services, and alerting systems
Timeplus also contributed back Timeplus connectors to the open source ecosystem for Redpanda Connect standalone users to integrate with Timeplus Enterprise or the open source Timeplus Proton.
Learn more about this in our blog: https://www.timeplus.com/post/timeplus-and-redpanda-connect
Reading or writing data in Apache Pulsar or StreamNative via External Stream
Congratulations to the Apache Pulsar community for the 4.0 LTS(Long-Term-Support) release recently. We’re thrilled to announce that in Timeplus Enterprise v2.5, a new type of External Stream is implemented in our C++ core engine. With a single binary, developers can read millions of events in Pulsar each second with streaming SQL, or write data to Pulsar topics in JSON, CSV, Protobuf or Avro format.
Learn more about this in our blog: https://www.timeplus.com/post/introducing-pulsar-external-stream
Grafana Plugin for Timeplus is Production Ready
Grafana has long been one of the most popular tools for real-time monitoring and data visualization, helping organizations track metrics and analyze trends through a single, user-friendly interface. In October 2023, we launched the first version of the Timeplus Grafana Plugin — one of the pioneering streaming data sources in the Grafana ecosystem. This plugin was designed to leverage Grafana’s new Grafana Live capability, allowing users to keep their SQL query results up-to-date without the need to refresh their dashboards. We received highly positive feedback, with users highlighting the plugin’s ease of integration and the unique value of real-time streaming insights within Grafana.
Today, together with the release of Timeplus Enterprise v2.5, we’re thrilled to announce the release of Timeplus Grafana Plugin v2.0, which is production ready. It’s available now on our GitHub repository. This latest version introduces significant improvements in both performance and stability. Whether you’re using Timeplus Enterprise or the open-source Timeplus Proton, this plugin allows you to seamlessly monitor real-time data through Grafana, utilizing both simple and complex analytics to track live data trends.
Learn More
To explore the capabilities of Timeplus Enterprise v2.5, please check out https://docs.timeplus.com/enterprise-v2.5 for installation guide as well as change logs.
Thank you for being a part of our journey. We look forward to your continued support and collaboration as we move towards a future powered by real-time streaming analytics.
Ready to try Timeplus Enterprise? Try for free for 30-days.
Join our Timeplus Community! Connect with other users or get support in our Slack community.