Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
Even when relying on cutting-edge tools from data warehouse providers such as Snowflake and Databricks, enterprises may still find themselves struggling to deal with certain mission-critical workloads.
But San Francisco-based startup e6data claims to have a solution.
The startup, which has just raised $10 million from Accel and others, has developed a “reimagined” Kubernetes-native compute engine that can slot into any mainstream data intelligence platform, allowing customers to handle compute-intensive workloads with 5x better performance and half the total-cost-of-ownership (TCO) as compared to other mainstream compute engines.
The offering is still new compared to mainstream vendor-backed and open-source compute engines including Spark Trino/Presto (including Starburst), but major industry players, including Freshworks, are already beginning to adopt it for potential price-performance benefits.
How exactly does e6data solve performance bottlenecks?
Today, nearly every modern data platform — from Snowflake and Databricks to Google BigQuery and Amazon Redshift — has a compute engine at its heart to handle data workloads.
It essentially acts as a workhorse that processes large volumes of data in response to queries, executing operations like data transformation, analysis and modeling.
While most engines are pretty good at handling traditional workloads like analytical dashboarding and reporting, things begin to get complicated with next-gen use cases like real-time analytics (such as fraud detection or personalization) and generative AI.
These workloads revolve around high query volumes, large-scale data processing or queries on near real-time data, which demands faster computing from the central engine and increases the associated costs.
“These workloads are non-discretionary and growing very, very fast for our customers… It’s not uncommon for the spending on these heavy workloads to be increasing 100-200% per annum…The larger and more mature the enterprise is, the more this pain is being felt today. But this pain is coming for every enterprise data leader,” Vishnu Vasanth, founder and CEO at e6data, tells VentureBeat.
The main reason behind these performance bottlenecks, Vasanth says, is the architecture behind most commercial and open source compute engines.
Being 10-12 years old, most engines are dominated by a central coordinator or driver system responsible for several critical activities across a query’s or job’s lifecycle. The approach works, but when faced with high load, concurrency, or complexity of heavy workloads, these centralized, monolithic components become a source of resource inefficie …