Connect with us

Hi, what are you looking for?


Quickwit takes on Elasticsearch with an open source search engine for large datasets

paul masurel e1646916222250

Join today’s leading executives online at the Data Summit live now! Watch here.

Let the OSS Enterprise newsletter guide your open source journey! Sign up here.

Search plays a fundamental role in just about every modern application, from Amazon and Netflix, to Slack and Salesforce. On top of that, every application generates swathes of log data, which includes timestamped information about events from inside the software — this might be details on what resources were accessed, the runtime characteristics of an application, and anything relevant to the operation of that system.

Being able to search and make sense of all these computer-generated logs is important, because it helps companies troubleshoot errors and bugs, solve bottlenecks and latency issues, comply with regulations or internal security policies, and better understand what’s going on under the hood. It all constitutes part of what is known as “observability” — that is, the ability to measure the internal state of a software system by analyzing the raw outputs.

Typically, companies pool all their log data in a centralized database system such as ClickHouse. But administering and managing all this comes with many challenges, while the costs associated with storing log data can lead companies to ditch portions of it. This is a problem that Quickwit is setting out to solve, with an open source, cloud-native search and analytics engine built for large datasets.

Founded in 2020, Quickwit touts its ability to run sub-second queries on terabytes of data on object storage services such as Amazon S3 — and it promises to do so at up to “ten times cheaper” than Elasticsearch. But Quickwit isn’t necessarily designed as a direct replacement for every scenario covered by search incumbents such as Elasticsearch — it has more limited, targeted use-cases in mind.

To build on the momentum it has built since its first release last July, the company today announced a $2.6 million seed round of funding co-led by FirstMark and firstminute, with participation from a slew of notable angel backers including MongoDB cofounder Eliot Horowitz; Dataiku CEO and CTO Florian Douetteau and Clément Stenac; and SendGrid founder Isaac Saldana.

Advertisement. Scroll to continue reading.


Paul Masurel: Quickwit cofounder and CEO, and Tantivy creator
Paul Masurel: Quickwit cofounder and CEO, and Tantivy creator

Built on top of the open source, Rust-based search engine library Tantivy, which was created by Quickwit CEO Paul Masurel five years ago, Quickwit is a distributed search engine that serves up additional features on top of Tantivy’s “low-level building blocks” for search. Quickwit delivers a REST API to index data, perform search queries, manage indexes, and manage clusters, with a suite of pre-built connectors spanning data sources such as Apache Kafka, Amazon Kinesis, and Amazon S3.

At its core, Quickwit is aimed at so-called “immutable” datasets (data that is never deleted or updated), which makes it perfect for companies wishing to store and search log data. It also promises sub-second latency of as low as 140 milliseconds, which is just fine for log management searches. However, it’s well understood that milliseconds matter in the online world, which is why Quickwit isn’t targeting use-cases such as ecommerce websites which require lower latency.

“A search query requires at least two round-trips to object storage — however, object storage systems have higher latencies than local disks used by systems such as Elasticsearch,” QuickWit cofounder François Massot told VentureBeat. “The bottom line is, Quickwit can never answer faster than 130-140 ms, which is acceptable for the use-cases that we target, but not for ecommerce ones where higher latencies correlate with losses in sales.”

Quickwit cofounder Francois Massot
Quickwit cofounder Francois Massot

That all said, Quickwit does compete with the likes of Elasticsearch for some purposes, including searching through logs, cloud storage backups, and providing full-text search functionality for online analytical processing (OLAP) databases like ClickHouse. But it’s in these use-cases that Quickwit hopes to differentiate itself enough to win the hearts and minds of small companies and enterprises alike.

For starters, object storage is cheaper to store data than hard disks, while Quickwit is written in Rust which is known to consume less memory than Java, which is what Elasticsearch is based on. Moreover, Quickwit’s decision to separate compute and storage may set it apart in the analytics space — it claims to be the first open source search engine with such an architecture in place.

“Quickwit instances are stateless and can be started or shut down in seconds — you don’t have to move data like in Elasticsearch, as storage is separated from compute,” Quickwit cofounder Adrien Guillo explained.

adrien guillo e1646916341628
Quickwit cofounder Adrien Guillo

In theory, this all translates into faster and cheaper for Quickwit’s target use-cases. And given this cost-efficiency promise, companies may be more inclined to retain more log data which will improve the insights they garner into their system’s performance.

“A lot of companies end up reducing their log retention in order to curb their cost,” Massot added. “Quickwit makes it unnecessary to throw away this valuable data.”

In terms of the types of business Quickwit is gunning for, Guillo argues that it will be suited to companies of all sizes. Smaller firms will want to use Quickwit as their log search observability building block, while bigger enterprises will build entire applications on top of Quickwit, spanning application and log management, search analytics, data lake search and analytics, and more.

“Companies struggle to run existing search systems at scale, and must mobilize considerable resources and capital to do so, especially for ever-growing number of logs generated by applications, systems, and business events,” Guillo said. “Quickwit offers an unparalleled cost-efficiency.”

While Quickwit didn’t give much away in terms of its early customer base, Guillo did confirm that they worked with French unicorn Contentsquare on a proof-of-concept.

For now, Quickwit’s business model is based on a simple dual license approach — an open source AGPL license for free use, and a commercial license which includes support and gives the licensee a “voice in our roadmap,” Guillo said.

Advertisement. Scroll to continue reading.

While the company may consider offering a SaaS in the future, this isn’t currently on Quickwit’s agenda.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn More

Click to comment

Leave a Reply


Online Business Success

About eight years ago, while working in isolated rural communities in India, Rebecca Hui became increasingly aware of the powerful array of indigenous art...

Online Business Success

Steven Sulley – Founder of Woodbury House getty Wall writings and drawings have been around for almost as long as humans have. From cave...


Interested in learning what’s next for the gaming industry? Join gaming executives to discuss emerging parts of the industry this October at GamesBeat Summit...


In a nutshell: Atari is celebrating its 50th anniversary with a collection of more than 90 playable games spanning six platforms and generations. Atari...

Online Business Success

By Anna Anisin, founder at DataScience.Salon, overseeing community and business development. getty Since the first quarter of 2020, there has been a major shift...

Loan And Finance

“We’re thrilled to be able to engage these three very talented students in the work we do as a growing global insurance start-up,” said...


You May Also Like


Introductions get a lot of attention. I’ve explored the topic of how to write them even though as a reader, I always skip them....

Online Business Success

The internet is now our nervous system. We are constantly streaming and buying and watching and liking, our brains locked into the global information...

SEO Guide

There are all kinds of pictures of the world on the internet, but to find one of these specific pictures that you want to...

Online Business Success

You can think of link building in many ways. I like to call it tedious, painful, and a test of patience. It’s also necessary...