I’m thinking of this as a lightweight (single node) alternative in the same way duckdb is focused on data that can be processed by a single node.
I think / hope sqlflow will be a viable/lightweight/cost effective/easy to operate alternative to flink when working with small-medium sized data (on the orders of <10,000 messages / second)
Hello, I’ve been working on a project that embeds duckdb for stream processing.
I just added support for websocket sources which enables sql over the Bluesky firehouse.
https://github.com/turbolytics/sql-flow?tab=readme-ov-file#c...
Duckdb does all the sql execution, and python is responsible for sourcing the data.
The project is still quite young and I’m very much still experimenting, but I’d love any feedback. Thank you.
How do you position this relative to Flink SQL?
I’m thinking of this as a lightweight (single node) alternative in the same way duckdb is focused on data that can be processed by a single node.
I think / hope sqlflow will be a viable/lightweight/cost effective/easy to operate alternative to flink when working with small-medium sized data (on the orders of <10,000 messages / second)