StreamSpan - Advancing Stream Data Systems with Spanning Events
An Indo-French Collaborative Research Project Funded by Indo-French Centre for the Promotion of Advanced Research (CEFIPRA).
About StreamSpan
StreamSpan is a cutting-edge research initiative aimed at revolutionizing stream data processing systems by enabling seamless handling of spanning events. This three-year project, a collaboration between International Institute of Information Technology Bangalore, India, and the University of Nantes, France, is supported under the Collaborative Scientific Research Program (CSRP) of CEFIPRA — a bilateral organization fostering research collaborations between India and France.
The Challenge
In today’s data-driven world, efficient and scalable processing of time-based data is crucial. Events can either be point-time events (single timestamp) or spanning events (time intervals). While existing data processing systems are optimized for point-time events, spanning events, such as a phone call or a video stream with start and end times, present unique challenges. These challenges arise because conventional systems lack native support for handling time intervals, requiring developers to build complex pipelines to fill this gap.
Our Vision
StreamSpan envisions a generic, distributed, and asynchronous architecture for big data processing systems that treats point-time events as a subset of the broader class of spanning events. By addressing the limitations of existing systems, we aim to transform how data streams are processed across diverse domains like:
- Telecommunications: Monitoring phone calls and ensuring Quality of Service (QoS).
- Industrial IoT: Tracking task durations in smart machinery and digital twins.
- Streaming Platforms: Capturing listening or watch times for enhanced user insights.
Why StreamSpan Matters
Back in 2020, our team developed AIR, one of the fastest data processing engines for handling point-time events with ultra-low latency. Building on this success, StreamSpan takes the next leap by focusing on spanning events, which require efficient handling of time intervals and complex event semantics.
Key features of StreamSpan include
- Unified Processing Model: Seamlessly handles both point-time and spanning events.
- Open-Source Framework: Provides robust, off-the-shelf tools for developers.
- Distributed Asynchronous Design: Ensures scalability for real-time industrial use cases.
- Multi-Query Optimization: Enhances efficiency by optimizing query execution across diverse scenarios.
Our Approach
StreamSpan will introduce
- A Novel Architecture: For generic, asynchronous, and distributed stream processing systems.
- Benchmark Schemes: To evaluate and compare spanning event processing capabilities.
- Baseline Implementation: Built on modern streaming systems, validated through real-world applications.
The project also explores multi-query optimization by addressing window query types, time slicing schemes, input stream profiles, and message ordering challenges.
Domains of Impact
- Big Data Processing
- Streaming Data Systems
- Complex Event Processing
Collaboration & Funding
StreamSpan is a proud collaboration between Indian researchers and the University of Nantes, France, supported by CEFIPRA's Collaborative Scientific Research Program (CSRP). This program exemplifies the spirit of Indo-French scientific partnership, driving innovation to address global challenges.