Event Processing: Three Important Open Problems

comments

By Miyuru Dayarathna, WSO2

1. Introduction

Event Processing (EP) is a paradigm which analyzes streams of events to extract useful insights of real world events. As shown in Figure 1, we can divide EP into two main areas called Event Stream processing and Complex Event Processing (CEP). The first area, stream (i.e., event) processing supports many kinds of continuous analytics such as filter, aggregation, enrichment, classification, joining, etc. The second area, CEP uses patterns over sequences of simple events to detect and report composite events.

Figure 1: Difference between event processing terminologies

Based on a recent survey and using multiple analysis conducted on the trends in the industry we identify three most important open problems in event processing as follows,

• How to develop and debug event processing applications?
• How to tackle the system scalability issue?
• How to manage the system state?

2. How to develop and debug event processing applications?

First, we need to address key issues of how the event processing applications get developed. There are two key sub-issues as follows,

2.1 Can we use a single programming language across different stream processors?

Event processing applications are developed following three main techniques. First, via the use of an API written using a standard programming language such as Java/C++/Python. Some examples for such event processing platforms (all of which are Apache projects) are Flink, Samza, Apex, Storm, Heron, etc. Figure 2 shows a breakdown of the usage of programming languages by event processing software.

Figure 2: Event Processing Software Programming Language usage. (a) aggregated view (b) breakdown of the programming languages used in different event processing systems.

Second, they can be developed using a custom (i.e., domain specific) language. For example, IBM Infosphere Stream’s Stream Processing Language (SPL), Software AG’s Apama, etc. The program statements written in the custom language get translated to intermediary languages such as C/C++/Perl, etc. and then they get compiled into application executables.

Third, event processing applications can be developed using SQL or SQL like languages. Examples include WSO2 Siddhi (WSO2 Stream Processor), Confluent’s KSQL, FlinkSQL, SparkSQL, BeamSQL, etc. The SQL like languages (also known as Streaming SQL) provide familiar syntax for application developers with relational database background. Streaming SQL provides core operations which cover 90% of the use cases.

The diversity of the programming languages has created the need to follow a common syntax so that usability and portability issues prevalent across event stream processors could be addressed. In doing so the third approach described above (Streaming SQL) has significant potential of becoming the standard for specifying event stream processing applications. The ideal scenario would be to create a streaming SQL system which is ANSI SQL compliant. Currently, there are bits and pieces missing such as supporting set operations in Streaming SQL (see).

2.2 Query Writing and Debugging Environment

Most of the stream processors express business logic using programming language code such as Streaming SQL, Scala, Java, etc. Hence it is useful to have right set of tools for developing stream processing applications. To develop large complex applications most of the time non-programmers prefer graphical user interface (GUI) based environments over command line interface (CLI) for application development. Large portion of the stream processor users are non-programmers such as data scientists. Hence not having a proper query writing and debugging environment makes the life of non-programmers a very difficult one. Data analytics notebooks are emerging as a query writing environment for stream processor applications. Almost all the event stream processors have some kind of application debugging support. Yet, the depth of debugging support varies significantly.

3. How can the system scalability issue be solved?

System scalability and performance are important metrics which measure the ability of a stream processor to handle large workloads. A scalable stream processor can expand its operational scale. The traditional scaling approaches are based on adding more resources and they are called vertical scaling (adding more resources to the same server) and horizontal scaling (adding multiple servers). However, some other techniques such as elastic scaling, approximate computing could also be used to address the issues of system scalability.

4. How to manage the system state effectively?

State of an event processor is the set of values being repeatedly accessed during the processing of the events from the incoming data stream. State has three types called Application state, User state, and System state. Application state refers to the values being created and maintained while running the application. This can be for example, state required to detect conditions such as patterns or content of a length window. Application state resides in the ephemeral store (See Figure 2) such as main memory and periodically gets flushed to the permanent store. User state is the user data which is accessed by the application to make runtime decisions. For example, a permanent store such as a RDBMS can contain information of user’s credit history. System state refers to everything else the framework provides to make sure that if the stream processor crashes, it is recoverable back to its normal operations.

Figure 3: A typical data stream processor with state management functionalities

Reliable and fault tolerant state management is important for a stream processor to provide highly available service. Traditionally, reliability and fault tolerance of the stream processors have been achieved by checkpointing their operator states into distributed file system, or to a Relational Database Management System (RDBMS) or to a distributed message queue (such as Apache Kafka). However, checkpointing introduces latency penalty since we need to stop execution of stream application before taking the checkpoint. Several techniques have been introduced to reduce the time taken for saving checkpoints. E.g., Incremental checkpointing keeps a log of changes made to the state and recovers using the log rather than taking full checkpoints of the application state every time. Hence this technique is only beneficial in use cases where the snapshot recovery cost can be subdued by the performance gain made by taking incremental checkpoints. Another approach would be instead of separating out computation layer and database layer into two, they can be merged into one layer and have the other tier as the storage layer for streams and snapshots (see for details).

Conclusion

How an event processing application can be programmed, how it can be scaled, and how to manage its state are three main issues to be solved in event processing paradigm. Finding effective solutions for the above three questions may lead us to the next generation stream processors.

Bio: Miyuru Dayarathna is a Senior Technical Lead at WSO2. He is a computer scientist with multiple research interests and contributions in stream computing, graph data management and mining, cloud computing, performance engineering, IoT, etc.

Related:

• The Executive Guide to Data Science and Machine Learning
• Data Science vs Machine Learning vs Data Analytics vs Business Analytics
• Actionable Insights: Obliterating BI, Data Warehousing as We Know It