Databases, Data Streams, and Quantum Mechanics

What does algorithmic trading, medical systems monitoring, web searches and clicks, and fraud detection have in common?  Part of the answer is streams of data.  Large amounts of data, sources continuously streaming it.

Traditional databases (DBMS) are oriented towards processing and storing transactions, representing discrete events.  The view one has is typically a table.  If one extends the transactions over sufficient time, a data warehouse approach might be used, with an eye towards data mining and data analytics, mostly reading (and seldom updating) the stored data for OLAP (online analytic processing) with its classic cube.  In both databases and data warehouses, tools (e.g., SQL and OLAP tools) are in the hands of skilled business analysts and data analysts, not just for programmers.

Classical physics in its approach to light treated it as waves, with different colors having different wave lengths.  Wave behavior allows for interference patterns, and coherence of light, with coherence epitomized by lasers.

Along came Einstein and his contemporaries with fresh insights, revealing that light can also exhibit particle behavior, discrete entities rather than the continuity of waves.  This and far more is explained by quantum physics.

Faster networks in recent years, and faster, larger storage of data, have brought us streaming data.  A stream of data; a continuous wave of it.  New paradigms are needed to fully cope with this, to facilitate new insights.  The tables of relational DBMS and cubes of OLAP only give us a partial view.  Perhaps other geometries or coordinate systems (polar or spherical instead of Cartesian) could provide other insights.

To maximize the effectiveness and value for business, the  new tools for these new insights need to be accessible to the same skilled people, business analysts and data analysts, not just to programmers.

New solutions are evolving, glimmerings of new paradigms; for example, IBM’s BigInsights.  For the curious, look at major vendors, with topics like “data streaming”, and “complex event processing”.