Dataflow model and its applications in big data processing - Nifei BI; Guangyao DING; Qihang CHEN; Chen XU; Aoying ZHOU (大数据)
Unbounded,unordered and large scale datasets are increasingly common in recent years.Meanwhile,the processing requirements from data consumers are becoming more and more sophisticated,such as event time,window and latency.In order to deal with the evolved processing requirements on these unbounded,unordered and large scale datasets,the dataflow model in big data processing was introduced.On one hand,the dataflow graph of the dataflow model in big data processing was analyzed from the level of execution engine.On other hand,the dataflow programming model of the dataflow model in big data processing was analyzed from the level of unified programming.Furthermore,the different implementations of dataflow graph and dataflow programming model in multiple execution engines were analyzed,including Spark,a batch processing engine,and Flink,a stream processing engine.