What is Apache Flink:

Apache Flink is a  new project at the Apache Software Foundation that is compatible with the Hadoop ecosystem and runs on top of HDFS and YARN. Flink pushes the technology forward in many ways: The system is built on the principle "write like a programming language, execute like a database", using a unique style of execution engine that aggressively uses in-memory execution, but very gracefully degrades to disk-based execution when memory is not enough, allowing very robust execution behavior. Flink introduces native closed-loop iteration operators, making graph analysis and machine learning application very fast on the platform.Flink programs are not executed directly but are optimized by Flink's cost-based optimizer This means that Flink applications require little (re-)configuration and little maintenance when the cluster characteristics change and the data evolves over time. Finally, Flink's runtime is a true data streaming engine, and ongoing work in the community is unifying batch and true stream processing (rather than mini batches) in a single system. Flink is an active open source project with more than 50 contributors from industry and academia.

Flink is an open-source Big Data system that fuses processing and analysis of both batch and streaming data. The data-processing engine, which offers APIs in Java and Scala as well as specialized APIs for graph processing, is presented as an alternative to Hadoop’s MapReduce component with its own runtime. Yet the system still provides access to Hadoop’s distributed file system and YARN resource manager.

The open-source community around Flink has steadily grown since the project’s inception at the Technical University of Berlin in 2009. Now at version 0.7.0, Flink lists more than 70 contributors and sponsors, including representatives from Hortonworks, Spotify and Data Artisans (a German startup devoted primarily to the development of Flink).

More info: http://flink.apache.org/