Distributed in-memory JVM cache

Abstract

Big Data analysis and processing is a very interesting field with plenty of ongoing research. One of the major constraints in this field is that a single machine's processing power is not growing in proportion with the size of the ever growing dataset sizes. In order to handle this problem, one workaround that has been in place for a while now are distributed systems such as Hadoop for analysis. However most of the existing solutions are largely disk based, with low latency solutions being even fewer. With a distributed in-memory JVM cache, the aim to achieve multi column query results quickly is fulfilled. A performance study of data analysis in java heap vs. java off heap on row store and column store was conducted, which lead to the conclusion that column store off heap java cache will deliver the best performance. Existing hadoop design is referred for communication between nodes and handling failure cases; and some improvements are done in the existing design.