Research on Best Database Solution
- Arushi Singhal
The database which we are planning to select is a NoSQL database with Key-value data stores. Key-value data stores have the simplest data model – arbitrary byte strings for keys, paired with a byte string for the corresponding value, they can be quickly incorporated into applications that have very simplistic storage needs. Their APIs tend to be equally simple, providing only very primitive operations.
We are choosing our database with low-latency with optimal read, write and query time. Along with it, the DB size required to store values should also be optimal.
The database should be an embedded NoSQL (Key/Value store) database engine and should not have a separate server process. The database should be able to read and write directly to ordinary disk files. An in-memory database is a database management system that primarily relies on main memory for computer data storage. But as we want to handle big data and hence our database should hold big volumes of data larger than RAM can hold so we want embedded database with on-disk storage mode along with in-memory capabilities to improve its performance speed.
The Database should be open-sourced and able to support APIs for multiple programming languages as it can be extended in future but should definitely support C++ APIs as currently, we are using C++ APIs.
Popular Key-Value store NoSQL Databases comparison. Also, one of the good comparisons can be found here.
| Databases | Open Source | C++ APIs | Key-Value Storage | Embedded database with on-disk storage mode | In-Memory (Caching) Capabilities | Note |
|---|---|---|---|---|---|---|
| Aerospike |  | | |  | | |
| Berkeley DB (one of the best) | | | | | | Developed by Oracle Inc. Support APIs for multiple programming languages. Support many Operating systems as the server. One of the oldest and developed in the year 1994. |
| Redis | | | | only In-memory capabilities. Can only use it as cache store so cannot store data larger than RAM size. | | One of the best but cannot store big volumes of data larger than RAM hence cannot be used for our purpose. |
| LevelDB | | | | | | Developed by Google Inc. IoS is not supported as a server operating system. Developed in 2011. |
| RocksDB (also very much recommended) | | | | | | Developed by Facebook Inc. Developed in 2013. Only supports APIs in language C++ and Java. RocksDB supports Transactions when using a TransactionDB or OptimisticTransactionDB. Transactions have a simple BEGIN/COMMIT/ROLLBACK API and allow applications to modify their data concurrently while letting RocksDB handle the conflict checking. RocksDB supports both pessimistic and optimistic concurrency control. |
| LMDB (Very high performant) | | | | | | |
| BadgerDB | | | | | | When Badger is running in in-memory mode, all the data is stored in the memory. Reads and writes are much faster in in-memory mode, but all the data stored in Badger will be lost in case of a crash or close. Badger doesn't support on-disk and in-memory mode together. |
| Amazon DynamoDB | | | | | | |
| SQLite (Not very good performance) | | | | | | Does not guarantee domain integrity. SQLite works great as the database engine for most low to medium traffic. |
Considering multiple parameters mentioned above along with support for Transactions etc. We decided to move ahead with RocksDB.