Monitoring of Big Data Applications

Daniel Pop, Gabriel Iuhasz

West University of Timisoara

Big Data is certainly a big hype nowdays and there is a termendous number of frameworks that enable companies to develop Big Data applications. The development of data intensive applications, like development of any other software application, needs a period of testing, validation and fine-tuning to ensure the performance and reliability the end-users expect. Throughout this period you need to constantly monitor the execution of the application in order to extract execution trends and spot the anomalies. Once in production, monitoring the application, together with its underlying infrastructure, is even more challenging: large volume of monitoring data is generated, the data is generated in different formats, and it is available either in log files, or via APIs.

There are monitoring systems on the market to help you with monitoring, but they rather tend to be focused on specific areas: Nagios, Ganglia can be used to monitor the infrastructure, Apache Chukwa or Sematext can be used to monitor Apache Hadoop, Apache Storm, or Apache Spark. Deploying and scaling these frameworks are time consuming and quite complex tasks.

This tutorial introduces the DICE Monitoring Platform (DMon), which tries to make your life easier when comes to collect, search, analyze, and visualize in real time the monitoring data being generated by the data-intensive applications. Key features of DICE Monitoring Platform are:

• leveraging on Elastic‘s open source stack – Elasticsearch for indexing and searching the data, Logstash for pre-processing incoming data and Kibana for real time visualization – DMon platform is fully distributed, highly available and horizontally scalable. All these core components of the platform have been wrapped in microservices accessible using HTPP RESTful APIs for an easy control.
• DMon is able to monitor your infrastructure, thanks to collectd plugins, and to collect data from multiple Big Data frameworks, such as Apache HDFS, YARN, Spark, or Storm (for now). With DMon you have one platform for monitoring both your infrastructure and your Big Data frameworks.
• it streamlines the control and configuration of its core components. With DMon controller service, you have a unique HTTP RESTful API you can use to both control the core components of the platform (change configuration parameters, start/stop) and to administer the monitored cluster, making possible to add, update, remove monitored nodes or start/stop services on them via GET/POST/PUT/DELETE calls.
• it supports customizable visualizations of collected data that can be structured in multiple dashboard based on your needs and roles (e.g., such as administrator, quality assurance engineer or software architect).
• deployment is possible either using Chef configuration maganagement system, or a Vagrant script for a single node installation, which you will find usefull for your development environment.

After the audience is introduced to the general concepts of Big Data Monitoring, the tutorial will deliver a hands-on experience on a testbed installation of DICE Monitoring Platform through which you will learn how to configure the platform for different Big Data technologies (e.g. Apache Spark / Storm), how to query the monitoring data stored in the Elasticsearch repository, and how to build custom visualization (e.g. system / technology focused charts). For more details about DICE Monitoring Platform please visit its Github Wiki page