Optimisation des requêtes dans des environnements parallèles : application au Big Data

Abstract

In order to validate my Masters in Software Engineering, I had the opportunity to perform an internship in the LIAS research laboratory. I was able to work on a topic related to Big Data, which represents a challenge not only for the socio-economic world but also for scientific research. The objective of my internship was to propose an application framework to manage the various data network transfers in a transparent and efficient way. The proposed tools should be integrated into the QDAG parallel data management system, which is being developed within the LIAS IDD team. During my internship, I started by studying how the QDAG system works. I have carefully analyzed each component requiring a data transfer mechanism. I could notice that the components of QDAG use two (2) types of transfer, namely the Shuffle (with its two special cases the collector and distributor) and Broadcast. After this step of requirement analysis, I made a state of the art on the data transfer mechanisms as part of the cluster computing. I was particularly interested in the orchestra framework [25]. I then proposed not only a new design and implementation of this framework but also the necessary extensions for QDAG system. With the LIAS PhD students, I could integrate my work into QDAG's execution engine.