daGui: A DataFlow Graphical User Interface

Loading...
Thumbnail Image
Journal Title
Journal ISSN
Volume Title
Perustieteiden korkeakoulu | Master's thesis
Date
2017-08-28
Department
Major/Subject
Cloud Computing and Services
Mcode
SCI3081
Degree programme
Master's Programme in ICT Innovation
Language
en
Pages
8+50
Series
Abstract
Big Data is a growing trend. It focuses on storing and processing a vast amount of data in a distributed environment. There are many frameworks and tools which can be used to work with this data. Many of them utilise Directed Acyclic Graphs (DAGs) in some way. A DAG is often used for expressing the dataflow of computation as it offers the possibility to optimise the execution, because it contains the overview of the whole computation. This thesis aims to create an Integrated Development Environment (IDE) like software, which is user-friendly, interactive and easily extendable. The software enables to draw a DAG which represents the dataflow of a program. The DAG is then transformed into launchable source code. Moreover, the software offers a simple way to execute the generated source code. It compiles the code (if necessary), and launches it based on the user's configuration, either on localhost or cluster. The software primarily aims to help beginners learn these technologies, but experts can also use it as visualisation for their workflow or as a prototyping tool. The software has been implemented using Electron and Web technologies, which ensure its platform independence. Its main features are code generation (i.e. translation of a DAG into source code) and code execution. It is created with extensibility in mind, to be able to plug-in support for more frameworks and tools in the future.
Description
Supervisor
Heljanko, Keijo
Thesis advisor
Dowling, Jim
Keywords
big data, apache spark, DAG, dataflow, GUI
Other note
Citation