Overview

Description

CALENDAR DESCRIPTION:

From health care to social media the world generates a tremendous amount of data every day, often too much to be processed on a single computer or even some-times a single data centre. In this graduate seminar we will learn about technologies and systems behind Big Data. In particular, we will discuss what challenges exist in processing and storing massive amounts of data. We will explore how these challenges are being solved in real-world systems as well as the limitations inherent in these designs. The evolution of these technologies will be explored by reading both current and historically significant research papers. Students with credit for CMPT 886 when offered as a Special Topics course in Big Data may not take this course for further credit.

COURSE DETAILS:

Data engineering turns a proof-of-concept data model into an efficient, maintainable service well-matched to running in large-scale data centres. Such services are resilient against failure of either their own instances or a service dependency, deployed in a controlled way, suited for co-tenancy with services with different latency goals, instrumented to detect performance problems, and compatible with the data centre scheduler. Data engineering ensures that the services run all night while the operations staff sleep all night.

Topics

• Design space analysis: Analyze the tradeoffs inherent in a cloud service design.
• Service level metrics, indicators, objectives, and agreements: Describe their distinct uses.
• Resilience and recovery: Estimate, test, and improve the resilience of a design to failure.
• Latency versus consistency: Analyze tradeoffs between connected and partitioned performance.
• Data centre operating systems: Differentiate approaches to managing the resources of a data centre.
• Instrumentation: Classify the approaches to instrumenting a system and add instrumentation.
• Consensus: Summarize the need for consensus, particularly for service metadata and their algorithms.

Grading