Engineering Cloud Computing

Announcements

Course Description

This course will teach the fundamental concepts, engineering principles, and practical skills pertaining to the effective use of cloud computing. This course will focus on both cloud applications and the design of cloud platforms. We will cover the relevant concepts from operating systems, computer networks, and distributed systems.

This course should be useful to anyone who wants a deeper understanding of how the cloud works, as well those who want to learn how to easily and effectively use the cloud for running their applications at low cost. We will look at a wide spectrum of cloud-based applications such as a parallel data processing (e.g., MapReduce), data storage and caching (e.g., key-value stores), scientific computing, interactive notebooks (e.g., Jupyter), etc.

We will also look at the challenges involved in the efficient operation of large-scale cloud platforms with hundreds of thousands of servers. The course will cover a wide gamut of data center optimization techniques such as hardware virtualization, distributed resource management, and software-defined datacenters.

This course will expose students to popular cloud platforms such as Amazon EC2, Google Cloud Platform, Microsoft Azure, etc., and introduce students to new developments such as serverless computing and edge-clouds.

Prerequisites

The course has no official prerequisites. However, it requires a high comfort-level with systems programming and debugging. The assignments in this course will include nontrivial programming in the language of your choice. A good way to guage your preparedness is to see how comfortable you are with the first programming assignment: Simple Key-Value Store. Spawn processes and sockets .

References

Text-books

  1. DS. Distributed Systems: Principles and Paradigms, 3rd Edition (Maarten Van Steen and Andrew Tanenbaum) Online version

Papers

Other references

  1. CCTP. Cloud Computing Theory and Practice. Dan C. Marinescu. (2nd edition)
  2. OS3EP : Operating Systems in Three Easy Pieces http://pages.cs.wisc.edu/~remzi/OSTEP/

Syllabus

Please see the readings for each module.

Module Topics Lectures
Cloud Overview Cloud-basics 1, 2
Preliminaries Networks, Sockets, Operating Systems, Syscalls 3, 4, 5, 6
Distributed Data Processing Map-Reduce, Spark 7, 8, 9
Cloud Architectures Computing abstractions, storage, containers 10, 11, 12
Hardware Virtualization CPU, Para-virt, Memory 5x
Cloud Efficiency Migration, Cluster-mgmt, Transient VMs, VM Deflation 3x
Scaling Queuing Theory, Parallel scaling, Elastic scaling 4x
Serverless Computing FaaS, Function KeepAlive, FaaS workflows 3x
Green Clouds MGHPCC, FB, Carbon-First 3x

Evaluation Criteria

Cloud computing is a fast evolving field. In the same spirit, the course is going to be fluid in its structure and evaluation, and also depend on student interest and capabilities. This is not a conventional "paint by numbers" course with structured homework etc.

The rough breakdown is as follows, but is subject to change:

Component Weight
Programming assignments (5) 50%
Homework and Readings 20%
Final exam 25%
Class participation 5%

Exams

The exams will test how well students have understood various virtualization techniques, cloud performance and cost tradeoffs, and how techniques learnt in class can be applied to emerging cloud offerings and applications.

Assignments

Students will implement various classic distributed algorithms (such as Map-Reduce, distributed key-value stores) on public clouds, and learn to use various cloud services such as Functions as a Service, various storage services, and how to use cloud VMs to develop and deploy applications.

The design oriented assignments will involve a large degree of programming and debugging. In most cases, the programming assignments are language agnostic (you can pick any reasonable programming language).

A key learning objective of this course is to design, architect, and implement a distributed system from scratch, and to design useful test-cases for evaluating the implementation. Therefore, no starter-code or templates will be provided, to give students the maximum flexibility and freedom to explore the unconstrained design space. Points will be awarded for correct and faithful designs, complete implementation, adequate testing, and reports and documentation.

Most programming assignments will take significantly longer than you anticipate. Start early. Please see the assignment descriptions below (from last year), to get a sense of how they will look like. In general, all programming assignments in this course only specify the "end goal", and you must figure out how to get there: what and how to implement, what libraries to use, etc. There will be no starter-code, no templates, no training wheels. You are on your own.

Likely assignments and schedule:

# Task Approx Due Date
1 Simple Key-Value Store. Spawn processes and sockets Lec 8
2 Deploy Assign 1 on GCP VMs using APIs Lec 14
3 Map-Reduce Lec 20
4 FaaS workflow Lec 25
5 FaaS KeepAlive contest End

Late submission policy

Students can avail a total of four late-days and use them as they wish. Beyond that, late submissions will not be accepted.

There is a tight integration of assignments and lectures. Hence, late submissions are discouraged. It is strongly recommended to start early—completing the assignments always takes more time than you think.

Administrative Information

Class Information

Where When
Luddy Hall Room 4063 Mondays and Wednesdays 9:45–11:00

Office Hours

Who Email Office Location Office Hours
Prateek Sharma prateeks Luddy 4126 Wed 4–5
Shubham Mohapatra shmoha Luddy 4000 Mon 2–3

Author: Prateek Sharma

Created: 2022-08-22 Mon 15:37

Emacs 25.3.50.1 (Org mode 8.2.10)

Validate