Data Science and Analytics Seminar | Efficient and Scalable Systems for End-to-end Computing

The emergence of high-speed networks and the exponential growth in internet-connected and embedded devices is pioneering a new era of ubiquitous computing. As we near the end of Moore's Law, while simultaneously dealing with phenomenal growth of data volumes and velocities, there is a rapidly growing need to make use of parallel and distributed systems, from the edge devices near data sources, and on premises clusters, through to specialized accelerators, supercomputers, and public clouds. Fortunately, high-speed networks and virtualization platforms make it possible for applications to be executed anywhere, where data is generated, cycles are freely available, or computation is cheap. However, due to the heterogeneity (in terms of hardware and interfaces), highly scalable, and geographically distributed nature of modern applications, it is increasingly challenging for developers to develop general solutions, provision infrastructures, and optimize their applications.

To address these challenges we need to rethink programming paradigms and develop new systems that simplify development, offload the burden of infrastructure management, and enable portability across heterogeneous systems such that developers can focus on the problems unique to their applications. Toward these goals I will present recent work on developing systems to support parallel programming in Python and distributed function execution across heterogeneous compute resources.