Organizations running large-scale Apache Spark workloads often face a trade-off between achieving lower cost and job reliability. These tradeoffs are more prominent when using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances or when their jobs process highly skewed datasets. Three shuffle-related challenges drive the pain:
This is a companion discussion topic for the original entry at https://aws.amazon.com/blogs/big-data/high-performance-remote-shuffle-service-on-amazon-emr-with-apache-celeborn/