Apache Spark - when things go wrong

Room 4A - Metrosoft

Are you familiar with this scenario: it’s a perfect morning, your new greenfield project (based on top of Apache Spark) runs just smoothly… and then something happens. It might be that your job drops suddenly on its initial performance or you ar struggling with an exception that ‘googled’ gave you just 3 results (2 of them in some foreign language). How to handle situations when things go wrong? This is hands-on talk, with many code examples & best practices. We will describe basic concepts like: tasks, stages, DAG schedulers. We will focus on problems of partitioning, shuffling, GC. We will see how profiles, spark-ui and solid knowledge of the API can help you with your daily Spark challenges.