A scheduled Spark step on an Amazon EMR on EC2 cluster began failing after a dependency bundle was updated and daily input increased. Amazon CloudWatch shows brief CPU spikes and several pending YARN containers immediately before the step exits. Diagnostic output is retained in Amazon S3. The team is considering a larger cluster because the job must finish reliably. What should the data engineer do first?
Show the explanation
The retained diagnostics can identify application or dependency exceptions, while prolonged YARN metric pressure indicates insufficient resources. Brief activity immediately before termination does not establish that additional nodes would address the failure.