Number of executors and memory calculation in spark

Consider below cluster configuration:

No.of nodes - 6

No.of cores on each node - 16

Memory on each node - 64GB

Number of executors can be considered dynamically based on the usage with the property spark.dynamicAllocation.enabled=true;

If need fixed no.of resources,Good way to design is:

1. Each node should have 1 core and 1GB memory for AM.

==>each node will have 15 cores. It is good to have max of 5 cores for each executor.

==>total no. of executors per node = 15/5 = 3

                      ==>--num-executors 3*6 =18 --executor-cores=5

2. Total memory available in each node - 63GB (As we excluded 1GB for Application Master)

==> Total Memory per each executor = 63/3=21GB

==>0.07*Total memory for Yarn overhead for each executor i.e 0.07*21 ~=2GB

                       ==>Final memory per executor=21-2=19GB i.e --executor-memory=19G