Wednesday, 20 May 2020

Airflow SparkSubmitOperator example

Creating a connection




Note:
  • config values passed from SparkSubmitOperator takes precedence over the connection parameters. 
  • Using the spark-home in extra properties will work for all versions in case of multiple spark installations
  • default deployment mode is client. Adding 'deploy-mode':'cluster' in extra properties will change it to cluster mode
-
Labels: , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

The Mindset Behind Reliable Data Systems I’ve been in data engineering long enough to see the stack change many times over. Tools come and g...

  • Hadoop Interview Questions 2015 - Part 1
    Few more questions.. Happy Reading 1.Explain how Hadoop is different from other parallel computing solutions. 2.What are the modes ...
  • Airflow SparkSubmitOperator example
    Creating a connection Note: config values passed from SparkSubmitOperator takes precedence over the connection parameters. ...
  • Spark Notes
    Spark is a distributed in memory (mostly) computing framework/processing engine designed for batch and streaming data featuring SQL querie...