updated the csd for Airflow 1.9.0

Author: Naveen481

README.md

@@ -5,20 +5,21 @@ This repository allows you to install [Apache Airflow](https://airflow.apache.or
 ## Requirements
 - A supported operating system.
 - MySQL or PostgreSQL database in which to store Airflow metadata.
-- [Airflow](https://github.com/teamclairvoyant/apache-airflow-parcels) and [RabbitMQ](https://github.com/teamclairvoyant/rabbitmq-cloudera-parcel) parcels need to be installed.
+- [Airflow](https://github.com/teamclairvoyant/apache-airflow-parcels)
 
 ### Currently Supported Versions of Airflow
 - Airflow 1.7.1.3
 - Airflow 1.8.0
+- Airflow 1.9.0
 
 ### Currently Supported Operating Systems
 - CentOS 6 & 7
 - RHEL 6 & 7
 
 ## Installing the CSD
-1. Download the Jar file.  [Airflow CSD](https://teamclairvoyant.s3.amazonaws.com/apache-airflow/cloudera/csd/AIRFLOW-1.8.0.jar)
+1. Download the Jar file.  [Airflow CSD](http://archive.clairvoyantsoft.com/airflow/parcels/latest/AIRFLOW-<Airflow Version>.jar)
 2. Copy the jar file to the `/opt/cloudera/csd` location on the Cloudera Manager server.
-3. Restart the Cloudera Manager Server service. `service cloiudera-scm-server restart`
+3. Restart the Cloudera Manager Server service. `service cloudera-scm-server restart`
 
 ## Requirements
 1. A database needs to be created.
@@ -61,19 +62,16 @@ There are seven roles defined in the CSD.
 1. Airflow Webserver
 2. Airflow Scheduler
 3. Airflow Worker
-4. RabbitMQ
-5. Airflow Flower
-6. Kerberos
-7. Gateway
+4. Airflow Flower
+5. Kerberos
+6. Gateway
 
 Airflow Webserver: Airflow Webserver role is used to start the Airflow Web UI. Webserver role can be deployed on more than instances. However, they will be the same and can be used for backup purposes.
 
 Airflow Scheduler: Airflow Scheduler role is used to schedule the Airflow jobs. This is limited to one instance to reduce the risk of duplicate jobs.
 
 Airflow Worker: Airflow Worker role picks jobs from RabbitMQ and executed them on the nodes. Multiple instances can be deployed.
 
-RabbitMQ: RabbitMQ role facilitates the use of RabbitMQ as the messaging broker. Currently the number of roles is limited to 1.
-
 Airflow Flower: Airflow Flower is used to monitor  celery clusters. Multiple instances are supported
 
 Kerberos: Kerberos is used to enable Kerberos protocol for the Airflow. It internally executes `airflow kerberos`. An external Kerberos Distribution Center must be setup. Multiple instances can be setup for load balancing purposes.
@@ -149,7 +147,7 @@ jar -cvf AIRFLOW-1.0.0.jar -C src/ .
 4. Only 'airflow.contrib.auth.backends.password_auth' mechanism is supported for Airflow user authentication.
 
 ## Future work:
-1. RabbitMQ needs to installed in Cluster Mode.
+1. Build RabbitMQ parcel.
 2. Test Database connection.
 3. Add the support for more Airflow user authentication methods.
 

src/descriptor/service.sdl

@@ -15,14 +15,14 @@
   "name" : "AIRFLOW",
   "label" : "Airflow",
   "description" : "Airflow is a platform to programmatically author, schedule and monitor workflows.  <span class=\"error\">Before adding this service, ensure that the Airflow and RabbitMQ parcels have been activated.</span>",
-  "version" : "1.7.1.3",
+  "version" : "1.9.0",
   "runAs" : { 
     "user" : "root",
     "group" : "root"
   },
   "parcel" : {
     "repoUrl" : "https://teamclairvoyant.s3.amazonaws.com/apache-airflow/cloudera/parcels/",
-    "requiredTags" : [ "airflow" ],
+    "requiredTags" : [ "airflow"],
     "sTags" : []
   },
   "parameters" : [
@@ -114,7 +114,7 @@
       "label" : "Broker URL",
       "description" : "Broker URL value for Celery ",
       "type" : "string",
-      "default" : "sqla+mysql://username:password@host:3306/dbname",  
+      "default" : "sqla+mysql://username:password@host:3306/dbname",	
       "configurableInWizard": true,
       "required" : "true"
     },
@@ -815,7 +815,7 @@
   ],
   "gateway" : {
     "alternatives" : {
-      "name" : "airflow",
+      "name" : "airflow_conf",
       "priority" : 50,
       "linkRoot" : "/tmp/airflowTemp/"
     },
@@ -828,15 +828,11 @@
         "dbName" : "${dbName}",
         "dbPort" : "${dbPort}",
         "dbPass" : "${dbPass}",
-        "broker_url": "${broker_url}",
+	"broker_url": "${broker_url}",
         "authenticate" : "${Airflow.authenticate}",
         "AIRFLOW_USER" : "${Airflow.username}",
         "AIRFLOW_PASS" : "${Airflow.password}",
         "AIRFLOW_EMAIL" : "${Airflow.email}",
-        "RABBITMQ_HOST" : "${RabbitMQ.host}",
-        "RABBITMQ_USER" : "${RabbitMQ.username}",
-        "RABBITMQ_PASS" : "${RabbitMQ.password}",
-        "RABBITMQ_PORT" : "${RabbitMQ.port}",
         "airflow_home" : "${airflow_home}",
         "AIRFLOW_HOME" : "${airflow_home}",
         "dags_folder" : "${dags_folder}",
@@ -865,7 +861,7 @@
     "configWriter" : {
       "generators" : [
         {
-          "filename" : "airflow/airflowTemp.cfg",
+          "filename" : "airflow_conf/airflowTemp.cfg",
           "configFormat" : "properties"
         }
       ]
@@ -1030,6 +1026,11 @@
           "AIRFLOW_HOME" : "${airflow_home}"
          }
       },
+      "stopRunner" : {
+         "runner" : {
+             "program" : "scripts/stop_airflow_scheduler.sh"
+        }
+      },
       "parameters" : [
         {
           "name" : "job_heartbeat_sec",
@@ -1166,7 +1167,7 @@
          "loggingType" : "log4j"
       }
     },
-    {
+      {
       "name" : "AIRFLOW_FLOWER",
       "label" : "Flower (Optional)",
       "pluralLabel" : "Flowers",

src/scripts/mkuser.py

@@ -0,0 +1 @@
+for pid in `ps -ef | grep -v "grep" | grep "airflow scheduler" | awk '{print $2}'` ; do kill -9 $pid || true ; done

src/scripts/start_rabbitmq.sh

@@ -32,6 +32,8 @@ if [ "$dbType" == "mysql" ]; then
 elif [ "$dbType" == "postgresql" ]; then
     sql_alchemy_conn="postgresql+psycopg2://${dbUser}:${dbPass}@${dbHost}:${dbPort}/${dbName}"
 fi
+
+echo $sql_alchemy_conn
 
 # Updating airflow.cfg
 
@@ -94,4 +96,4 @@ echo "export PATH=${AIRFLOW_DIR}/bin:\$PATH" >> /usr/bin/airflow-mkuser
 echo "${AIRFLOW_DIR}/bin/mkuser.sh \$@" >> /usr/bin/airflow-mkuser
 
 
-echo "Airflow configurations updated." 
+echo "update.cfg script ends"