Merge pull request #5 from teamclairvoyant/simplify

Simplify

Author: razorsedge

Makefile

@@ -0,0 +1,47 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright 2019 Clairvoyant, LLC.
+
+VERSION-CSD = $(shell bash ./version)
+VERSION-PARCEL = $(shell bash ./version-parcel)
+PACKAGE_NAME = AIRFLOW-$(VERSION-CSD)
+SHA_CMD := $(shell { command -v sha1sum || command -v sha1 || command -v shasum; } 2>/dev/null)
+
+.PHONY: help dist validate clean
+help:
+	@echo 'Please use "make <target>" where <target> is one of:'
+	@echo '  dist     : Create a CSD jarfile'
+	@echo '  validate : Run unit tests'
+	@echo '  clean    : Clean up all generated files'
+
+dist: clean validate
+	@mkdir -p target/$(PACKAGE_NAME)
+	@echo "*** Building CSD jarfile ..."
+	cp -pr src/{aux,descriptor,images,scripts} target/$(PACKAGE_NAME)
+	sed -e 's|{{ version }}|$(VERSION-CSD)|' -e 's|{{ parcel_version }}|$(VERSION-PARCEL)|' \
+		src/descriptor/service.sdl >target/$(PACKAGE_NAME)/descriptor/service.sdl
+
+	jar -cvf target/$(PACKAGE_NAME).jar -C target/$(PACKAGE_NAME) .
+	$(SHA_CMD) target/$(PACKAGE_NAME).jar | awk '{ print $$1 }' > target/$(PACKAGE_NAME).jar.sha
+	@echo "*** complete"
+
+validate: src/descriptor/service.sdl
+	@echo "*** Validating service config ..."
+	@java -jar ../../cloudera/cm_ext/validator/target/validator.jar -s src/descriptor/service.sdl
+
+validate-mdl: src/descriptor/service.mdl
+	@echo "*** Validating monitor config ..."
+	@java -jar ../../cloudera/cm_ext/validator/target/validator.jar -z src/descriptor/service.mdl
+
+clean:
+	rm -rf target

README.md

@@ -5,18 +5,17 @@ This repository allows you to install [Apache Airflow](https://airflow.apache.or
 ## Requirements
 - A supported operating system.
 - MySQL or PostgreSQL database in which to store Airflow metadata.
-- [Airflow](https://github.com/teamclairvoyant/apache-airflow-parcels) and [RabbitMQ](https://github.com/teamclairvoyant/rabbitmq-cloudera-parcel) parcels need to be installed.
 
 ### Currently Supported Versions of Airflow
-- Airflow 1.7.1.3
-- Airflow 1.8.0
+- Airflow 1.10
 
 ### Currently Supported Operating Systems
-- CentOS 6 & 7
-- RHEL 6 & 7
+- CentOS/RHEL 6 & 7
+- Debian 8
+- Ubuntu 14.04, 16.04, & 18.04
 
 ## Installing the CSD
-1. Download the Jar file.  [Airflow CSD](https://teamclairvoyant.s3.amazonaws.com/apache-airflow/cloudera/csd/AIRFLOW-1.8.0.jar)
+1. Download the Jar file.  [Airflow CSD](http://archive.clairvoyantsoft.com/airflow/csd/)
 2. Copy the jar file to the `/opt/cloudera/csd` location on the Cloudera Manager server.
 3. Restart the Cloudera Manager Server service. `service cloudera-scm-server restart`
 
@@ -57,28 +56,26 @@ create_postgresql_dbs-airflow.sh --host <host_name> --user <username> --password
 ```
 
 ## Roles
-There are seven roles defined in the CSD.
-1. Airflow Webserver
-2. Airflow Scheduler
-3. Airflow Worker
-4. RabbitMQ
-5. Airflow Flower
-6. Kerberos
-7. Gateway
+There are six roles available for deployment:
 
-Airflow Webserver: Airflow Webserver role is used to start the Airflow Web UI. Webserver role can be deployed on more than instances. However, they will be the same and can be used for backup purposes.
+1. Webserver
+2. Scheduler
+3. Worker
+4. Flower Webserver
+5. Kerberos
+6. Gateway
 
-Airflow Scheduler: Airflow Scheduler role is used to schedule the Airflow jobs. This is limited to one instance to reduce the risk of duplicate jobs.
+Webserver: Airflow Webserver role runs the Airflow Web UI. Webserver role can be deployed on more than instances. However, they will be the same and can be used for backup purposes.
 
-Airflow Worker: Airflow Worker role picks jobs from RabbitMQ and executed them on the nodes. Multiple instances can be deployed.
+Scheduler: Airflow Scheduler role is used to schedule the Airflow jobs. This is limited to one instance to reduce the risk of duplicate jobs.
 
-RabbitMQ: RabbitMQ role facilitates the use of RabbitMQ as the messaging broker. Currently the number of roles is limited to 1.
+Worker: Airflow Worker role picks jobs from the Scheduler and executes them. Multiple instances can be deployed.
 
-Airflow Flower: Airflow Flower is used to monitor  celery clusters. Multiple instances are supported
+Flower Webserver: Flower Webserver role is used to monitor Celery clusters. Celery allows for the expansion of Worker  Only one instance is needed.
 
-Kerberos: Kerberos is used to enable Kerberos protocol for the Airflow. It internally executes `airflow kerberos`. An external Kerberos Distribution Center must be setup. Multiple instances can be setup for load balancing purposes.
+Kerberos: Airflow Kerberos role is used to enable Kerberos protocol for the other Airflow roles and for DAGs. This role should exist on each host with an Airflow Worker role.
 
-Gateway: The purpose of the gateway role is to write the configurations from the configurations tab into the airflow.cfg file. This is done through the update_cfg.sh file which is executed from the scriptRunner within the gateway role.
+Gateway: The purpose of the gateway role is to make the configuration available to CLI clients.
 
 ## Using the Airflow binary:
 Here are some of the examples of Airflow commands:
@@ -103,55 +100,40 @@ The DAG file has to be copied to `dags_folder` directory within all the nodes. I
 In order to enable authentication for the Airflow Web UI check the "Enable Airflow Authentication" option. You can create Airflow users using one of two options below.
 
 ### Creating Airflow Users using UI:
-1. Navigate to Airflow CSD. In the configurations page, enter the Airflow Username, Airflow Email, Airflow Password you want to create.
-2. Deploy the client configurations to create the Airflow user.
+One way to add Airflow users to the database is using the `airflow-mkuser` script.  Users can be added as follows:
+
+1. Navigate to Airflow WebUI.
+2. In the Admin dropdown choose Users.
+3. Choose Create and enter the username, email, and password you want to create.
 
 Note: Although the last created user shows up in the Airflow configurations, you can still use the previously created users.
 
-### Using mkuser.sh
-Another way to add Airflow users is using the `mkuser.sh` script.  Users can be added as follows:
-1. Navigate to the current working directory of the CSD under `/var/run/cloudera-scm-agent/process`
-2. Export PYTHONPATH and AIRFLOW_HOME environment variables. By default these are:
+### Using airflow-mkuser
+Another way to add Airflow users to the database is using the `airflow-mkuser` script.  Users can be added as follows:
 
-   PYTHONPATH:
-   ```bash
-   export PYTHONPATH=/opt/cloudera/parcels/AIRFLOW/usr/lib/python2.7/site-packages:$PYTHONPATH
-   ```
-   Airflow Home:
-   ```bash
-   export AIRFLOW_HOME=/var/lib/airflow
-   ```
-3. Within the scripts directory, you can find the `mkuser.py` file. Execute `mkuser.py` to add a user to Airflow:
-   ```bash
-   /opt/cloudera/parcels/AIRFLOW/bin/python2.7 mkuser.py <Username> <UserEmail> <Password>
-   ```
-   For example, this can be like
-   ```bash
-   /opt/cloudera/parcels/AIRFLOW/usr/bin/python2.7 mkuser.py airflowUser airflow@email.com airflowUserPassword
-   ```
+```bash
+airflow-mkuser <username> <email> <password>
+```
+For example, this can be like:
+```bash
+airflow-mkuser admin admin@localdomain password123
+```
 
 ## Building the CSD
 ```bash
 git clone https://github.com/teamclairvoyant/apache-airflow-cloudera-csd
 cd apache-airflow-cloudera-csd
-mvn clean package
-```
-or
-```bash
-java -jar target/validator.jar -s src/descriptor/service.sdl
-jar -cvf AIRFLOW-1.0.0.jar -C src/ .
+make dist
 ```
+Update the `version` file before running `make dist` if creating a new release.
 
 ## Limitations:
-1. Number of RabbitMQ instances is limited to 1.
-2. The IP address of the RabbitMQ instance has to be manually entered during installation configuration.
-3. After deploying configurations, there is no alert or warning that the specific roles needs to be restarted.
-4. Only 'airflow.contrib.auth.backends.password_auth' mechanism is supported for Airflow user authentication.
+1. After deploying configurations, there is no alert or warning that the specific roles needs to be restarted.
+2. Only 'airflow.contrib.auth.backends.password_auth' mechanism is supported for Airflow user authentication.
 
 ## Future work:
-1. RabbitMQ needs to installed in Cluster Mode.
-2. Test Database connection.
-3. Add the support for more Airflow user authentication methods.
+1. Test Database connection.
+2. Add the support for more Airflow user authentication methods.
 
 ## Known Errors:
 
@@ -164,7 +146,8 @@ Upon many deployments, you may face an error called 'Markup file already exists'
 Occasionally, we experienced some delay in DAG execution. We are working to fix this.
 
 ## Resources:
-1. https://github.com/cloudera/cm_ext/wiki/The-Structure-of-a-CSD
-2. https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference
-3. https://github.com/cloudera/cm_csds
+1. https://github.com/teamclairvoyant/apache-airflow-parcels
+2. https://github.com/cloudera/cm_ext/wiki/The-Structure-of-a-CSD
+3. https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference
+4. https://github.com/cloudera/cm_csds
 

assembly.xml

@@ -0,0 +1,4 @@
+#!/bin/bash
+##
+# Generated by Cloudera Manager and should not be modified directly
+##