Addresses changes made to security demo config install tool (#233)

* Addresses changes made to security demo config install tool

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

* Fixes remainder admin occurences, adds version check for admin password determination and adds 2.12.0 to CI matrix

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

---------

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>

Author: DarshitChanpura

.github/workflows/CI.yml

@@ -34,7 +34,7 @@ jobs:
     strategy:
       matrix:
         logstash: [ "7.16.3", "7.17.1", "8.3.2", "8.12.1" ]
-        opensearch: [ "1.3.4", "2.1.0" ]
+        opensearch: [ "1.3.4", "2.1.0", "2.12.0" ]
         secure: [ true, false ]
 
     name: Integration Test logstash-output-opensearch against OpenSearch

DEVELOPER_GUIDE.md

@@ -186,13 +186,14 @@ Build the gem locally and install it using:
 ## Configuration for Logstash Output OpenSearch Plugin
 
 To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
 
 ```
 output {
     opensearch {
         hosts       => ["hostname:port"]
         user        => "admin"
-        password    => "admin"
+        password    => "<your-admin-password>"
         index       => "logstash-logs-%{+YYYY.MM.dd}"
     }
 }

README.md

@@ -32,12 +32,14 @@ The logstash-output-opensearch plugin helps to ship events from Logstash to Open
 ## Configuration for Logstash Output Opensearch Plugin
 
 To run the Logstash Output Opensearch plugin, add following configuration in your logstash.conf file.
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
+
 ```
 output {
     opensearch {
         hosts       => ["hostname:port"]
         user        => "admin"
-        password    => "admin"
+        password    => "<your-admin-password>"
         index       => "logstash-logs-%{+YYYY.MM.dd}"
     }
 }
@@ -62,21 +64,24 @@ output {
 In addition to the existing authentication mechanisms, if we want to add new authentication then we will be adding them in the configuration by using auth_type.
 
 Example Configuration for basic authentication:
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
+
 ```
 output {
     opensearch {
           hosts  => ["hostname:port"]
           auth_type => {
               type => 'basic'
               user => 'admin'
-              password => 'admin'
+              password => '<your-admin-password>'
           }
           index => "logstash-logs-%{+YYYY.MM.dd}"
    }
 }
 ```
 
 To ingest data into a `data stream` through logstash, we need to create the data stream and specify the name of data stream and the `op_type` of `create` in the output configuration. The sample configuration is shown below:
+Note: For logstash running with OpenSearch 2.12.0 and higher the admin password needs to be a custom strong password supplied during cluster setup.
 
 ```yml
 output {
@@ -85,7 +90,7 @@ output {
           auth_type => {
               type => 'basic'
               user => 'admin'
-              password => 'admin'
+              password => '<your-admin-password>'
           }
           index => "my-data-stream"
           action => "create"

release/docker/logstash-opensearch-sample.conf

@@ -12,6 +12,6 @@ output {
     hosts => ["http://localhost:9200"]
     index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
     #user => "admin"
-    #password => "admin"
+    #password => "<your-admin-password>"
   }
 }

scripts/logstash-run.sh

@@ -6,7 +6,16 @@ export PATH=$BUILD_DIR/gradle/bin:$PATH
 SERVICE_URL="http://integration:9200"
 
 if [[ "$SECURE_INTEGRATION" == "true" ]]; then
-  SERVICE_URL="https://integration:9200 -k -u admin:admin"
+  OPENSEARCH_REQUIRED_VERSION="2.12.0"
+  # Starting in 2.12.0, security demo configuration script requires an initial admin password
+  COMPARE_VERSION=`echo $OPENSEARCH_REQUIRED_VERSION $OPENSEARCH_VERSION | tr ' ' '\n' | sort -V | uniq | head -n 1`
+  if [ -n "$OPENDISTRO_VERSION" ] || [ "$COMPARE_VERSION" != "$OPENSEARCH_REQUIRED_VERSION" ]; then
+    CREDENTIAL="admin:admin"
+  else
+    CREDENTIAL="admin:myStrongPassword123!"
+  fi
+
+  SERVICE_URL="https://integration:9200 -k -u $CREDENTIAL"
 fi
 
 wait_for_es() {

scripts/opendistro/docker-compose.override.yml

@@ -7,6 +7,7 @@ services:
     environment:
       - INTEGRATION=true
       - SECURE_INTEGRATION=${SECURE_INTEGRATION:-false}
+      - OPENDISTRO_VERSION=${OPENDISTRO_VERSION}
 
   integration:
     build:

scripts/opensearch/docker-compose.override.yml

@@ -7,6 +7,7 @@ services:
     environment:
       - INTEGRATION=true
       - SECURE_INTEGRATION=${SECURE_INTEGRATION:-false}
+      - OPENSEARCH_VERSION=${OPENSEARCH_VERSION:-latest}
 
   integration:
     build:
@@ -19,6 +20,7 @@ services:
     environment:
       - discovery.type=single-node
       - bootstrap.memory_lock=true
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123!
     ports:
       - "9200:9200"
     user: opensearch

spec/integration/outputs/index_spec.rb

@@ -140,7 +140,7 @@
   end
   describe "a secured indexer", :secure_integration => true do
     let(:user) { "admin" }
-    let(:password) { "admin" }
+    let(:password) { OpenSearchHelper.admin_password }
     let(:opensearch_url) {"https://integration:9200"}
     let(:config) do
       {
@@ -172,7 +172,8 @@
       :auth_type => {
         "type"=>"basic",
         "user" => "admin",
-        "password" => "admin"}
+        "password" => OpenSearchHelper.admin_password
+        }
     } }
     let(:user) {options[:auth_type]["user"]}
     let(:password) {options[:auth_type]["password"]}

spec/opensearch_spec_helper.rb

@@ -72,6 +72,15 @@ def self.check_version?(*requirement)
     end
   end
 
+  # set admin password based on version
+  def self.admin_password
+    if check_version?('< 2.12.0') || check_version?('> 7')
+      "admin"
+    else
+      "myStrongPassword123!"
+    end
+  end
+
   def clean(client)
     client.indices.delete_template(:name => "*")
     client.indices.delete_index_template(:name => "logstash*") rescue nil