zookeeper 3.4.14 and cp kafka 6.2.1 (#93)

* zookeeper 3.4.14 and cp kafka 6.2.1

* test by adding ensure

* attempt at fixing dub

* added user to volume

* applied fixes to volumes

* final updates

* test user

* test new ZK image

* added container_name

* removed volumes

* migrate to confluent inc

* fixed fullstack

Author: simplesteph

.github/workflows/main.yml

@@ -50,9 +50,6 @@ jobs:
 
       - name: run test multiple-schema-registry 7
         run: ./test.sh zk-multiple-kafka-multiple-schema-registry.yml 7
-      
-      - name: run test full-stack-no-ui 6
-        run: ./test.sh full-stack-no-ui.yml 6
 
-      - name: run test full-stack 10
-        run: ./test.sh full-stack.yml 10
+      - name: run test full-stack 7
+        run: ./test.sh full-stack.yml 7

README.md

@@ -15,17 +15,15 @@ This replicates as well as possible real deployment configurations, where you ha
 
 ## Stack version
 
-  - Zookeeper version: 3.4.9
-  - Kafka version: 2.5.0 (Confluent 5.5.1)
-  - Kafka Schema Registry: Confluent 5.5.1
-  - Kafka Schema Registry UI: 0.9.5
-  - Kafka Rest Proxy: Confluent 5.5.1
-  - Kafka Topics UI: 0.9.4
-  - Kafka Connect: Confluent 5.5.1
-  - Kafka Connect UI: 0.9.7
-  - ksqlDB Server: Confluent 6.1.1
+  - Zookeeper version: 3.5.9
+  - Kafka version: 2.8.0 (Confluent 6.2.1)
+  - Kafka Schema Registry: Confluent 6.2.1
+  - Kafka Rest Proxy: Confluent 6.2.1
+  - Kafka Connect: Confluent 6.2.1
+  - ksqlDB Server: Confluent 6.2.1
   - Zoonavigator: 0.8.0
 
+For a UI tool to access your local Kafka cluster, use the free version of (Conduktor)[https://www.conduktor.io/download]
 
 # Requirements
 
@@ -102,37 +100,15 @@ docker-compose -f zk-multiple-kafka-multiple.yml up
 docker-compose -f zk-multiple-kafka-multiple.yml down
 ```
 
-## Full stack (no UI)
-
-For this, we recommend using [Conduktor](https://conduktor.io) as your tool to bring a unified UI to all these components
-
- - Single Zookeeper: `$DOCKER_HOST_IP:2181`
- - Single Kafka: `$DOCKER_HOST_IP:9092`
- - Kafka Schema Registry: `$DOCKER_HOST_IP:8081`
- - Kafka Rest Proxy: `$DOCKER_HOST_IP:8082`
- - Kafka Connect: `$DOCKER_HOST_IP:8083`
- - KSQL Server: `$DOCKER_HOST_IP:8088`
- - Zoonavigator Web: `$DOCKER_HOST_IP:8004`
-- (experimental) JMX port at `$DOCKER_HOST_IP:9999`
-
- Run with:
- ```
- docker-compose -f full-stack-no-ui.yml up
- docker-compose -f full-stack-no-ui.yml down
- ```
-
 ## Full stack
 
-Note: the UI may be outdated or not maintained. See above for Full Stack No UI in combination with [Conduktor](https://conduktor.io)
+For this, we recommend using [Conduktor](https://conduktor.io) as your tool to bring a unified UI to all these components
 
  - Single Zookeeper: `$DOCKER_HOST_IP:2181`
  - Single Kafka: `$DOCKER_HOST_IP:9092`
  - Kafka Schema Registry: `$DOCKER_HOST_IP:8081`
- - Kafka Schema Registry UI: `$DOCKER_HOST_IP:8001`
  - Kafka Rest Proxy: `$DOCKER_HOST_IP:8082`
- - Kafka Topics UI: `$DOCKER_HOST_IP:8000`
  - Kafka Connect: `$DOCKER_HOST_IP:8083`
- - Kafka Connect UI: `$DOCKER_HOST_IP:8003`
  - KSQL Server: `$DOCKER_HOST_IP:8088`
  - Zoonavigator Web: `$DOCKER_HOST_IP:8004`
 - (experimental) JMX port at `$DOCKER_HOST_IP:9999`
@@ -153,7 +129,7 @@ A: Add the following line to your docker-compose environment variables: `KAFKA_L
 
 **Q: How do I delete data to start fresh?**
 
-A: Your data is persisted from within the docker compose folder, so if you want for example to reset the data in the full-stack docker compose, first do a `docker-compose -f full-stack.yml down`, then remove the directory `full-stack`, for example by doing `rm -r -f full-stack`.
+A: Your data is persisted from within the docker compose folder, so if you want for example to reset the data in the full-stack docker compose, do a `docker-compose -f full-stack.yml down`.
 
 **Q: Can I change the zookeeper ports?**
 
@@ -175,7 +151,7 @@ A: yes. Say you want to change `zoo1` port to `12181` (only relevant lines are s
 A: yes. Say you want to change `kafka1` port to `12345` (only relevant lines are shown). Note only `LISTENER_DOCKER_EXTERNAL` changes:
 ```
   kafka1:
-    image: confluentinc/cp-kafka:5.5.1
+    image: confluentinc/cp-kafka:6.2.1
     hostname: kafka1
     ports:
       - "12345:12345"
@@ -189,7 +165,7 @@ A: yes. This is for testing only!!! Reduce the KAFKA_LOG_SEGMENT_BYTES to 16MB a
 
 ```
   kafka1:
-    image: confluentinc/cp-kafka:5.5.1
+    image: confluentinc/cp-kafka:6.2.1
     ...
     environment:
       ...
@@ -205,7 +181,7 @@ For example, if the IP of your machine is `50.10.2.3`, follow the sample mapping
 
 ```
   kafka1:
-    image: confluentinc/cp-kafka:5.5.1
+    image: confluentinc/cp-kafka:6.2.1
     ...
     environment:
       ...

full-stack-no-ui.yml

@@ -2,22 +2,18 @@ version: '2.1'
 
 services:
   zoo1:
-    image: zookeeper:3.4.9
-    restart: unless-stopped
+    image: confluentinc/cp-zookeeper:6.2.1
     hostname: zoo1
+    container_name: zoo1
     ports:
       - "2181:2181"
     environment:
-      ZOO_MY_ID: 1
-      ZOO_PORT: 2181
-      ZOO_SERVERS: server.1=zoo1:2888:3888
-    volumes:
-      - ./full-stack/zoo1/data:/data
-      - ./full-stack/zoo1/datalog:/datalog
-
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_SERVER_ID: 1
+      ZOOKEEPER_SERVERS: zoo1:2888:3888
 
   kafka1:
-    image: confluentinc/cp-kafka:5.5.1
+    image: confluentinc/cp-kafka:6.2.1
     hostname: kafka1
     ports:
       - "9092:9092"
@@ -34,13 +30,11 @@ services:
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_JMX_PORT: 9999
       KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
-    volumes:
-      - ./full-stack/kafka1/data:/var/lib/kafka/data
     depends_on:
       - zoo1
 
   kafka-schema-registry:
-    image: confluentinc/cp-schema-registry:5.5.1
+    image: confluentinc/cp-schema-registry:6.2.1
     hostname: kafka-schema-registry
     ports:
       - "8081:8081"
@@ -52,19 +46,9 @@ services:
       - zoo1
       - kafka1
 
-  schema-registry-ui:
-    image: landoop/schema-registry-ui:0.9.5
-    hostname: kafka-schema-registry-ui
-    ports:
-      - "8001:8000"
-    environment:
-      SCHEMAREGISTRY_URL: http://kafka-schema-registry:8081/
-      PROXY: "true"
-    depends_on:
-      - kafka-schema-registry
-
+  
   kafka-rest-proxy:
-    image: confluentinc/cp-kafka-rest:5.5.1
+    image: confluentinc/cp-kafka-rest:6.2.1
     hostname: kafka-rest-proxy
     ports:
       - "8082:8082"
@@ -79,22 +63,9 @@ services:
       - kafka1
       - kafka-schema-registry
 
-  kafka-topics-ui:
-    image: landoop/kafka-topics-ui:0.9.4
-    hostname: kafka-topics-ui
-    ports:
-      - "8000:8000"
-    environment:
-      KAFKA_REST_PROXY_URL: "http://kafka-rest-proxy:8082/"
-      PROXY: "true"
-    depends_on:
-      - zoo1
-      - kafka1
-      - kafka-schema-registry
-      - kafka-rest-proxy
-
+  
   kafka-connect:
-    image: confluentinc/cp-kafka-connect:5.5.1
+    image: confluentinc/cp-kafka-connect:6.2.1
     hostname: kafka-connect
     ports:
       - "8083:8083"
@@ -133,19 +104,9 @@ services:
         confluent-hub install --no-prompt confluentinc/kafka-connect-datagen:0.4.0
         /etc/confluent/docker/run
 
-  kafka-connect-ui:
-    image: landoop/kafka-connect-ui:0.9.7
-    hostname: kafka-connect-ui
-    ports:
-      - "8003:8000"
-    environment:
-      CONNECT_URL: "http://kafka-connect:8083/"
-      PROXY: "true"
-    depends_on:
-      - kafka-connect
-
+  
   ksqldb-server:
-    image: confluentinc/cp-ksqldb-server:6.1.1
+    image: confluentinc/cp-ksqldb-server:6.2.1
     hostname: ksqldb-server
     ports:
       - "8088:8088"