이번 글에서는 Apache Kafka Quickstart 가이드에 따라서 실행해 보도록 하겠습니다.
이전의 글(Apache Kafka(아파치 카프카) 소개)에서 Apache Kafka에 대해서 개념을 살펴보았습니다.
이제 다음과 같은 방법으로 실습을 진행하도록 하겠습니다.
STEP 1: GET KAFKA
최신 Kafka release 버전을 다운로드 합니다. 그리고 tar를 이용해 압축을 해제합니다.
$ tar -xzf kafka_2.13-3.4.0.tgz $ cd kafka_2.13-3.4.0
저는 windows가 설치되어있어서 WSL(Windows Subsystem Linux)를 Ubuntu로 설치 후 wget 유틸리티를 이용하여 다운로드 받고 압축을 해제하였습니다.
lswhh@DESKTOP-HQPQNKV:~$ mkdir kafka lswhh@DESKTOP-HQPQNKV:~$ cd kafka/ lswhh@DESKTOP-HQPQNKV:~/kafka$ ls lswhh@DESKTOP-HQPQNKV:~/kafka$ wget https://dlcdn.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz --2023-03-10 17:50:34-- https://dlcdn.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz Resolving dlcdn.apache.org (dlcdn.apache.org)... 151.101.2.132, 205.251.193.47, 205.251.196.115, ... Connecting to dlcdn.apache.org (dlcdn.apache.org)|151.101.2.132|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 106290956 (101M) [application/x-gzip] Saving to: ‘kafka_2.13-3.4.0.tgz’ kafka_2.13-3.4.0.tgz 100%[==========================================================================================================>] 101.37M 9.63MB/s in 11s 2023-03-10 17:50:45 (9.60 MB/s) - ‘kafka_2.13-3.4.0.tgz’ saved [106290956/106290956] lswhh@DESKTOP-HQPQNKV:~/kafka$ tar -xzf kafka_2.13-3.4.0.tgz lswhh@DESKTOP-HQPQNKV:~/kafka$ ls kafka_2.13-3.4.0/ kafka_2.13-3.4.0.tgz
STEP 2: START THE KAFKA ENVIRONMENT
NOTE: Java 8 이상이 설치되어있어야 한다.
Apache Kafka ZooKeeper 또는 KRaft를 사용하여 시작할 수 있습니다. 두 가지 구성 중 하나를 시작하려면 아래 섹션 중 하나를 따라하시면 됩니다.
저는 이 중 Zookeeper와 함께 동작하는 카프카를 테스트 하도록 하겠습니다.
Kafka with ZooKeeper
모든 서비스를 올바른 순서로 시작하기 위해서는 아래의 명령으로 zookeeper를 실행합니다.
# Start the ZooKeeper service $ bin/zookeeper-server-start.sh config/zookeeper.properties
실제로 local WSL ubuntu에서는 다음과 같이 실행되었습니다.
lswhh@DESKTOP-HQPQNKV:~/kafka/kafka_2.13-3.4.0$ ls LICENSE NOTICE bin/ config/ libs/ licenses/ site-docs/ lswhh@DESKTOP-HQPQNKV:~/kafka/kafka_2.13-3.4.0$ bin/zookeeper-server-start.sh config/zookeeper.properties [2023-04-20 18:39:16,327] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,330] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,348] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,348] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,348] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,348] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,352] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [2023-04-20 18:39:16,352] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager) [2023-04-20 18:39:16,352] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager) [2023-04-20 18:39:16,352] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2023-04-20 18:39:16,355] INFO Log4j 1.2 jmx support not found; jmx disabled. (org.apache.zookeeper.jmx.ManagedUtil) [2023-04-20 18:39:16,356] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,356] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,356] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,356] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,356] INFO observerMasterPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,357] INFO metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2023-04-20 18:39:16,357] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain) [2023-04-20 18:39:16,374] INFO ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@2ed94a8b (org.apache.zookeeper.server.ServerMetrics) [2023-04-20 18:39:16,380] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2023-04-20 18:39:16,395] INFO (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO ______ _ (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO |___ / | | (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO / / ___ ___ | | __ ___ ___ _ __ ___ _ __ (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO / / / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__| (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO / /__ | (_) | | (_) | | < | __/ | __/ | |_) | | __/ | | (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO /_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_| (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO | | (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,395] INFO |_| (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,396] INFO (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,398] INFO Server environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.server.ZooKeeperServer) [2023-04-20 18:39:16,398] INFO Server environment:host.name=DESKTOP-HQPQNKV.localdomain (org.apache.zookeeper.server.ZooKeeperServer)
이제 zookeeper가 수행되었으므로 다른 터미널을 열어서 다음의 명령을 수행합니다.
# Start the Kafka broker service $ bin/kafka-server-start.sh config/server.properties
Quickstart에는 위와 같이 단순하게 나와있지만 실제 테스트 시에는 kafka의 설정 중 listeners를 localhost로 시작하였습니다. 저는 호스트를 가지고 있지 않습니다.
vi 편집기에서 34번째 라인을 localhost로 바꿔줍니다.
28 # The address the socket server listens on. If not configured, the host name will be equal to the value of 29 # java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092. 30 # FORMAT: 31 # listeners = listener_name://host_name:port 32 # EXAMPLE: 33 # listeners = PLAINTEXT://your.host.name:9092 34 listeners=PLAINTEXT://localhost:9092
.png)
그리고, 실제 위에서 설명한 다음 명령을 이용하여 Kafka Service를 시작합니다.
# Start the Kafka broker service
$ bin/kafka-server-start.sh config/server.properties
lswhh@DESKTOP-HQPQNKV:~/kafka/kafka_2.13-3.4.0$ bin/kafka-server-start.sh config/server.properties
[2023-04-20 18:44:16,242] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-04-20 18:44:16,590] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-04-20 18:44:16,699] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2023-04-20 18:44:16,703] INFO starting (kafka.server.KafkaServer)
[2023-04-20 18:44:16,704] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2023-04-20 18:44:16,745] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2023-04-20 18:44:16,751] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2023-04-20 18:44:16,751] INFO Client environment:host.name=DESKTOP-HQPQNKV.localdomain (org.apache.zookeeper.ZooKeeper)
[2023-04-20 18:44:16,751] INFO Client environment:java.version=1.8.0_241 (org.apache.zookeeper.ZooKeeper)
[2023-04-20 18:44:16,751] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2023-04-20 18:44:16,751] INFO Client environment:java.home=/usr/java/jdk1.8.0_241/jre (org.apache.zookeeper.ZooKeeper)
....
[2023-04-20 18:44:19,835] INFO Loaded member MemberMetadata(memberId=connect-1-bfe8c64b-75d4-4eba-b190-fe347a3f693e, groupInstanceId=None, clientId=connect-1, clientHost=/127.0.0.1, sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, supportedProtocols=List(sessioned)) in group connect-cluster with generation 28. (kafka.coordinator.group.GroupMetadata$)
[2023-04-20 18:44:19,835] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-13 in 46 milliseconds for epoch 0, of which 39 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2023-04-20 18:44:19,836] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-28 in 47 milliseconds for epoch 0, of which 47 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
지금 까지 간략하게 kafka를 실행시켜 보았습니다. kafka를 테스트 용도로 실행 시키는 것은 간단하게 되고 있습니다.
다음 단계에서는 토픽을 생성하고 토픽에 Write와 Read를 해보도록 하겠습니다.
이 작업은 다음 포스팅에서 다루겠습니다.
아래 글을 참고하였습니다.
.png)
