release notes | UserGuide: 4.2, 5.0, 5.1, 5.2, 6.0, 6.1, 6.2, 7.0, 7.1, 7.2 (unreleased), 8.0, 8.1, 7.2 (unreleased) | Book: 4.2, 5.0, 5.1, 5.2, 6.0, 6.1, 6.2, 7.0, 7.1, 7.2 (unreleased), 8.0, 8.1, 7.2 (unreleased) | Wiki | Q&A black_bg
Edit me on GitHub

Apache-Kafka with dCache, or how to push events outside of dCache.

Starting from version 4.1 we introduced a new approach to external messaging. With the introduction of Apache-Kafka ( as a message transport, for now in Billing, we start switching to a more modern and efficient messaging system. The goal for the Billing service is easier inclusion with systems like Elasticsearch: Instead of adding messages to a textual log and having an external component parse that log, a Kafka-aware ingester can import them directly into the remote system.

What do you need to download to enable Kafka in dCache?

  • ZooKeeper Framework

  • Apache Kafka

Apache Kafka Installation.

1. Download

To install Kafka on your machine, click on the below link −

Extract the tar file

cd opt/
tar -zxf kafka_2. tar.gz
cd kafka_2.

2. Enable kafka in dCache.

dcache.enable.kafka = true

Set the broker address, or list of broker addresses

dcache.kafka.bootstrap-servers = localhost:9092

Set kafka topic name

dcache.kafka.topic = billing

“billing” is default. The following serice level variables reference dcache.kafka.topic:

dcap.kafka.topic = ${dcache.kafka.topic} ftp.kafka.topic = ${dcache.kafka.topic} nfs.kafka.topic = ${dcache.kafka.topic} pool.kafka.topic = ${dcache.kafka.topic} webdav.kafka.topic = ${dcache.kafka.topic} xrootd.kafka.topic = ${dcache.kafka.topic}

3. Start Server

bin/ config/

4. Start consumer

bin/ --bootstrap-server localhost:9092 --topic billing --from-beginning