I will show you the kafka producer workflow using kafka-python:
Initialize
1 |
|
1 |
|
Get cluster meta data
1 |
|
Serialize
1 |
|
Partition
1 |
|
Accumulate records
1 |
|
1 |
|
Group by broker(node) and send
1 |
|
Compression
1 |
|
Socket
The Kafka producer uses the I/O multiplexing model Linux I/O model 和 JAVA NIO/AIO
Create connection
1 |
|
Send Request
1 |
|
Delivery timeout
1 |
|
SSL(Secure Sockets Layer)
In a SSL handshake, the broker will send their cetificate to client, the client will check the validity by checking if it is signed by the trusted CA. Then they will communicate one symmetric key using the SSL asymmetric public-private key which is used for the remained of the secure connection.
Create a private certificate authority(CA)
1 |
|
Create certificate for brokers
1 |
|
Sign the cetificate
1 |
|
To use SSL in kafka-python, export pem file from the keystore as Connect to Apache Kafka from Python using SSL said.
Python clients:
confluent-kafka-python : created by Confluent which is based on C client: librdkafka
kafka-python: created by open source community which is fully written by python