Also I am wondering if Kafka Connect should be deployed on the same machine with the data source machines or if it is ok they resides on different machines. So I won't need advanced connectors which is not supported in apache version of Kafka.īut I am not sure if I am understanding the usage or scenario of Kafka Connect the right way. Beside, we can avoid installing flumes on machines belonging to others and avoid the risk of incompatible environment to ensure the stable ingestion of data from every remote machine.īesides, the most ingestion scenario is only to ingest real-time-written log text file on remote machines(on linux and unix file system) into Kafka topics, that is it. It looks with Kafka Connect we can deploy it in a centralized way with our Kafka cluster so that the develops cost can go down.
#FLUME LOGIN INSTALL#
Another reason for the consideration is that the machines' os environment varies, if we install flumes on a variety of machines, some machine with different os and jdks(I have met some with IBM jdk) just cannot make flume work well which in worst case can result in zero data ingestion.But if we use flume we need to install the agent on each remote machine which generates tons of workload for further devops, especially at the place where I am working where the authority of machines is managed in a rigid way that maintaining utilities on machines belonging to other departments.
The reason why I am considering the switch can be concluded mainly into: Now in my working scenario, now I am considering replacing the architecture of the our real time data ingestion platform which is currently based on flume -> Kafka with Kafka Connect and Kafka. I have been looking into the concepts and application of Kafka Connect, and I have even touched one project based on it in one of my intern.
0 kommentar(er)
