https://docs.confluent.io/current/connect/devguide.html https://docs.confluent.io/3.2.0/connect/managing.html

https://enfuse.io/a-diy-guide-to-kafka-connectors/ https://hackernoon.com/writing-your-own-sink-connector-for-your-kafka-stack-fa7a7bc201ea

How to build a custom Kafka Connect Connector

  1. abstract config (kafka clients) + valiodators configdef config

  2. source connector: depends on TaskType

  3. task class: Class :(

  4. task configs: test[Pprops] :(

  5. task “SourceTask” (or SinkTask)

  6. poll of SourceRecords (source par, source offset), metadata on the source (files, twitter, anything), to know where to start back

    • source records are stored in Kafka (?)
  7. ~> ConnectRecords (key: obj + scheme, value: obj+scheme, ts, headers, topic, partition)

  8. can use CountDownLatch (or Phaser) to pause during the polling (according to the type of source)

  9. stop would batch, release to stop

  10. await(duration): true if countdown(), good to know if task must stop - false if timeout

Ready to work with me?

Tell me everything!
© Copyright 2018 · Stéphane Derosiaux · All Rights Reserved.