How to build a custom Kafka Connect Connector

  1. abstract config (kafka clients) + valiodators configdef config
  2. source connector: depends on TaskType
  3. task class: Class :(
  4. task configs: test[Pprops] :(
  5. task “SourceTask” (or SinkTask)
  6. poll of SourceRecords (source par, source offset), metadata on the source (files, twitter, anything), to know where to start back

    • source records are stored in Kafka (?)
  7. ~> ConnectRecords (key: obj + scheme, value: obj+scheme, ts, headers, topic, partition)
  8. can use CountDownLatch (or Phaser) to pause during the polling (according to the type of source)
  9. stop would batch, release to stop
  10. await(duration): true if countdown(), good to know if task must stop - false if timeout
Ready to work with me?

Tell me everything!
© Copyright 2018-2022 · Stéphane Derosiaux · All Rights Reserved.