Kafka Developer

Role - Kafka Developer

Location - Toronto, ON (Onsite)

Job Description :

• Expert level development experience with Python and scalable solutions.
• Experience with Kafka (confluent or Apache Kafka or MSK) is a must.
• Sound knowledge and work experience with SQL. Experience with spark and Spark Streaming (either python or scala is MUST)
• Ability to integrate different components in a streaming data pipeline.
• Sound knowledge on AWS (data related) components, ex : AWS kafka, AWS Airflow, AWS S3, AWS EMR, Lambda, AWS kinesis and firehose.
• Should be able to Design and Develop Streaming pipelines with Kafka by developing publisher and subscriber scripts. Able to troubleshoot data issues and suggest fixes by monitoring CloudWatch and other AWS logs.
• Ability to write testcases in python, Unit test, pytest etc.
• Good understanding and experience working with different Bigdata file formats along with regular JSON, xml and txt formats.
• Shell scripting is a plus.
• Should be able to work with Terraform or Cloud formation and able to handle deployment activities in AWS.
• 10-12 years of overall IT experience and 8+ years in expert level development experience with Python and scalable solutions.
• Experience with Kafka (confluent or Apache Kafka or MSK) is a must.
• Experience with spark and Spark Streaming (either python or scala is MUST) Ability to integrate different components in a streaming data pipeline.
• Sound knowledge on AWS (data related) components, ex : AWS Kafka, AWS Airflow, AWS S3, AWS EMR, Lambda, AWS kinesis and firehose.
• Should be able to Design and Develop Streaming pipelines with Kafka by developing publisher and subscriber scripts.
• Good understanding and experience working with different Bigdata file formats along with regular JSON, xml and txt formats.