Big Data Engineer with Spark Streaming
Fulltime
Mississauga, Ontario, Canada- Hybrid
Note : Client is looking Big Data Engineer with Spark Streaming.
Spark Streaming or FLINK is Mandatory for this role.
Job Description : Responsibilities :
Responsibilities :
- Lead a group of developers building data pipelines using big data technologies (Spark, Flink, Databricks) on medium to large scale datasets for batch as well as real time streaming patterns
- Maintain existing data pipelines to support on premise clients and build the next generation ETL processes leveraging Databricks in Cloud
- Work in a creative & collaborative environment driven by agile methodologies with focus on CI / CD, Application Resiliency Standards
- Experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the team
- Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in around data ingestion & transformation onprem and in cloud
- Collaborate with product owners, business stakeholders to understand requirements, prioritize tasks and work with the scrum master / development team to meet program timelines
- Mentor team on technical aspects, participate in design / code reviews to ensure code is designed well, elegantly coded, and effectively tuned for performance.
Requirements :
- 10+ years of hands-on experience in solution, design and development of ETL applications using Spark (Pyspark preferred ) within Big Data space.
- 5+ years’ experience managing a team of developers focused on data pipelines, providing oversight on implementation, performing code reviews and managing Production rollouts
- 2+ years of experience in streaming applications using Spark Streaming / Flink
- 2+ year’s experience building and working with rest api’s using Java or python
- Knowledge of best practices , architectural patterns, limitations within Big Data ecosystem
- 2+ years of experience working with Agile Engineering practices.
- Knowledge and experience in non-functional requirements like Security, Performance, Vulnerabilities, etc.
Data Engineer with Python
Role: Python Data Engineer(Databricks/Pyspark, SQL, Azure). Hands-on development experience with Snowflake features such as Snow SQL; Snow Pipe; Python; Tasks; Streams; Time travel; Zero Copy Cloning; Optimizer; Metadata Manager; data sharing; and stored procedures. ETL developers are responsible fo...
Ingénieur support cloud II : Big Data (français) / Cloud Support Engineer II : Big Data (French[...]
Text French first, English follows.En tant que premier fournisseur mondial de services en nuage, Amazon Web Services (AWS) a transformé le secteur de l'informatique en nuage avec plus de 200 produits et services exceptionnels.Les équipes d'ingénieurs support d'AWS sont présentes dans le monde entier...
Data Engineer with Java Development
Document data engineering processes, creating data flow diagrams, and maintaining metadata for data lineage and cataloging. Stay current with emerging technologies and data engineering practices to recommend and adopt innovations that improve data systems. Title: Data Engineer with Java Development....
Biomedical Big Data Engineer
As a Biomedical Big Data Engineer at Myant, you will play a key role in designing, developing, and maintaining our big data infrastructure. Proven experience as a Big Data Engineer or similar role, with expertise in designing and implementing large-scale data processing solutions. Such an interface ...
Staff Software Engineer (Streaming Data)
Proven experience working with streaming data pipelines, preferably using Apache Beam, Dataflow, or similar technologies. Extensive experience with large-scale databases such as Google Bigtable/Cassandra, including building and optimizing APIs to serve data. Staff Software Engineer (Streaming Data)....
Sr Data Engineer with SQL, GCP, Python, Big Query and Kubernetes experience to pull customer data from source systems and transform the data.
Sr Data Engineer with SQL, GCP, Python, Big Query and Kubernetes experience to pull customer data from source systems and transform the data. ...
Data Engineer with Pyspark, Data bricks and Azure experience.
Engineer with Pyspark, Databricks skillset 8+ years of experience Data Engineering with min 6+ years hands on experience in Pyspark for data transformations</span></span></span></span></p> <p class="x"><span style="font-size:11pt"><spa...
Senior Data Analyst to provide analytical support using SQL, PowerBI and SAS/Python/Spark with one of our major banking clients- 35222
Senior Data Analyst to provide analytical support using SQL, PowerBI and SAS/Python/Spark with one of our major banking clients-. Google Cloud, Hadoop, Big Data, Data Lake experience. Synthesize data across multiple platforms to identify trends and deliver key insights and business recommendations a...
Biomedical Big Data Engineer
Position Overview: As a Biomedical Big Data Engineer at Myantx, you will play a key role in designing, developing, and maintaining our big data infrastructure. Proven experience as a Big Data Engineer or similar role, with expertise in designing and implementing large-scale data processing solutions...
Data Engineer with Pyspark Data bricks and Azureexperience
Engineer with Pyspark Databricks skillset 8 years of experienceData Engineering with min 6 years hands on experience in Pysparkfor datatransformations. Data Engineerwith Pyspark Data bricks and Azureexperience. Should also be able toextract and transform data especially unstructured data acrossvario...