Airflow Data Engineer

Full Time
Mississauga, ON
$110,000–$120,000 a year
Posted 17 days ago
Job description

The Airflow Data Engineer is responsible for designing, implementing, deploying, and supporting various data management technologies and architectures. In partnership with business leaders, key stakeholders and cross-functional project teams, the Data Engineer will be an active contributor in a collaborative team structure and will have the opportunity to accelerate the delivery of and improve the quality of products providing increased operational excellence, a greater client experience and other strategic objectives.

  • Experience with data pipeline and workflow management tools: Airflow.
  • Experience with developing guidelines for Airflow clusters and DAG's/Task's etc.
  • Experience with Performance tuning of the DAG and task implementation
  • Develop DAG - data pipeline to on-board and change management of datasets
  • Experience installing Apache Airflow, configuring, and monitoring Airflow cluster
  • Understanding of airflow rest services and integration of airflow platform eco-system
  • Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with databases.
  • Knowledge about big data eco systems like Cloudera/Hortonworks components hadoop,spark,Hive etc..
  • Experience in building and optimizing - big data- data pipelines, architectures, and data sets.
  • Build processes supporting data transformation, data structures, metadata, dependency and workload management.
  • Orchestrating the Airflow/workflow in hybrid cloud platform AWS/Azure/GCP setup and administration is a plus.
  • Proficient in modern programming languages (Python,Java) with experience and open-source technologies.
  • Possess professional software engineering practices and best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations.
  • Knowledge in DEVOPS models with experience in containerization platform via building the docker,assembly,deployment,automation etc.. and best practices for managing the Applications.
  • Experience in managing a shared services is a plus and knowledge about Infrastructure management capacity assessment,forecast,planning etc..
  • Ability to maintain clean and secure data environments
  • Fast learner and a team player
  • Apache Airflow Fundamentals
  • Agile/Scrum principles

Thanx and regards

Pinky

Job Type: Full-time

Salary: $110,000.00-$120,000.00 per year

Schedule:

  • 8 hour shift

Intrested in this job?


Related Jobs

All Related Listed jobs

Vaco Toronto, ON Today

The Senior Specialist, Security Architecture will assist in the execution of the application security testing program that will serve to improve the security posture of Information Technology

Baycrest Toronto, ON 11 days ago

AFTER HOURS SUPERVISOR, PART TIME. Non-Union | 15 Bi-Weekly Hours (minimum 15 hours bi-weekly but opportunity to commit up to 35 hours ).

Date de clture (mm/jj/aaaa): 08/31/2022 Type de travailleur : Permanent Langue(s) requise(s): anglais Dure dtermine (en mois) : Connaissance de la gestion des

peoplecanfly Montréal, QC 30+ days ago

Design and implement backend services running on the PCF Online Services Platform. Contribute to the architecture of the online services platform itself.

BrightPath Kids Canada Edmonton, AB Today

Center: Millwoods, AB Government Wage Enhancement offered on top of hourly rate Job Type: Part-Time, Casual No evenings or weekends! BrightPath Kids, part of the Busy Bees Global Childcare

RONA Lachine Lachine, QC Today

Titre du poste : CommisDpartement : Entrept et cour Description sommaire : Rencontre et dpasse les exigences du dpartement en donnant un service courtois et efficace et en prenant des

Kinatex Sports Physio Joliette, QC 30+ days ago

Pionnier en radaptation au Qubec, Kinatex Sports Physio reprsente plus de 55 cliniques et plus de 900 employs passionns! Effectuer le suivi des patients.

Craig Budd - Desjardins Insurance Agent Grimsby, ON $50,000–$80,000 a year 19 days ago

Life insurance license (must be able to obtain). Independent contractor agents are responsible for and make all employment decisions regarding their employees.

Nuna Logistics Limited Edmonton, AB Today

The Nuna Group of Companies is now accepting resumes for Project Coordinator(s) with large earthworks construction projects experience. Work locations / positions available in:

Minimum of twenty-five days of vacation. Participates as a member of the Corporate Services management team and contributes to the development of strategies,