The Airflow Data Engineer is responsible for designing, implementing, deploying, and supporting various data management technologies and architectures. In partnership with business leaders, key stakeholders and cross-functional project teams, the Data Engineer will be an active contributor in a collaborative team structure and will have the opportunity to accelerate the delivery of and improve the quality of products providing increased operational excellence, a greater client experience and other strategic objectives.
- Experience with data pipeline and workflow management tools: Airflow.
- Experience with developing guidelines for Airflow clusters and DAG's/Task's etc.
- Experience with Performance tuning of the DAG and task implementation
- Develop DAG - data pipeline to on-board and change management of datasets
- Experience installing Apache Airflow, configuring, and monitoring Airflow cluster
- Understanding of airflow rest services and integration of airflow platform eco-system
- Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with databases.
- Knowledge about big data eco systems like Cloudera/Hortonworks components hadoop,spark,Hive etc..
- Experience in building and optimizing - big data- data pipelines, architectures, and data sets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
- Orchestrating the Airflow/workflow in hybrid cloud platform AWS/Azure/GCP setup and administration is a plus.
- Proficient in modern programming languages (Python,Java) with experience and open-source technologies.
- Possess professional software engineering practices and best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations.
- Knowledge in DEVOPS models with experience in containerization platform via building the docker,assembly,deployment,automation etc.. and best practices for managing the Applications.
- Experience in managing a shared services is a plus and knowledge about Infrastructure management capacity assessment,forecast,planning etc..
- Ability to maintain clean and secure data environments
- Fast learner and a team player
- Apache Airflow Fundamentals
- Agile/Scrum principles
Thanx and regards
Job Type: Full-time
Salary: $110,000.00-$120,000.00 per year
- 8 hour shift