Data Pipeline Engineer

Job Description

REQUIREMENTS

• Bachelor’s Degree
• Minimum one year of research computing experience
• Proficiency in Linux administration (system layout, file permissions, shell, utilities)
• Experience with Apache Airflow
• Experience with database management, specifically Postgres
• Proficiency in Python and Bash scripting
• Experience with team software development tools including Git/GitHub

Preferred

• Experience with Apache Airflow version 3.0
• Knowledge of Docker and Docker Compose
• Experience with pandas, Flask, and PyPI publishing
• Familiarity with Elastic, Kibana, and FileBeat
• Proficiency with GitHub Actions and Jira Software

RESPONSIBILITIES

• Monitor and maintain the health and efficiency of data pipelines
• Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues
• Communicate with data providers to manage changes in data delivery
• Implement fixes and enhancements to improve data quality and pipeline performance
• Collaborate with data scientists and analysts to implement effective data solutions
• Develop strategies for data validation and quality assurance
• Document and manage data pipeline architectures and maintenance protocols
• Ensure compliance with data governance and security standards

Are you interested in this position?

Apply by clicking on the “Apply Now” button below!

#CrossChannelJobs #JobSearch
#CareerOpportunities #HiringNow
#Employment #JobOpenings
#JobSeekers
#FacebookLinkedIn