Python Sr. Data Engineer

Company: Milestone Technologies, Inc.
Location: Burbank
Posted on: May 13, 2022

Job Description:

JOB DESCRIPTION: (W2 Contract; No Visa Sponsorship/No C2C; Must be Local to Los Angeles, Seattle, or NYC. Must have: Python, SQL, Airflow Candidate should articulate how they have used Airflow to build data pipelines Must be local in South California , Seattle, or NYC. Data Engineer will be building data pipelines to move accounting and finance data from operational tools to a data lake for analytics. Will work on either an A/R or A/P pipeline. This is data that comes from streaming content ad revenue. They use Python, SQL and Airflow to build the pipelines. Team doesn't have a lot of Airflow so they want this engineer to have it so he/she can guide team on best practices, e.g., around logging, creating modular code, etc. They are moving to AWS Airflow but are OK with either on-prem or managed (AWS) Airflow. Full Job Description: Description Our team is looking for hardworking team players to join the Ad Engineering team, who will embrace unconventional thinking, and who are passionate about contributing through strategic hard work and determination. We are looking for a Senior Data Pipeline Engineer with experience in building data pipeline solutions integrating components built on top of an AWS technology stack. If you have experience building financial applications, taking ownership in the technical direction of your team---s product - we---d like to talk with you. The Data Pipeline Engineer will be responsible for building efficient data pipelines that populate our data lake, apply calculations and aggregations across the data set and load the results into SQL databases that serve both analytical and operational use cases. This role will be working closely with different engineering teams and product managers to meet data requirements of various initiatives in Ad Engineering. Basic Qualifications Think and communicate critically about architecture, design, and best practices and guide your team to adopting them. Design data systems that allow managed growth of the data model to minimize risk and cost of change. Write transformation and validation code that applies complex data aggregation and calculation using SQL and Python Drive implementation of automated testing for data pipelines within a CI environment Create new pipelines or rewrite existing pipelines and build reusable components at scale to support accounting functions, as well as reporting & analytics. Collaborate with other teams to identify and document shifting data requirements while also advocating for a minimal change set for your team. Solve complex data issues and perform root cause analysis to proactively resolve product and operational issues. Collaborate with leadership and other engineers to develop technical story backlog derived from high level business requirements and design collaboration and estimating story points. BS or MS in Computer Science, a related field, or equivalent industry experience. 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow. 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR Experience with binary data serialization formats such as Parquet. Deep understanding of data structures and algorithms. Understanding of code versioning tools such as Git. Have a passion for data solutions. Preferred Qualifications Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue. Experience integrating with Ad Tech platforms such as Operative and STAQ. Exposure and opinions regarding alternate orchestration tooling beyond Airflow. Understanding of SOX compliance needs and how they affect system design. Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS Can flex between Python and JavaScript/TypeScript. Technical Environment Aurora/Hive (databases) Spark (large-scale data processing) Airflow (workflow management) Docker (software packaging and delivery) AWS (development and hosting) Education BS/MS in Computer Science or similar field

Keywords: Milestone Technologies, Inc., Burbank , Python Sr. Data Engineer, Engineering , Burbank, California