Python Sr. Data Engineer
Company: Milestone Technologies, Inc.
Location: Burbank
Posted on: May 13, 2022
|
|
Job Description:
JOB DESCRIPTION: (W2 Contract; No Visa Sponsorship/No C2C; Must
be Local to Los Angeles, Seattle, or NYC. Must have: Python, SQL,
Airflow Candidate should articulate how they have used Airflow to
build data pipelines Must be local in South California , Seattle,
or NYC. Data Engineer will be building data pipelines to move
accounting and finance data from operational tools to a data lake
for analytics. Will work on either an A/R or A/P pipeline. This is
data that comes from streaming content ad revenue. They use Python,
SQL and Airflow to build the pipelines. Team doesn't have a lot of
Airflow so they want this engineer to have it so he/she can guide
team on best practices, e.g., around logging, creating modular
code, etc. They are moving to AWS Airflow but are OK with either
on-prem or managed (AWS) Airflow. Full Job Description: Description
Our team is looking for hardworking team players to join the Ad
Engineering team, who will embrace unconventional thinking, and who
are passionate about contributing through strategic hard work and
determination. We are looking for a Senior Data Pipeline Engineer
with experience in building data pipeline solutions integrating
components built on top of an AWS technology stack. If you have
experience building financial applications, taking ownership in the
technical direction of your team---s product - we---d like to talk
with you. The Data Pipeline Engineer will be responsible for
building efficient data pipelines that populate our data lake,
apply calculations and aggregations across the data set and load
the results into SQL databases that serve both analytical and
operational use cases. This role will be working closely with
different engineering teams and product managers to meet data
requirements of various initiatives in Ad Engineering. Basic
Qualifications Think and communicate critically about architecture,
design, and best practices and guide your team to adopting them.
Design data systems that allow managed growth of the data model to
minimize risk and cost of change. Write transformation and
validation code that applies complex data aggregation and
calculation using SQL and Python Drive implementation of automated
testing for data pipelines within a CI environment Create new
pipelines or rewrite existing pipelines and build reusable
components at scale to support accounting functions, as well as
reporting & analytics. Collaborate with other teams to identify and
document shifting data requirements while also advocating for a
minimal change set for your team. Solve complex data issues and
perform root cause analysis to proactively resolve product and
operational issues. Collaborate with leadership and other engineers
to develop technical story backlog derived from high level business
requirements and design collaboration and estimating story points.
BS or MS in Computer Science, a related field, or equivalent
industry experience. 3 years of professional experience engineering
complex, high-volume data pipelines using SQL, Python, and Airflow.
3 years of experience building cloud scalable and high-performance
data lake / data warehouse solutions using AWS products - S3,
Athena, Glue, and EMR Experience with binary data serialization
formats such as Parquet. Deep understanding of data structures and
algorithms. Understanding of code versioning tools such as Git.
Have a passion for data solutions. Preferred Qualifications
Exposure to AWS cloud data pipeline tools such as Managed Airflow
and Glue. Experience integrating with Ad Tech platforms such as
Operative and STAQ. Exposure and opinions regarding alternate
orchestration tooling beyond Airflow. Understanding of SOX
compliance needs and how they affect system design. Have worked
with a variety of Airflow Operator types, including REST, Lambda,
ECS Can flex between Python and JavaScript/TypeScript. Technical
Environment Aurora/Hive (databases) Spark (large-scale data
processing) Airflow (workflow management) Docker (software
packaging and delivery) AWS (development and hosting) Education
BS/MS in Computer Science or similar field
Keywords: Milestone Technologies, Inc., Burbank , Python Sr. Data Engineer, Engineering , Burbank, California
Click
here to apply!
|