Engineer II, Big Data
Pilot Flying J is the 10th largest privately held company in North America with more than 28,000 team members. As the industry-leading network of travel centers, we have more than 950 retail and fueling locations in 44 states and six Canadian provinces. Our energy and logistics division is a top supplier of fuel, employing one of the largest tanker fleets and providing critical services to oil operations in our nation's busiest basins. Pilot Company supports a growing portfolio of brands with expertise in supply chain and retail operations, logistics and transportation, technology and digital innovation, construction, maintenance, human resources, finance, sales and marketing.
Founded in 1958, we are proud to be family owned and consider our team members to be part of the family. Our founding values, people-first culture and commitment to giving back remains true to us today. Whether we are serving guests, a fellow team member, or a trucking company, we are dedicated fueling people and keeping North America moving.
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status.
Pilot Flying J is part of the Pilot Company family of brands that keeps North America's drivers moving, including E-Z Trip, Mr. Fuel, One9 Fuel Stop, Pride, StaMart and Xpress Fuel.
- Experienced in working with the business partners and cross teams very frequently to get the requirements as well as designing optimized solutions given the AWS services and SLAs
- Experienced in developing features, designing the solution, creating a design document and mapping document for the dataset for the given sprint story
- Experienced with Git, Jenkins, Jupiter or any notebooks
- Expert with Pyspark and Python programming and experienced in optimizing the Pyspark configuration as well as design robust/fault tolerant end to end pipelines from cleansing, transforming and making the data available for downstream in cloud-based databases like Redshift/Snowflake.
- Expert SQL
- Experienced in working with Schema catalogs and the ability to incorporate auditing and/or logging mechanisms using Python
- Experienced working with different orchestration tools, knowledge on Bash/Powershell/python for automation or data cleaning activities
- Experienced with EMR/Databricks, migrating from SQL based applications to Pyspark based applications and working with Data science or ML teams
- Finally, your conceptual knowledge of dimensional modeling, ETL, reporting tools, data governance, data warehousing, structured and unstructured data is something you're passionate about.
As a Data Engineer, you will work closely with a cross-functional Agile team to build high quality data pipelines driving analytic solutions. These solutions will generate insights from our connected data, enabling Pilot to advance our data-driven decision-making capabilities.
Furthermore, you will:
- Design, develop, optimize, and maintain data architecture and pipelines that adhere to ETL principles and business goals
- Solve interesting data problems to deliver insights that helps our business to achieve their goals
- Create data products for analytics and data science team members to improve their productivity
- Engineer effective features for modeling in close collaboration with data scientists and translators
- Foster a culture of sharing, re-use, design for scale stability, and operational efficiency of data and analytical solutions
- Have a constant engagement with stakeholders throughout life cycle of development for any assumptions or enhancements needed
Every single day we are proud to make life better for our team members and our guests. We are fueling life's journeys and a career with Pilot is an opportunity to work hard, have fun and live up to your full potential. Do you want to become part of the Pilot family?
All your information will be kept confidential according to EEO guidelines.