Senior Data Engineer
Pilot Flying J is the 10th largest privately held company in North America with more than 28,000 team members. As the industry-leading network of travel centers, we have more than 950 retail and fueling locations in 44 states and six Canadian provinces. Our energy and logistics division is a top supplier of fuel, employing one of the largest tanker fleets and providing critical services to oil operations in our nation's busiest basins. Pilot Company supports a growing portfolio of brands with expertise in supply chain and retail operations, logistics and transportation, technology and digital innovation, construction, maintenance, human resources, finance, sales and marketing.
Founded in 1958, we are proud to be family owned and consider our team members to be part of the family. Our founding values, people-first culture and commitment to giving back remains true to us today. Whether we are serving guests, a fellow team member, or a trucking company, we are dedicated fueling people and keeping North America moving.
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status.
Pilot Flying J is part of the Pilot Company family of brands that keeps North America's drivers moving, including E-Z Trip, Mr. Fuel, One9 Fuel Stop, Pride, StaMart and Xpress Fuel.
The purpose of this job is to 1) lead projects and provide technical expertise for research, development and modification of extract, transform, load processes and jobs in support of a Big Data infrastructure for Pilot Flying J (PFJ) and, 2) lead the implementation and support of the enterprise AWS Cloud environment, with emphasis in EMR and Redshift.
- Research, develop, document, and modify Big Data Lake processes and jobs per data architecture and modeling requirements; collaborate with Business Intelligence (BI) data strategists and data scientists
- Provide technical expertise to implement BI specifications
- Provide design, capacity planning, cluster design, performance fine-tuning, monitoring, structure planning, scaling of an enterprise AWS environment
- Ensure team accountability for completing key enterprise data warehouse (EDW) and Big Data Lake project activities; communicate project status and project risks to stakeholders
- Serve on cross-functional project teams and provide the big data perspective on executing key deliverables
- Troubleshoot complex, escalated issues including connection, failed jobs, application errors, server alerts and space thresholds within predefined service level agreements (SLAs)
- Proactively maintain and tune all code according to Big Data and EDW best practices to prevent issues
- Review and ensure appropriate documentation for all new development and modifications of the Big Data Lake processes and jobs
- Perform code reviews and oversee unit testing for solutions developed, and ensure integrity and security of institutional data
- Translate business requirements into technical specifications including the creation of mapping documents to support the Big Data development process and the execution of Big Data Lake processes
- Educate customers on the usage and benefits of the EDW, Big Data Lake and related technologies
- Mentor and guide less experienced team members and provide feedback on project work
- Model behaviors that support the company’s common purpose; ensure guests and team members are supported at the highest level
- Ensure all activities are in compliance with applicable rules, regulations, policies, and procedures of the organization
- Complete other duties as assigned
Bachelor’s degree in computer science, engineering, information technology , or related field, required
Minimum five years’ technology operations experience required; seven years’ experience preferred
Expert SQL knowledge and skills required
Strong knowledge of Postgres and SQLServer required
Expert knowledge of AWS or similar Big Data platform required
Experience with Spark/Spark-streaming, Message Que technologies and Python required
Expert knowledge of EDW data models with a focus on Star Schema data modeling techniques required
PFJ institutional knowledge
Excellent analytical skills and the ability to identify solutions to complex data problems
Ability to provide excellent customer service
Strong organizational skills with attention to detail and accuracy
Excellent written and verbal communication skills
Ability to communicate effectively across organizational levels and with leadership
Willingness to learn and embrace new technologies
Ability to prioritize and execute tasks in a fast-paced and dynamic environment
Ability to take initiative, work independently and with a team
Travel required less than 10%
General office work requiring sitting or standing for long periods of time
Participate in a 24/7 on-call rotation for after hour and weekend production support
All your information will be kept confidential according to EEO guidelines.