Job Description

Data Engineer

Role Overview:

As a Data Engineer at SPFM, you will play a crucial role in designing, building, and maintaining scalable data pipelines and systems. Your expertise will help transform raw data into actionable insights, supporting our data-driven decision-making processes. You will work closely with data scientists, analysts, and other stakeholders to ensure the availability, accuracy, and efficiency of our data infrastructure.

Key Responsibilities:

• Data Pipeline Development: Design, build, and optimize scalable data pipelines to handle large volumes of data efficiently.
• API Development & Integrations: Create APIs for new systems to integrate with, and integrate with existing APIs.
• Data Modeling: Develop and maintain relational data models using SQL, ensuring they are efficient and scalable to meet business needs.
• System Administration: Work with Linux environments to manage and deploy data infrastructure, troubleshoot issues, and ensure system reliability.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet their needs.
• Documentation: Maintain comprehensive documentation of data pipelines, models, and processes to ensure clarity and support future development.

Required Minimum Work Experience:

• Python or R: Minimum of 2 years of experience with Python or R, with a solid understanding of their libraries and frameworks for data processing and analysis.
• SQL: Minimum of 1 year of experience with SQL, preferably with SQL Server. Proficiency in creating efficient and scalable relational data models.
• Version Control: Minimum of 1 year of experience with Git or another version control system, demonstrating the ability to manage and track code changes effectively.
• Linux: Minimum of 1 year of experience working with Linux, including system administration and troubleshooting.

Preferred Experience:

• Docker: Experience with Docker for containerizing applications for deployment.
• CI/CD Pipelines: Develop and maintain continuous integration and continuous deployment (CI/CD) pipelines to automate and streamline data workflows.

Required Experience for Python engineer:

• Experience with the following Python libraries:
• pandas
• SQLAlchemy
• Flask (or similar web frameworks, e.g. Django)
• Experience with R package development, which should also entail having experience with the following R packages:
• Sphinx (or similar documentation generators)
• pytest
• With the experience of creating Python libraries one should have experience creating clearly defined functions which are documented and tested.

Required Experience for R engineer:

• Experience with the following R packages:
• data.table
• DBI
• shiny
• openxlsx
• curl
• Experience with R package development, which should also entail having experience with the following R packages:
• roxygen2
• testthat
• With the experience of creating R packages one should have experience creating clearly defined functions which are documented and tested. Ability to use the base R package when necessary instead of creating unnecessary 3rd party dependencies.

Job Tags

Similar Jobs