Azure Data Engineering Architect - freelance

Description

Tasks

• Collaborate with stakeholders to understand business requirements and translate them into data engineering solutions.

• Design and oversee the overall data architecture and infrastructure, ensuring scalability, performance, security, maintainability, and adherence to industry best practices.

• Define data models and data schemas to meet business needs, considering factors such as data volume, velocity, variety, and veracity.

• Select and integrate appropriate data technologies and tools, such as databases, data lakes, data warehouses, and big data frameworks, to support data processing and analysis.

• Create scalable and efficient data processing frameworks, including ETL (Extract, Transform, Load) processes, data pipelines, and data integration solutions.

• Ensure that data engineering solutions align with the organization's long-term data strategy and goals.

• Evaluate and recommend data governance strategies and practices, including data privacy, security, and compliance measures.

• Collaborate with data scientists, analysts, and other stakeholders to define data requirements and enable effective data analysis and reporting.

• Provide technical guidance and expertise to data engineering teams, promoting best practices and ensuring high-quality deliverables. Support to team throughout the implementation process, answering questions and addressing issues as they arise.

• Oversee the implementation of the solution, ensuring that it is implemented according to the design documents and technical specifications.

• Stay updated with emerging trends and technologies in data engineering, recommending and implementing innovative solutions as appropriate.

• Conduct performance analysis and optimization of data engineering systems, identifying and resolving bottlenecks and inefficiencies.

• Ensure data quality and integrity throughout the data engineering processes, implementing appropriate validation and monitoring mechanisms.

• Collaborate with cross-functional teams to integrate data engineering solutions with other systems and applications.

• Participate in project planning and estimation, providing technical insights and recommendations.

• Document data architecture, infrastructure, and design decisions, ensuring clear and up-to-date documentation for implementation, reference and knowledge sharing.

Requirements

• Proven work experience as a Data Engineering Architect or a similar role and strong experience in in the Data & Analytics area.

• Strong understanding of data engineering concepts, including data modeling, ETL processes, data pipelines, and data governance.

• Expertise in designing and implementing scalable and efficient data processing frameworks.

• In-depth knowledge of various data technologies and tools, such as relational databases, NoSQL databases, data lakes, data warehouses, and big data frameworks (e.g., Hadoop, Spark).

• Experience in selecting and integrating appropriate technologies to meet business requirements and long-term data strategy.

• Ability to work closely with stakeholders to understand business needs and translate them into data engineering solutions.

• Strong analytical and problem-solving skills, with the ability to identify and address complex data engineering challenges.

• Proficiency in Python, PySpark, SQL.

• Familiarity with cloud platforms and services, such as AWS, GCP, or Azure, and experience in designing and implementing data solutions in a cloud environment.

• Knowledge of data governance principles and best practices, including data privacy and security regulations.

• Excellent communication and collaboration skills, with the ability to effectively communicate technical concepts to non-technical stakeholders.

• Experience in leading and mentoring data engineering teams, providing guidance and technical expertise.

• Familiarity with agile methodologies and experience in working in agile development environments.

• Continuous learning mindset, staying updated with the latest advancements and trends in data engineering and related technologies.

• Strong project management skills, with the ability to prioritize tasks, manage timelines, and deliver high-quality results within designated deadlines.

• Strong understanding of distributed computing principles, including parallel processing, data partitioning, and fault-tolerance.

• Bachelor's degree in Computer Science, Information Technology, or a related field. A Master's degree may be preferred.

Missing one or two of these qualifications? We still want to hear from you! If you bring a positive mindset, we'll provide an environment where you feel valued and empowered to learn and grow.