Senior Data Architect / Engineer

As part of our aggressive growth strategy, we are seeking a highly skilled and versatile Data Engineer with strong Database Administrator (DBA) expertise, Microsoft Azure and Azure Data Factory to join our team. This critical position will support our rapid expansion by managing and integrating complex data sets from both internal and external sources. The role involves designing and building data pipelines that process, transform, and consolidate data, making it accessible for custom software applications and advanced reporting tools. You will work closely with cross-functional teams to ensure that data is structured, optimized, and ready for consumption, all while ensuring that our Azure-based database systems remain secure, scalable, and high-performing.

This position reports directly to the Managing Director of Asset Management and plays a key role in helping us meet our growth objectives by transforming data into actionable insights for key business decisions.

Responsibilities

Data Engineering:

• Design and Build Data Pipelines: Develop and implement ETL (Extract, Transform, Load) pipelines using Azure Data Factory to integrate data from various sources into our Azure data infrastructure, ensuring data is clean, structured, and ready for analysis.
• Data Architecture: Architect and maintain scalable, efficient, and secure data systems on Microsoft Azure, utilizing services such as Azure SQL Database, Azure Synapse Analytics, and Azure Blob Storage to handle large datasets, both structured and unstructured. As well as familiarity with SQL Server due to transitional data. Structure data for effective consumption by Large Language Models (LLMs), leveraging your understanding of LLM schemas and data structuring techniques.
• Data Processing: Optimize data storage, processing, and retrieval systems for performance and scalability across large-scale Azure environments.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable, high-quality data solutions leveraging Azure services.
• Data Quality:Ensure data integrity, quality, and consistency across the pipeline by implementing data validation and cleansing procedures within Azure environments.
• Automation: Automate ETL processes using Azure Data Factory and Azure Functions to streamline workflows and minimize manual tasks.
• Monitoring and Maintenance: Utilize Azure Monitor and Azure Log Analytics to monitor the performance and health of data systems, troubleshooting issues as they arise to ensure systems run efficiently and are up to date.

Database Administration:

• Azure SQL Database Management: Manage and optimize Azure SQL Databases and Azure Synapse Analytics for performance, scalability, and security.
• Database Security: Implement robust database security measures, access controls, and user management using Azure Active Directory, ensuring compliance with organizational policies and industry standards.
• Backup and Recovery: Develop and manage backup strategies and disaster recovery procedures using Azure Backup and Azure Site Recovery for critical data systems, ensuring minimal data loss in case of failures.
• Database Optimization: Monitor and optimize database performance, including query optimization, indexing, and resource management, utilizing tools like Azure SQL Analytics to ensure fast and efficient data access.
• Database Upgrades & Patching: Perform regular maintenance tasks such as database upgrades, patching, and capacity planning within Azure environments to ensure database systems are up-to-date and secure.
• Data Migration & Integration: Oversee and assist with data migration tasks, including moving on-premises databases to Azure or integrating with third-party data systems using Azure Database Migration Service.
• Troubleshooting & Support: Resolve database-related issues, conduct root cause analysis, and implement corrective actions using Azure Support tools to ensure high availability and reliability of database services.

Qualifications

Required Skills:

• Proven Experience: Minimum 5 years of experience as a Data Engineer, with at least 5 years of experience as a Database Administrator or in similar roles within Azure environments.
• Azure Expertise: Strong expertise in managing and deploying data solutions on Microsoft Azure, including Azure Data Factory, Azure SQL Database, Azure Synapse Analytics, Azure Blob Storage, and other Azure data services. Azure Data
• Database Management: Extensive experience with SQL and NoSQL databases (e.g., Azure Managed Instance, Azure Cosmos DB, MongoDB, PostgreSQL), with expertise in database administration, performance tuning, and optimization.
• ETL Tools: Proficiency with ETL frameworks and tools such as Azure Data Factory, Apache NiFi, Talend, or similar.
• Programming: Proficiency in programming languages such as Python, Java, or C#, with the ability to develop and optimize data processing pipelines within Azure.
• Big Data Tools: Familiarity with big data technologies such as Azure Databricks, Hadoop, Spark, or Kafka is a plus.
• Cloud Services: Comprehensive knowledge of Azure services and architecture for data storage, computing, and analytics.
• Backup & Recovery: Experience with backup, recovery, and disaster recovery strategies within Azure Backup and Azure Site Recovery.
• Version Control: Proficiency with version control systems like Github.

Preferred Skills:

• Data Warehousing: Experience with modern data warehousing solutions like Azure Synapse Analytics, Snowflake, or similar.
• Data Modeling: Knowledge of data modeling techniques for both OLTP and OLAP systems within Azure environments.
• Containerization: Familiarity with Docker and Kubernetes for managing database and data engineering workflows on Azure Kubernetes Service (AKS).
• Business Intelligence: Experience with BI tools (e.g., Power BI, Tableau) for providing insights and reports to business teams, leveraging Azure Analysis Services.
• Large Language Models:Structure data for effective consumption by Large Language Models (LLMs), leveraging your understanding of LLM schemas and data structuring techniques.

Education:

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• Relevant certifications (e.g., Microsoft Certified: Azure Data Engineer Associate, Microsoft Certified: Azure Database Administrator Associate) are highly desirable.

Soft Skills:

• Problem Solving: Strong analytical and troubleshooting skills, with the ability to think critically and solve complex problems.
• Collaboration: Excellent communication and teamwork skills to collaborate with cross-functional teams.
• Attention to Detail: Ability to maintain accuracy and quality in database management and data engineering tasks.
• Time Management: Excellent time management skills, with the ability to handle multiple priorities and deadlines in a fast-paced environment.