Data Engineer for Bangalore

Job Description:

Position Overview:
We are seeking an experienced Data Engineer to join our innovative data team data team and help build scalable data infrastructure, software consultancy, and development services that powers business intelligence, analytics, and machine learning initiatives. The ideal candidate will design, develop, and maintain robust high-performance data pipelines and solutions while ensuring data quality, reliability, and accessibility across the organization working with cutting-edge technologies like Python, Microsoft Fabric, Snowflake, Dataiku, SQL Server, Oracle, PostgreSQL, etc.

Experience: 5 to 7 years
Location: Bengaluru
Employment Type: Full-time / Permanent

Key Responsibilities:
Data Pipeline Development
 Design, build, and maintain highly efficient and scalable ETL/ELT pipelines to process large volumes of structured and unstructured data
 Implement real-time and batch data processing solutions using modern data engineering tools (e.g., Fabric, Spark, Kafka, Airflow, )
 Optimize data workflows for performance, reliability, and cost-effectiveness
 Monitor pipeline health and implement automated alerting and recovery mechanisms Data Architecture & Infrastructure
 Collaborate with architects to design and implement data warehouse and Data Warehouse, Data Lake, and Lakehouse solutions (e.g., Fabric, Snowflake).
 Build and maintain cloud-based data infrastructure (Azure or AWS, GCP)
 Implement data governance frameworks and ensure compliance with data privacy regulations
 Design schema and data models for optimal query performance and scalability Data Quality & Integration

 Develop data validation frameworks and implement quality checks throughout pipelines
 Integrate data from multiple sources including APIs, databases, streaming platforms, and third-party services
 Create and maintain data documentation, lineage tracking, and metadata management
 Troubleshoot data quality issues and implement corrective measures Collaboration & Support
 Partner with data scientists, analysts, and business stakeholders to understand data requirements
 Support analytics teams with data access, modeling, and optimization
 Participate in code reviews and maintain high standards for data engineering practices
 Mentor junior team members and contribute to team knowledge sharing

Skills
 Knowledge of machine learning workflows and MLOps practices
 Familiarity with data visualization tools (Tableau, Looker, Power BI)
 Experience with stream processing and real-time analytics
 Experience with data governance and compliance frameworks (GDPR, CCPA)
 Contributions to open-source data engineering projects
 Relevant Cloud certifications (e.g., Microsoft Certified: Azure Data Engineer Associate, AWS Certified Data Engineer, Google Cloud Professional Data Engineer).
 Specific experience or certifications in Microsoft Fabric, or Dataiku, Snowflake.

Required Qualifications
Technical Skills
 Programming Languages: Proficiency in Python
 Cloud Platforms: Hands-on experience with Azure (Fabric, Synapse, Data Factory, Event Hubs)

 3+ years of experience in data engineering or related roles with 5+ years of Software Engineering Experience.
 Databases: Strong SQL skills and experience with both relational (Microsoft SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases
 Version Control: Proficiency with Git and collaborative development workflows
 Proven track record of building production-grade data pipelines handling large-scale data or solutions.
 Desired experience with containerization (Docker) and orchestration (Kubernetes) technologies.