Training Expert
Want to work in a culture built on mutual trust and respect? How about having the freedom to make work fit into your life (and not the other way round)? A career with Thinkproject could be just the opportunity you're looking for.
What do we do?
Thinkproject is a European market-leader in digitalisation tools for construction companies. It sounds complex, but we'll explain further! Construction companies used to use manual administration and physical paperwork for projects (sometimes hundreds of thousands of bits of paperwork for one project!). Using our construction intelligence solutions, businesses can go digital, which benefits everyone from the construction companies to the wider public.
Our mission is to deliver digitalisation to make a safer, healthier and more sustainable AECO (Architecture, Engineering, Construction, Operations) industry. This is a really exciting time to join our company, since our founding in 2000 we have gone from strength-to-strength and have lots of exciting developments coming up soon that you could be a part of.
Job Title: Data Engineer (Matillion & Snowflake)
Location : Pune
Contract : Permanent
Department: R&D
What your day will look like
· ETL Pipeline Development: Design, build, and maintain ETL pipelines using Matillion to extract data from SQL databases, transform it according to business rules, and load it into Snowflake.
· Data Transformation in Snowflake: Write and manage transformations directly within Snowflake using Snowflake Tasks, Streams, and Stored Procedures to handle specific business logic or real-time data requirements.
· Porting ETL to Snowflake & Spark: Migrate existing ETL pipelines from Matillion into Snowflake and Spark, utilizing the native features of both platforms to optimize data processing, storage, and analytics.
· Collaborate with Stakeholders: Work closely with data architects, business analysts, and other teams to understand data requirements and ensure that data pipelines are aligned with business needs.
· Performance Optimization: Continuously optimize ETL and data transformation processes in both Matillion and Snowflake for enhanced performance and reduced operational costs.
· Monitoring & Troubleshooting: Monitor the health of data pipelines and Snowflake transformations, troubleshooting issues, and implementing solutions to ensure data accuracy and availability.
· Data Preparation: Implement data cleansing, aggregation, and enrichment processes within Snowflake to prepare data for downstream reporting and analytics.
· Snowflake Management: Manage and optimize Snowflake's data architecture, ensuring best practices in data modeling, partitioning, and indexing to support analytics and reporting.
· Compliance & Governance: Ensure all ETL processes and transformations comply with data governance, privacy, and security requirements.
What you need to fulfill the role
· Expertise in Snowflake: Strong understanding of Snowflake architecture, including experience writing and managing Snowflake Data Warehouse, Snowflake Tasks, Streams, and Stored Procedures for real-time data processing and transformations. Additionally, experience optimizing Snowflake billing and credit consumption.
· SQL Databases: Proven experience working with SQL databases (e.g., MySQL, PostgreSQL, Microsoft SQL Server) and writing complex SQL queries.
· ETL Design & Optimization: Ability to design scalable ETL processes and optimize the existing ETL processes.
· Data Transformation: Experience in data cleansing, aggregation, and transformation
· Performance Tuning: In-depth knowledge of performance optimization techniques, ensuring efficient data processing.
· Proficiency in Matillion, Kafka and Spark are plus
· ML engineering or ML operations is preferred.
· Cloud Platforms: Familiarity with Power BI and Azure is an added advantage.
· Programming Languages: Experience with languages such as Python or Java is a plus.
What we offer
Lunch 'n' Learn Sessions I Women's Network I LGBTQIA+ Network I Coffee Chat Roulette I Free English Lessons I Thinkproject Academy I Social Events I Volunteering Activities I Open Forum with Leadership Team (Tp Café) I Hybrid working I Unlimited learning
We are a passionate bunch here. To join Thinkproject is to shape what our company becomes. We take feedback from our staff very seriously and give them the tools they need to help us create our fantastic culture of mutual respect. We believe that investing in our staff is crucial to the success of our business.
