DirecciónBig Data Engineer
Position Purpose :
We are seeking a highly skilled and experienced Big Data Engineer to join our international team. Working on Big Data environments and projects such as Global Data Lake, and including our focus on Sustainable Estimatics offerings.
Sustainable Estimatics is a leading suite within the company, renowned for its significant impact on the industry. With our cutting-edge and certified algorithm, we offer a unique advantage to our customers, enabling them to achieve substantial cost savings by effectively reducing waste and optimizing resource usage.
By integrating sustainability principles into our Estimatics practices, we actively contribute to the industry's collective effort to minimize environmental impact.
Our commitment to sustainability extends beyond individual projects, as we strive to drive industry-wide innovation. Through the continuous development of new technologies and practices, we aim to create a positive ripple effect that benefits not only the environment but also society as a whole.
As a Big Data Engineer, you will play a critical role in our organization's data processing initiatives, working with large amounts of data from diverse sources across the globe.
In this position, you will be responsible for developing and maintaining scalable and efficient data processing pipelines, leveraging Cloudera and Spark technologies among others, and implementing global APIs.
Your expertise in designing and implementing data ingestion, transformation, and storage processes will ensure the availability, integrity, and quality of our data across our international locations.
Your participation on different projects will assure the high level quality deliverables.
Working closely with cross-functional teams, you will collaborate to understand business requirements and translate them into technical specifications for data processing and analysis.
As part of our international team, you will be at the forefront of optimizing Spark jobs, ensuring they meet strict latency and throughput requirements while processing massive datasets.
Your ability to collaborate effectively with data scientists and analysts will enable timely and reliable delivery of data sets for advanced analytics and machine learning models.
If you are a passionate and skilled Big Data Engineer, with a strong background in Spark technologies, Spring Boot experience, and thrive in an international, data-intensive environment, we invite you to join our team.
Together, we will unlock the potential of our global data and drive meaningful insights that contribute to our organization's success.
What You'll Do
- Develop and maintain scalable and efficient data processing pipelines using Cloudera and Spark technologies among others to handle large volumes of data from diverse sources.
- Design and implement data ingestion, transformation, and storage processes to ensure data quality, integrity, and availability across international locations.
- Develop and maintain scalable API solutions to provide data to both : external and internal customers.
- Collaborate with cross-functional teams to understand business requirements and translate them into technical specifications for data processing and analysis.
- Optimize data workflows and performance of Spark jobs to meet strict latency and throughput requirements in processing massive datasets.
- Implement data governance practices and ensure compliance with data privacy and security regulations across different regions.
- Conduct troubleshooting and performance tuning of the Cloud or On-premises infrastructure to identify and resolve performance bottlenecks and optimize resource utilization.
- Collaborate with data scientists and analysts to provide timely and reliable data sets for advanced analytics and machine learning models.
- Stay updated with emerging technologies and industry trends related to Big Data processing, Cloudera, and Spark, and propose innovative solutions to enhance data processing capabilities.
- Provide technical leadership, mentorship, and guidance to junior members of the team, fostering a collaborative and learning culture within the international team.
- Participate in agile development practices, including sprint planning, code reviews, and continuous integration and deployment, to ensure high-quality software delivery.
What You'll Need
- Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
- Proven at least 5 years of working experience as a Big Data Engineer, preferably working with Cloudera and Spark technologies.
- Strong proficiency in programming languages such as Scala, Python, or Java.
- Strong knowledge of API architectures preferably developed with Spring Boot.
- In-depth knowledge of distributed computing principles and frameworks, including Hadoop and Spark.
- Hands-on experience with Cloudera distribution and related tools like HDFS, Hive, Impala, and HBase.
- Proficiency in designing, implementing, and optimizing data pipelines using Spark Streaming, Spark SQL, or other Spark modules.
- Expertise in data modeling and database design principles, including schema design, partitioning, and indexing.
- Familiarity with data ingestion techniques and tools, such as Kafka, Flume, Sqoop, or Nifi.
- Strong understanding of SQL and NoSQL databases, data warehousing concepts, and ETL processes.
- Experience with cloud platforms like AWS or Azure, and knowledge of containerization technologies like Docker or Kubernetes is a plus.
- Understanding of data governance, data privacy, and security practices, particularly in an international context.
- Excellent problem-solving and analytical skills, with the ability to troubleshoot and optimize data processing workflows.
- Strong communication and collaboration skills to work effectively in an international team setting.
- Ability to prioritize tasks, manage multiple projects simultaneously, and deliver high-quality results within defined timelines.
- Continuous learning mindset to stay updated with emerging technologies, industry trends, and best practices in Big Data Engineering.
- The ability to thrive in a fast-paced environment and take on new responsibilities quickly.
