Hadoop Developer

Job Description

Industry: Investment Banking
Seniority for this role: Mid-Senior level
An opportunity to join our client team as ETL Hadoop Developer. The team is responsible for the design, development, delivery, and support of the technical platform behind the products and services used by the Business. Roles and Responsibilities ETL & Hadoop Development : Design and develop ETL and Hadoop applications, focusing on performance optimization and data load efficiency Collaboration & Support : Work closely with teams through all phases of testing, from development to post-implementation support Project Delivery : Manage end-to-end project delivery, including gathering business requirements, creating functional specifications, and ensuring solutions meet business needs Transformation & Tuning : Develop new transformation processes, optimize ETL code, and improve Hadoop platform performance Analysis & Enhancements : Analyze existing designs and interfaces, applying modifications or enhancements as needed Coding & Documentation : Code and document data processing scripts and stored procedures Business Insights : Provide ad-hoc data analysis and business insights Testing & Debugging : Perform software component testing, troubleshooting, and preparing migration documentation Reporting : Provide periodic updates on project/task status, ensuring transparency and clear communication Core Skills Programming & Technologies : Python, Spark, PySpark, HDFS, Hive, Hadoop Data Warehousing : Experience with Hive/Impala/Spark, strong understanding of SQL and stored procedures Relational Databases : Strong SQL skills and experience with relational databases (Teradata is a plus) ETL Development : Design and development of ETL applications, performance tuning for large data loads Scripting : UNIX Shell scripting for data warehousing solutions Optimization : Spark SQL optimization and large data load optimizations Problem-Solving : Excellent analytical, troubleshooting, and debugging skills Secondary & Desired Skills Team Collaboration : Strong team player with experience in cross-functional collaboration OLAP/Relational Databases : Experience with OLAP or relational databases is beneficial Agile Environment : Exposure to Agile development methodologies TWS Scheduler : Knowledge of TWS Scheduler is an added advantage Dimensional Modelling : Understanding of dimensional modeling and ETL solutions architecture Data Warehousing Domain : In-depth understanding of the data warehousing domain and data conversion strategies Show more Show less