PySpark Expert

April 23, 2024

PySpark Expert

ReferenceBNP024580

  • Standard / Permanent
  • IN-TN-Chennai
  • INFORMATION TECHNOLOGY
Apply for this job

About BNP Paribas India Solutions:

Established in 2005, BNP Paribas India Solutions is a wholly owned subsidiary of BNP Paribas SA, European Union’s leading bank with an international reach. With delivery centers located in Bengaluru, Chennai and Mumbai, we are a 24×7 global delivery center. India Solutions services three business lines: Corporate and Institutional Banking, Investment Solutions and Retail Banking for BNP Paribas across the Group. Driving innovation and growth, we are harnessing the potential of over 10000 employees, to provide support and develop best-in-class solutions.

About BNP Paribas Group:

 BNP Paribas is the European Union’s leading bank and key player in international banking. It operates in 65 countries and has nearly 185,000 employees, including more than 145,000 in Europe. The Group has key positions in its three main fields of activity: Commercial, Personal Banking & Services for the Group’s commercial & personal banking and several specialised businesses including BNP Paribas Personal Finance and Arval; Investment & Protection Services for savings, investment, and protection solutions; and Corporate & Institutional Banking, focused on corporate and institutional clients. Based on its strong diversified and integrated model, the Group helps all its clients (individuals, community associations, entrepreneurs, SMEs, corporates and institutional clients) to realize their projects through solutions spanning financing, investment, savings and protection insurance. In Europe, BNP Paribas has four domestic markets: Belgium, France, Italy, and Luxembourg. The Group is rolling out its integrated commercial & personal banking model across several Mediterranean countries, Turkey, and Eastern Europe. As a key player in international banking, the Group has leading platforms and business lines in Europe, a strong presence in the Americas as well as a solid and fast-growing business in Asia-Pacific. BNP Paribas has implemented a Corporate Social Responsibility approach in all its activities, enabling it to contribute to the construction of a sustainable future, while ensuring the Group’s performance and stability

Commitment to Diversity and Inclusion

At BNP Paribas, we passionately embrace diversity and are committed to fostering an inclusive workplace where all employees are valued, respected and can bring their authentic selves to work. We prohibit Discrimination and Harassment of any kind and our policies promote equal employment opportunity for all employees and applicants, irrespective of, but not limited to their gender, gender identity, sex, sexual orientation, ethnicity, race, colour, national origin, age, religion, social status, mental or physical disabilities, veteran status etc. As a global Bank, we truly believe that inclusion and diversity of our teams is key to our success in serving our clients and the communities we operate in.

About Businessline/Function :

Under BNP Paribas IT Group, FRESH is the IT framework encompassing Projects & Systems of Finance and RISK functions, with a strong focus on efficiency, proximity, agility and time-to-market.

FRESH is a worldwide organizational model that will contribute to tackle major challenges of the Group’s transformation of today and tomorrow, building the best offering to the Finance and Risk Community through a progressive rotation to new technologies and best practices. The FRESH organization is the outcome of several convergences: between the Project & Systems teams of Group Finance, CIB Finance and RISK and their related IT teams.

FRESH is part of the IT Group organization (ITG in Europe and ITO overseas) with a threefold sponsorship by ITG, Finance and RISK (CIB-ITO; RISK Solutions and ITG-FIT). 

Job Title:

Developer

Date:

25-Mar-2024

Department:

FRESH – IHC ETL

Location:

Global Infocity – Chennai

Business Line / Function:

IHC ETL

Reports to:

(Direct)

Grade: (if applicable)

(Functional)

Number of Direct Reports:

0

Directorship / Registration:

NA

Position Purpose

The Senior Developer will be a part of the ISPL  Chennai – IHC ETL projects team.  The developer position will primarily work on Informatica ETL tool ,Unix ,Autosys and DB  

Responsibilities

  • Developed using Apache Spark & Python Structured data API / Library.
  • Strong hands on experience in Processing Framework – Spark 2.x/3.x (Core, structured API,Streaming, MLLib) Language & Package – Python (Scripting & PySpark), Scala using Spark API,Unix Shell, SQL Query (basic & advanced)
  • Knowledge on Streaming Platform – Apache Kafka, Apache Nifi, Spark Streaming, Flink, Storm
  • Extensive hands on experience in designing, building and executing data pipeline using ETL/ELT tool Apache Nifi (Source, Extraction and Processors & Sink modules).
  • Big Data Hadoop – Hortonworks HDP 3.1.x & core Components covering data engineering, management, operation stack
  • Big Data Hadoop – Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage – HDFS (File Format – Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning) , Data Lake (Object Store),
  • Expertise in RDBM solutions (Postgres & Oracle)

Technical & Behavioral Competencies

•            Minimum 6 to 10 years hands-on experience with PySpark ,Kubernetes,Docker

•            Strong technical expertise PySpark ,Kubernetes,Docker and be in position to handle all technical difficulties

•            Strong in designing (data warehousing) concepts.

•            Good working knowledge in Unix/Ubuntu ( should be able to write wrapper scripts )

•            Capable of tuning the code to handle the huge data volume.

•            Responsible for translating/understanding the functional requirements so as to meet the specified technical requirements.

•            Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied

•            Involved in coding, testing, implementing, debugging and documenting the complex programs.

•            Involved in creating proper technical documentation in the work assignments.

•            Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process.

•            Ensures that all the standard requirements have been met and is involved in performing the technical analysis.

•            Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements.

•            Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming.

Behavioral Competencies

·       Excellent verbal and written communication skills.

·       Conduct meetings with global stakeholders, prepare minutes & summaries.

·       Assertiveness, Negotiation, Proactivness & Prioritization skills are important.

·       Discipline in documenting, following-up on issues and changes.

·       Experience in interacting with global stakeholders and independently managing discussions.

Specific Qualifications (if required)

Skills Referential

Behavioural Skills: (Please select up to 4 skills)

Communication skills – oral & written

Ability to synthetize / simplify

Attention to detail / rigor

Organizational skills

Transversal Skills: (Please select up to 5 skills)

Analytical Ability

Ability to understand, explain and support change

Ability to develop and adapt a process

Ability to manage / facilitate a meeting, seminar, committee, training…

Ability to develop and leverage networks

Education Level:

Bachelor Degree or equivalent

Experience Level

At least 7 years

Other/Specific Qualifications (if required)

Additional knowledge on reporting tools is advantage

About BNP Paribas India Solutions:

Established in 2005, BNP Paribas India Solutions is a wholly owned subsidiary of BNP Paribas SA, European Union’s leading bank with an international reach. With delivery centers located in Bengaluru, Chennai and Mumbai, we are a 24×7 global delivery center. India Solutions services three business lines: Corporate and Institutional Banking, Investment Solutions and Retail Banking for BNP Paribas across the Group. Driving innovation and growth, we are harnessing the potential of over 10000 employees, to provide support and develop best-in-class solutions.

About BNP Paribas Group:

 BNP Paribas is the European Union’s leading bank and key player in international banking. It operates in 65 countries and has nearly 185,000 employees, including more than 145,000 in Europe. The Group has key positions in its three main fields of activity: Commercial, Personal Banking & Services for the Group’s commercial & personal banking and several specialised businesses including BNP Paribas Personal Finance and Arval; Investment & Protection Services for savings, investment, and protection solutions; and Corporate & Institutional Banking, focused on corporate and institutional clients. Based on its strong diversified and integrated model, the Group helps all its clients (individuals, community associations, entrepreneurs, SMEs, corporates and institutional clients) to realize their projects through solutions spanning financing, investment, savings and protection insurance. In Europe, BNP Paribas has four domestic markets: Belgium, France, Italy, and Luxembourg. The Group is rolling out its integrated commercial & personal banking model across several Mediterranean countries, Turkey, and Eastern Europe. As a key player in international banking, the Group has leading platforms and business lines in Europe, a strong presence in the Americas as well as a solid and fast-growing business in Asia-Pacific. BNP Paribas has implemented a Corporate Social Responsibility approach in all its activities, enabling it to contribute to the construction of a sustainable future, while ensuring the Group’s performance and stability

Commitment to Diversity and Inclusion

At BNP Paribas, we passionately embrace diversity and are committed to fostering an inclusive workplace where all employees are valued, respected and can bring their authentic selves to work. We prohibit Discrimination and Harassment of any kind and our policies promote equal employment opportunity for all employees and applicants, irrespective of, but not limited to their gender, gender identity, sex, sexual orientation, ethnicity, race, colour, national origin, age, religion, social status, mental or physical disabilities, veteran status etc. As a global Bank, we truly believe that inclusion and diversity of our teams is key to our success in serving our clients and the communities we operate in.

About Businessline/Function :

Under BNP Paribas IT Group, FRESH is the IT framework encompassing Projects & Systems of Finance and RISK functions, with a strong focus on efficiency, proximity, agility and time-to-market.

FRESH is a worldwide organizational model that will contribute to tackle major challenges of the Group’s transformation of today and tomorrow, building the best offering to the Finance and Risk Community through a progressive rotation to new technologies and best practices. The FRESH organization is the outcome of several convergences: between the Project & Systems teams of Group Finance, CIB Finance and RISK and their related IT teams.

FRESH is part of the IT Group organization (ITG in Europe and ITO overseas) with a threefold sponsorship by ITG, Finance and RISK (CIB-ITO; RISK Solutions and ITG-FIT). 

Job Title:

Developer

Date:

25-Mar-2024

Department:

FRESH – IHC ETL

Location:

Global Infocity – Chennai

Business Line / Function:

IHC ETL

Reports to:

(Direct)

Grade: (if applicable)

(Functional)

Number of Direct Reports:

0

Directorship / Registration:

NA

Position Purpose

The Senior Developer will be a part of the ISPL  Chennai – IHC ETL projects team.  The developer position will primarily work on Informatica ETL tool ,Unix ,Autosys and DB  

Responsibilities

  • Developed using Apache Spark & Python Structured data API / Library.
  • Strong hands on experience in Processing Framework – Spark 2.x/3.x (Core, structured API,Streaming, MLLib) Language & Package – Python (Scripting & PySpark), Scala using Spark API,Unix Shell, SQL Query (basic & advanced)
  • Knowledge on Streaming Platform – Apache Kafka, Apache Nifi, Spark Streaming, Flink, Storm
  • Extensive hands on experience in designing, building and executing data pipeline using ETL/ELT tool Apache Nifi (Source, Extraction and Processors & Sink modules).
  • Big Data Hadoop – Hortonworks HDP 3.1.x & core Components covering data engineering, management, operation stack
  • Big Data Hadoop – Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage – HDFS (File Format – Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning) , Data Lake (Object Store),
  • Expertise in RDBM solutions (Postgres & Oracle)

Technical & Behavioral Competencies

•            Minimum 6 to 10 years hands-on experience with PySpark ,Kubernetes,Docker

•            Strong technical expertise PySpark ,Kubernetes,Docker and be in position to handle all technical difficulties

•            Strong in designing (data warehousing) concepts.

•            Good working knowledge in Unix/Ubuntu ( should be able to write wrapper scripts )

•            Capable of tuning the code to handle the huge data volume.

•            Responsible for translating/understanding the functional requirements so as to meet the specified technical requirements.

•            Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied

•            Involved in coding, testing, implementing, debugging and documenting the complex programs.

•            Involved in creating proper technical documentation in the work assignments.

•            Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process.

•            Ensures that all the standard requirements have been met and is involved in performing the technical analysis.

•            Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements.

•            Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming.

Behavioral Competencies

·       Excellent verbal and written communication skills.

·       Conduct meetings with global stakeholders, prepare minutes & summaries.

·       Assertiveness, Negotiation, Proactivness & Prioritization skills are important.

·       Discipline in documenting, following-up on issues and changes.

·       Experience in interacting with global stakeholders and independently managing discussions.

Specific Qualifications (if required)

Skills Referential

Behavioural Skills: (Please select up to 4 skills)

Communication skills – oral & written

Ability to synthetize / simplify

Attention to detail / rigor

Organizational skills

Transversal Skills: (Please select up to 5 skills)

Analytical Ability

Ability to understand, explain and support change

Ability to develop and adapt a process

Ability to manage / facilitate a meeting, seminar, committee, training…

Ability to develop and leverage networks

Education Level:

Bachelor Degree or equivalent

Experience Level

At least 7 years

Other/Specific Qualifications (if required)

Additional knowledge on reporting tools is advantage

Offers you may be interested in