Hadoop PySpark Developer Job at Purple Drive, Pittsburgh, PA

VnJ4YUVEa09WMU05eFBTeExFQTRjMG42
  • Purple Drive
  • Pittsburgh, PA

Job Description

Job Title: Hadoop PySpark Developer
Location: Pittsburgh, PA (Onsite)
Experience Level: 8+ years preferred

We're hiring a Hadoop PySpark Developer to work onsite in Pittsburgh. You'll be using tools like Spark, Python, Hive, and Kafka to build and manage data pipelines on the Cloudera Data Platform. The role involves working with large datasets, building ETL solutions, and making sure everything runs smoothly and efficiently. You'll also work with other teams to understand data needs and help deliver reliable solutions.

Roles & Responsibilities

  • Build ETL pipelines using Spark, Python, Hive, and Kafka
  • Understand business requirements and turn them into technical tasks
  • Work with data analysts and business users to gather data needs
  • Create design documents and source-to-target mappings
  • Make sure data pipelines run on time and handle large volumes
  • Improve performance of Spark jobs and existing pipelines
  • Set up data governance and ingestion frameworks
  • Help with production deployments and fix issues when needed
  • Support data migration and rebuilding history data
  • Review code and test cases, and help solve technical problems
  • Work independently and coordinate with global teams
  • Bring in ideas and best practices to improve solutions

Key Skills

  • Apache Spark (PySpark)
  • Python
  • Hive
  • Kafka
  • Cloudera Data Platform
  • Linux/Unix
  • Impala
  • Big Data tools
  • Cloud technologies
  • Performance tuning
  • Data governance
  • Good communication and teamwork

Job Tags

Full time,

Similar Jobs

Cracker Barrel

Loss Prevention Agent Job at Cracker Barrel

 ...protecting the assets of the Distribution Center (DC) and Ecommerce by preventing theft, ensuring safety, and enforcing security protocols within...  ...receivers, and operational processes to identify shrinkage or loss risks (including ecommerce).+ Perform routine security... 

Lensa

Data Scientist Job at Lensa

 ...via Lensa today! Experian is a global data and technology company, powering opportunities...  ..., uncover and prevent fraud, simplify healthcare, create marketing solutions, and gain...  ....com, Inc. Job Title: Data Scientist Job Location: 475 Anton Blvd., Costa... 

Walmart

Janitorial Associate Job at Walmart

 ...premiums may apply if applicable criteria are met.Employment Type: **Part-Time**Available shifts: **Mid-Shift, Closing**Location**Walmart Supercenter #1899**6250 VALLEY SPRINGS PKWY, RIVERSIDE, CA, 92507, USJob OverviewJanitorial associates ensure customers have a... 

Apple

Design Verification Engineer Job at Apple

Design Verification Engineer**Cupertino, California, United States****Hardware****Summary**Posted: **Jul 17, 2025**Role Number: **200572691**Does making the next great technology product excite you? Imagine what you could do here. At Apple, our new ideas have a way... 

NANA Regional Corporation

Ramp Service Agent (Seasonal ANC Airport Location) Job at NANA Regional Corporation

Ramp Service Agents (RSA) at Pegasus Aviation Services are responsible for marshalling aircraft in the gate, loading, unloading and sorting...  ...observing, inspecting, monitoring, and screening all passengers, airport personnel and crew regularly to detect any potential...