Job Specifications
Location : Charlotte,NC;Plano,TX;
Must Have Technical/Functional Skills
Primary Skil: Amazon Data Engineer
Secondary: AWS Data Engineer, Amazon S3, Shell Scripting, Autosys
Experience: Minimum 10 years
Roles & Responsibilities
Key Responsibilities
Design, develop, and execute Data Pipelines and test cases to ensure data integrity and quality.
Develop, implement, and optimize data pipelines that integrate Amazon S3 for scalable data storage, retrieval, and processing within ETL workflows.
Leverage Amazon S3 for data storage, retrieval, and management within ETL workflows, including the ability to write scripts for data transfer between S3 and other systems.
Utilize Amazon S3's advanced features such as versioning, lifecycle policies, access controls, and server-side encryption to ensure secure and efficient data management.
Write, maintain, and troubleshoot scripts or code (using PySpark, Shell, or similar languages) to automate data movement between Amazon S3 and other platforms, ensuring high performance and reliability.
Collaborate with cross-functional teams to troubleshoot and resolve data-related issues, utilizing Amazon S3 features such as versioning, lifecycle policies, and access management.
Document ETL processes, maintain technical documentation, and ensure best practices are followed for data stored in Amazon S3 environments.
Familiarity with Hadoop or Spark is often preferred.
Validate HiveQL, HDFS file structures, and data processing within the Hadoop cluster.
Strong analytical and troubleshooting skills.
Excellent communication for collaborating with developers and stakeholders.
Knowledge in Metadata dependent ETL process and batch/job framework
Tools & Skills
Amazon S3: Data storage, retrieval, and management; scripting for ETL data transfer; advanced features including versioning, lifecycle policies, access controls, and server-side encryption; automation of data movement using Python, Shell, or similar languages; troubleshooting and collaboration for data-related issues; documentation and best practices for ETL processes.
PySpark
SQL
Oracle
Domain: Banking knowledge, Payment’s knowledge preferred.
Environment: Cloudera Platform
Concept: Cloud Storage, Amazon S3, AWS, Data Warehousing, Data Transformation, ETL/ELT, Data Quality.
Salary :120,000-130,000/Per Annum
About the Company
NuStar Technolgies is a full service IT consulting company founded in 2013, specializing in IT Training, Staffing and Consulting. NuStar helps companies achieve business objectives by combining top-tier talent with proven processes and leading technology. Our commitment to our customers is reflected throughout our approach. We are intensely selective in our hiring practices and our customer focus is reflected in all areas of our organization. We support our customers and partners with highly experienced and Certified Profess...
Know more