- Melbourne C B D
- $120000.00 - $130000 per annum
- Job Type
- Melissa Haddah
- over 1 year ago
The Big Data Engineer will expand and optimise our clients' data and data pipeline architecture, as well as optimise their data flow and collection for cross functional teams.
Your responsibilities include:
* Build robust, efficient and reliable data pipelines consisting of diverse data sources to ingest and process data into Azure
* Design and develop real time streaming and batch processing pipeline solutions* Assemble large, complex data sets that meet functional / non-functional business requirements.
* Design, develop and implement data pipelines for data migration & collection, data analytics and other data movement solutions.
* Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
* Build DevOps pipeline
* Work with data and analytics experts to strive for greater functionality in our data systems
Essential skills and experience:
* Proven working experience as Big Data engineer for 2+ years preferably in building data lake solution by ingesting and processing data from various source systems
* Experience with multiple Big data technologies and concepts such as HDFS, Hive, MapReduce, Spark, Spark streaming and NoSQL DB like HBase etc
* Experience with Microsoft Azure
* Hands-on experience in Big Data ETL tools like Informatica big data management or Talend big data management will be a huge plus
* Experience in one or more of Java, Scala, python and bash.
* Ability to work in team in diverse/ multiple stakeholder environment
* Experience in working in a fast-paced Agile environment
For more information call Melissa Haddad on 9236 7732