I am a senior BigData developer with more than five years of experience in Java and Data development. I am used to agile software development methods and to work independently. I like to teach junior developers and leading teams by example. Because of my broad knowledge in various technologies I am often the one who develops proof of concept applications for our customers. In my spare time, I am exploring new technologies, work on GitHub projects and take pictures.
- 5+ years of experience in development and architecture with key emphasis on the trending Big Data Technologies - Spark, Flink, Hadoop, Kafka, HBase, Hive
- Good Knowledge of Big Data and Real-time/Offline Architecture
- Architect, design & develop Big Data Solutions practice including set up Big Data roadmap, build supporting infrastructure and team to provide Big Data
- Architecting and implementing Portfolio Recommendation Analytics Engine using Hive, Oozie, Spark SQL, Spark MLlib and HBase
- Excellent understanding of Hadoop/Spark architecture and underlying framework including storage management
- Familiar with JAVA, JVM, Scala, Zookeeper, Git, SVN, Maven, Gradle, uDeploy
- Strong ability to tackle high performance, scalability, reliability issues
- Comfortable with learning cutting edge technologies and applications to greenfield projects
Education
- 2013 - 2016
-
Zhejiang University
Computer Science and Technology
Professional Experience
06.2018 - 11.2019
06.2018 - 11.2019
Senior Big Data Developer (Manager)
Citi Group - CSTC (Shanghai)- Provided architect responsibilities including roadmaps, leadership, planning, technical innovation, security, and IT governance, etc
- Developed technical presentations, proposals and performed presentations
- Evaluated new technologies; Executed proof-of-concepts and developed specialized solutions
- Designed and developed a real-time computation platform using Flink, Kafka and HBase
- Presented and persuaded the design architecture to various stakeholders
03.2017 - 05.2018
03.2017 - 05.2018
Senior Big Data Developer (Team Lead)
Manulife (Chengdu)- Focused on the whole architect and development of this Recommend System project as the founder, application architect and technical leader in the Big Data team
- Built the platform with Cloudera Manager, including Zookeeper, Hadoop, Yarn, Spark, Hive, HBase and Kafka etc.
- Used Hive to build offline Data Warehouse, and Kafka and HBase as the real-time system
- Used Spark MLLib to implemented recommendation system and user-loss prediction model
- Provided technical leaderships and governance of the big data team and the implementation of the solution architecture in Big Data
- Used Jenkins pipeline for Continuous Deployment and Jira for Continuous feedbacks
07.2016 - 03.2017
07.2016 - 03.2017
Senior BigData Developer
Yaowang Technology (Chengdu)- Built a user data analysis platform with a data analyst
- Implemented architectures for all the ETL data loads coming in from the source system which were then loaded into the data warehouse
- Built, managed and continuously monitored the Hadoop cluster through Cloudera Manager
- Provided expertise on batch & stream with Hadoop, Kafka and Spark
- Caught up with emerging tools and technologies and recommend
07.2014 - 06.2016
07.2014 - 06.2016
Senior Java Developer / BigData Developer
SAIC Motor Corp - Saike Electronic Commerce Co., Ltd. (Shanghai)- Designed and developed an online mall system
- Designed and developed a high performance, robust and secure Big Data architecture based on Cloudera Manager ecosystems
- Performed data cleaning and data manipulation activities using Hadoop
- Designed and developed an online mall product recommendation system