Introduction to Big Data and Hadoop
Instructor
Engr. Palash Gupta
Overview:
Introduction to Big Data and Hadoop’ is an ideal course package for individuals who want to understand the basic concepts of Big Data and Hadoop. On completing this course, trainees will be able to interpret what goes behind the processing of huge volume & variety of data as the industry moves from relational database analytics consuming high processing time to real-time analytics with extensively expedited processing speed. In addition, the trainees would be able to relate when it is efficient designing solution on top of Big Data and how it is designed to store large data set in distributed mode using a cost-effective model.
Learning Outcomes:
The course focuses on the basics of Big Data, Hadoop & Spark explaining both distributed storage and computing. It is a blend of both theory and hands-on demonstration & exercises which will give the trainee a working level experience about the operational process and associated benefits of Big Data.
Evaluation Criteria:
- Assignments 20 Marks
- Mid Term Exam 30 Marks
- Final Exam 50 Marks
Big Data Hands-on Training Syllabus:
| Week | Lesson | Topics | Hours | Type |
| 1 | Introduction to Big Data & Hadoop | • What is Big Data? • How Big is Big Data? • What are we trying to solve in Big Data? • Types of Data Structure • Hadoop System Principle • History of Hadoop • Comparison with RDBMS • Hadoop Eco System • Hadoop Distribution • Supported Operating System, Hardware and Resources |
2 Hours | Theory |
| 2 | Understanding Hadoop HDFS & Map Reduce | • HDFS Concept • HDFS Architecture • Introduction to Map Reduce • Working Methodology of Map Reduce |
2 Hours | Theory |
| 3 | Basic Hadoop Configuration, Setup, Administration and Command Reference | • Design a Hadoop Cluster • Procedure setting up a basic Hadoop Cluster • Setting Up a Hadoop Cluster • Basic Administration of Hadoop • Basic Command Reference |
2 Hours | Theory & Practical |
| 4 | Understanding Spark Essential, Architecture | • Introduction to Apache Spark • Sparck Architecture • Introduction to Spark RDD, Dataset, DataFrame and DAG • Introduction to Spark Component • Understanding Spark Execution Model • Why Spark where we have Hadoop? |
2 Hours | Theory |
| 5 | Spark Configuration, Administration & Setup | • Design a Spark Cluster • Procedure setting up a basic Spark Cluster • Setting Up a Spark Cluster • Basic Administration of Spark • Understanding of how Spark runs our job |
2 Hours | Theory & Practical |
| 6 | Fundamental of Python and Shell Scripting | • Introduction to Python • Installing & utilizing New Python Package • Introduciton to Shell Scripting • Running a Python Script with a Sheel Script in Shedule |
2 Hours | Practical |
| 7 | Programming with HDFS & Spark | • Accessing HDFS Files using Python • Loading HDFS Files in Spark • Running basic transformations and actions in Spark • Understanding of how Spark runs our job with an example |
2 Hours | Theory & Practical |
| 8 | Practical Experiment with a real-life problem | • Understand the problem • Desing the solution • Implement the solution • Q&A |
2 Hours | Practical |
 |
 |
 |
 |
-
- Course Brochure
- Instructor
- Participant List
- Class Schedule
- For Assignments:
- Assignment 1 & 2:
- Assignment 3:
- Download Presentations Slide and Q&A
-
SL File Name URL 1 Installing_Cent_OS7_in_Virtual_Box 
2 Big Data Hands-On Training_vim Basic 3 Big Data Hands-on Training_Important Linux Administrative commands 4 Download Putty 5 Pycharm-Community-2017.3.3 6 VirtualBox-5.2.4-119785-Win 7 Download Python-3.5.2 8 Download jdk-7u79-linux-x64 9 Download spark-2.0.0-bin-hadoop2.7 10 Download Hadoop-2.7.1 11 CentOS-7-x86_64-DVD-1708
- Hadoop Installation and Configuration Instruction
- Checking Attendance Status
- Checking Assignment Status
- First Exam Result
- Class Videos
- Photo Gallery
- Zoom Webinar user Instruction
- Show Final Result
- Download Certificate
List of Trainees who successfully completed the course
| No | Name | Institute | Country | Quiz (50 Marks) | Assignment (50 Marks) | Total Score (100 Marks) | Download Certificate |
| 1 | Kasun Vimukthi Chamara | University of peradeniya, sri lanka. | Sri Lanka | 34.00 | 50.00 | 84.00 |  |
| 2 | Md. Rakib Hassan | BAU | Bangladesh | 39.00 | 43.00 | 82.00 | |
| 3 | Mithun Kumer Ghose | Janata Bank Ltd | Bangladesh | 35.00 | 45.00 | 80.00 | |
| 4 | Sifat Ul Alam Nabil | Ahsanullah University of Science & Technology | Bangladesh | 31.00 | 47.00 | 78.00 | |
| 5 | Md. Rashid Al Asif | University of of Barishal | Bangladesh | 28.00 | 50.00 | 78.00 | |
| 6 | Md Mahmud Hasan | Agrani Bank Limited | Bangladesh | 26.00 | 50.00 | 76.00 | |
| 7 | Anita Sharma | Institute of Engineering, Tribhuwan University | Nepal | 35.00 | 40.00 | 75.00 | |
| 8 | Vishal Gewali | Nepal Police | Nepal | 33.00 | 42.00 | 75.00 | |
| 9 | Dinelka Dilshani Karunarathna | Faculty of Engineering, University of Peradeniya | Sri Lanka | 24.00 | 50.00 | 74.00 | |
| 10 | Md. Abdulla Al Mamun | Begum Rokeya University, Rangpur | Bangladesh | 34.00 | 40.00 | 74.00 | |
| 11 | Simon Das | Ahsanullah University of Science and Technology | Bangladesh | 35.00 | 37.00 | 72.00 | |
| 12 | Md. Ashraf Uddin | LankaBangla Securities Limited | Bangladesh | 31.00 | 40.00 | 71.00 | |
| 13 | Nawaraj Prasain | Madan Bhandari Memorial College | Nepal | 30.00 | 40.00 | 70.00 | |
| 14 | Pubudu Udara Premathilaka | University of Peradeniya | Sri Lanka | 28.00 | 42.00 | 70.00 | |
| 15 | Suneth Samarasinghe | University of Peradeniya | Sri Lanka | 36.00 | 32.00 | 68.00 | |
| 16 | prativa karki | Nepal Doorsanchar Company Limited | Nepal | 29.00 | 37.00 | 66.00 | |
| 17 | Sudam Wageesha Kalpage | University of Peradeniya | Sri Lanka | 31.00 | 34.00 | 65.00 | |
| 18 | Joydeb Kumar Sana | Bangladesh University of Engineering and Technology | Bangladesh | 24.00 | 40.00 | 64.00 | |
| 19 | Ratul Roy | Chittagong University of Engineering and Technology | Bangladesh | 23.00 | 40.00 | 63.00 | |
| 20 | Ayeshan Malinda Wimalasena | University of Peradeniya | Sri Lanka | 33.00 | 30.00 | 63.00 | |
| 21 | Nasir Ahmed Bhuiyan | Bangladesh University of Engineering and Technology (BUET) | Bangladesh | 21.00 | 40.00 | 61.00 | |
| 22 | Ramesh Kumar Pudasaini | Softwarica College of IT and E-commerce | Nepal | 21.00 | 40.00 | 61.00 | |
| 23 | Kiran Rana | Softwarica College of IT and E-commerce | Nepal | 21.00 | 40.00 | 61.00 | |
| 24 | Hiruna Samarakoon | University of Peradeniya | Sri Lanka | 29.00 | 32.00 | 61.00 | |
| 25 | D. L. Chamath Amarasinghe | University of Peradeniya | Sri Lanka | 26.00 | 34.00 | 60.00 | |
| 26 | Bibek Bajagain | Himalaya College of Engineering | Nepal | 33.00 | 27.00 | 60.00 | |
| 27 | MD SHIBLI MOLLAH | FDS SERVICES BANGLADESH | Bangladesh | 26.00 | 32.00 | 58.00 | |
| 28 | Md. Sabbir Al Ahsan | Chittagong University of Engineering and Technology | Bangladesh | 28.00 | 30.00 | 58.00 | |
| 29 | MD. Faisal Ahmed | Ahsanullah University of Science and Technology | Bangladesh | 28.00 | 30.00 | 58.00 | |
| 30 | Md. Mahamudur Rahman | Techno Crete Ltd | Bangladesh | 15.00 | 42.00 | 57.00 | |
| 31 | Pranto Biswas | CUET | Bangladesh | 25.00 | 30.00 | 55.00 | |
| 32 | Tahmima Chowdhury | Janata Bank Limited | Bangladesh | 24.00 | 30.00 | 54.00 | |
| 33 | Nasim Ahmed | Chittagong Veterinary and Animal Sciences University, Chittagong | Bangladesh | 23.00 | 30.00 | 53.00 | |
| 34 | Md Asaduzzaman | GETCO Telecommunicatio Ltd | Bangladesh | 33.00 | 20.00 | 53.00 | |
| 35 | Prasenjit Das | BRAC University | Bangladesh | 21.00 | 30.00 | 51.00 | |
| 36 | Saroj Khadka | World Fiber Net Pvt. Ltd. | Nepal | 25.00 | 25.00 | 50.00 | |
| 37 | Thilini Deshika | University of Peradeniya | Sri Lanka | 26.00 | 24.00 | 50.00 | |