CIS 612 Big Data and Parallel Database Processing Systems



Professor: Dr. Sunnie S. Chung
Office: ZOOM Meeting (FH 222: Not Available this semester)
Office Hours: Tues, Thurs 1:30PM - 3:30PM
Send me an email during the office hours


  • Class Syllabus
  • Big Data Projects
  • Lab Assignments
  • Class Lecture Notes

  • Class_Notes: Introduction to Big Data Technologies and Big Data Processing Systems on Cloud
  • Class_Notes: Web Application With Big Data Processing
  • Class_Notes: Big Data and Processing Techniques as Semi-Structured Models
  • Class_Notes: Big Data and Processing Techniques as Unstructured Texts
  • Class_Notes: Transaction and Concurrency Control
  • Class_Notes: Parallel Big Data Processing Systems on Cloud: NOSQL/NewSQL Systems

    • CIS 612 Announcement:


    15. Aug 25, 2020:

    TA Information
    TA: Yixi Luo

    Email: luoyixi.cn@gmail.com
    Office Hours: Mon, Wed 11:30 PM - 1:30 PM
    Location: ZOOM Meeting (FH305: Big Data Analytics Lab is not available)

    ZOOM Meeting ID: 670 584 2410
    Passcode: 191094

    If you have questions in Labs or grading your Labs, Send an email to TA to See During TA's Office Hours to Schedule a Zoom meeting
    For Any Questions on the Installation of SQL Server, Send email to TA


    8. Aug 25, 2020:

    The Output of each lab is your report in Doc file that shows your screen captures of, for example, for your Prelab: Lab0 outputs, your SQL Server Management Studio showing that your Codes (SQL/DDL/DML/Stored Proceures, etc) and the server returned the correct results.
    Each of your screen capture must show your results returned in the SAME window of your system to prove that your lab is done in your system correctly !!

    Lab Submission:
    1. Submit your report with source codes showing the Screenshots of your execution steps and correct outputs for grading. (in Class Submission is no longer needed for a Remote Class)
    2. Submit your Zip file that includes your report in .doc file and all the Source files, and output files on Blackboard for a timestamp and as a proof.

    3. If You did an Extra Credit Part, Mention about What Part is Done for Extra Credit at the Heading of the Front Page of Your Report in Bigger and Bold Font !



    12. Aug 25, 2020:

    For the installation guides of SQL Server in detail, see the Lab Section of CIS430/530 !
    Lab Section of CIS430/530

    Read the Installation Guides FIRST before starting downloading ! See the CIS430/530 Lab section for More details !

    7. Aug 24, 2020:
    For Mac, Either Go to Azure Cloud to Create SQL Server Instance Service
    Or
    Install VM(Virtual Machine) and Window 10 on Mac and then Install SQL Server on the Window VM:
    1. Install Mac OS X Host from
    Oracle VM
    Oracle VM Download OR VM Fusion
    2. Install Window OS on the VM on your Mac then
    3. Install SQL Server on Window VM.

    For more installation guides of SQL Server in detail, see the FAQs in the Lab Section of CIS430/530 !
    Lab Section of CIS430/530

    Read the Installation Guides FIRST before starting downloading ! See the CIS430/530 Lab section for More details !


    5. Aug 24, 2020:
    2016, 2019 MS SQL Server Packages Do not Have the Client Software - SQL Server Management Studio (SSMS). Download it seperately from the site specified and install it after Installing a SQL Server.
    Read the Instructions in the Lab0 Section First for This !

    For more guides on installation of SQL Server, see the Lab Section !

    4. Aug 24, 2020:
    If you can't install MS SQL Server, Go to the Oracle site below to download and install Oracle database server 11g Standard.

    http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

    Refer Online Oracle documents Oracle9i/Database Installation Guide



    3. Aug 24, 2020:

    Lab Assignment 0 : Prep for Lab Assignments

    Download Visual Studio, MS SQL Sever and its Client - MS SQL Management Studio (SSMS) and Install on Your Laptop

    From May 2019, Microsoft changed their platform to provide free download MS Software for CSU Students !

    NEW INSTRUCTION from May 2019 !

    How to Access MS Azure site to Download VS, MS SQL Server and Other Software

    1. Go to www.portal.azure.com
    Azure Potal Site
    2. Sign in with your CSU ID # and password.
    3. In the box on top, where it says "Search resources, services and docs", type "Education".
    4. Select "Education (preview)" from the list.
    5. Select "Software" from the list in the upper left corner.
    6. You should now be able to select from a variety of software, including Windows 10, Windows Server, Visio, Visual Studio, SQL Server, Project, Access, etc.
    7. Lots of documentation is under the tab marked "Learning".


    For any questions on this site or creating your account in this site, send email to TA or Jackie !


    See Lab0 Section for Instructions to Download and Install SQL Server and Visual Studio on Azure Potal Site

    SQL Server Installation Guide: See Lab Section for the Detail :br>
    1.Download and Installation of MS SQL SERVER Package : Read Instructions Carefully Below !

    Any SQL Server Standard/Developers/Enterprise Version with Service Packs recommended
    Make Sure to Install a Matching Version of Visual Studio first before installing SQL Server (Please see installation Guide in the Lab section for the detail)
    Read the installation Guides in the Lab0 Section FIRST before Starting to download !
    2. After everything is set up correctly, Run SQL Server Management Studio - a Client for your SQL Server (It is Under Microsoft SQL Server of your Start menu of your Window) to Play with Your SQL Server
    3. If your SQL Server is not running (You don't see the green arrow on your server name on the Top Left corner of your SQL Server Management Studio), Start the server manually in your SERVICE list - Search with "Service" to find the list. (Read the General Installation Instuctions for this in the Lab0 Section !)


    2. Aug 24, 2020

    For New Students, You Have to Create Your Account in Azure Potal site to Download MS Software -- Visual Studio, Window SQL Server

    How to Create Your Account to Login the Azure Potal Site to Download MS SQL Server

    1. Go to www.portal.azure.com
    Azure Potal Site
    
The students MUST use a vikes.csuohio.edu or csuohio.edu email address!

Note that the students should use CSUID@vikes.csuohio.edu, not f.lastname@vikes.csuohio.edu.

So, for example, 2345678@vikes.csuohio.edu, rather than m.jones@vikes.csuohio.edu.

Education Hub

This is where your students can download their software.

1.       Please clear the cache, cookies and browsing history. Then using Internet Explorer or Microsoft Edge, please enable the In-private browsing by pressing CTRL + Shift + P on the Keyboard or, for Chrome CTRL + Shift + N.

2.       Go to https://azureforeducation.microsoft.com/devtools

Azure Dev Tools for Teaching
This site uses cookies for analytics, personalized content and ads. By continuing to browse this site, you agree to this use. Learn moreat azureforeducation.microsoft.com
then click the blue sign in button.

3.       Enter your email address then hit Next.

4.       Enter the password then Sign In.

5.       Once signed in, it should take you to the Student Verification page. Please enter your school email address twice and a verification link will be sent to your email.

6.       Copy and paste the link to a new tab then hit continue.

7.       Please agree to the Terms and Conditions.


After completing the above mentioned steps, you should be able to access the Azure portal. 
    
If you were directed on the Azure Home page, please type in “Education” on the search box then select “Education” from the search results.

    Send an email to TA or Dr. Jackie Woldering (j.woldering@csuohio.edu) for any questions


    3. Aug 25, 2020:
    If you don't want to wait, Go to the Oracle site below to download and install Oracle database server 11g Standard.

    http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

    Refer Online Oracle documents Oracle9i/Database Installation Guide

    2. Aug 25, 2020:

    Lab Assignment 0 : Prep for Lab Assignments

    2) Download and Install a SQL SERVER package
    2017 or 2019 SQL Server Standard (Developers/Enterprise) Version with Service Packs recommended
    Make Sure to install a matching Version of Visual Studio first before installing SQL Server (Please see installation Guide in the Lab section for the detail)
    Read the installation Guides FIRST before Starting to download !
    3) After everything is set up correctly, Run SQL Server Management Studio - a Client for your SQL Server (It is Under All Programs of your start menu of your Window) to Play with your SQL Server
    4) If your SQL Server is not running (You don't see the green arrow on your server name on the top left of your SQL Server Management Studio), start the server manually in your SERVICE list. (Read the general installation guides for this in the Lab Section !)



    1. Aug 25, 2020:
    The class webpage can be reached from my webpage:
    Class Webpage
    If you have a trouble to display the webpage correctly with MS Internet Explorer, open it with Google Crome

    Final Exam schedule for Fall 2020:

    Tuesday Dec 8, 4:00PM - 6:00PM
    Check the Last Day to add and Drop and the Final exam schedule and the Semester Schedule below
    University's Official Academic Calendar for the Semester and the Final Exam schedules



  • Class Syllabus

    • Big Data Projects

    Project List is posted.
    Project List
    Project Guideline and Important dates !
    Project Task Timeline

    Important Dates for Project: (Tentative)
    Proposal Submission by Nov 7 !
    Status Report Submission by Nov 20 !
    Presentation Starts on Nov 30 !
    Final Project Report by Dec 4th !


    Lab Assignments


  • Installation and Setting Up Guides for Labs

    For the installation of SQL Server in detail, see the Lab Section of CIS430/530 !

    Lab Section of CIS430/530

    Read all the Installation instructions FIRST before starting downloading ! See the CIS430/530 Lab section for More details !

    How to Use the Client SSMS: How to Use SSMS


    How to Install SQL Server on Mac:
    1. Install Mac OS X Host VM(Virtual Machine) from
    Oracle VM
    Oracle VM Download OR VM Fusion
    2. Install Window OS on the VM on your Mac then
    3. Install SQL Server on Window VM .

    If you want to install SQL Server on Linux, download and install PostgreSQL.
    PostgreSQL to download

    For more instructions of installation of a SQL Server on Mac in detail, see the Lab Section of CIS430/530 !

    If you don't have your own system to install a SQL Server, go to the Azure Cloud to create a free student account and create a SQL Server instance in Azure Cloud.

    For more instructions on creating a SQL Server in the Azure Cloud in detail, see the Lab Section of CIS430/530 !



    PrepLab: Required Only For Those Who Didn't Take CIS430/530

    PrepLab0: Creating Company Database and SQL Querying over it
    Tips For Creating Database Schema and Initial Loading
    The installation instruction part in this document is for 2014/2016 SQL Server although most of tips are still the same, you need to know when installing 2019 SQL Server.
    For the installation procedure of 2019 SQL Server, see CIS430/530 Lab Section.


    For Those who Doesn't Have Enough Background for Basic Database Subjects:
    CIS430/530 Course Site
  • Review on Basics of RDBMS and Complex SQL

    Class Note_0: Review :Elmaseri TextBook Chapter 4
    Class Note_0: Review :Elmaseri TextBook Chapter 5_Updated
    Examples of LOJ with Group By and Aggregate Functions
    Example Outputs of LOJ with Group By and Aggregate Functions


    For Those Who Want to Install MS Data Analytic Softwares:

    Guide for Installing MS SQL Server Data Tool (SSDT) - BI for Analysis Service and Importing Adventure Work Data WarehouseUpdated on Sep 20
    Instructions (ReadMe file) for Importing Adventure Works Data Warehouse 2014 to 2014 SQL Server


  • Lab Assignment 0

    Basics of HTML, CSS, URL and Java Script

    LabAssignment 0
    See CIS408 Class Lecture Notes for Tutorials on HTML, URL, Java Script




  • Lab Assignment 1


    LabAssignment 1 On Trigger
    Lab 1 Output Example:
    Example Codes (different trigger syntax) in SQL Server and an Output example for LabAssignment 1 On Trigger
    Example Codes and an Output Example for Lab 1 On Trigger
    Correct Lab 1 Output and MS SQL Server Trigger Syntax Example on DeptAuditRecord Table NEW POST !
    Correct Lab 1 Output Example on EmpAuditRecord Table NEW POST !

    FAQ for Lab1
    Compound Trigger Code Examples to avoid Table Mutation Errors in Oracle Server





  • Lab Assignment 2

    Lab2 on User Defined Type, User Defined Table Function to Process Unstructured Text Data, and Visualization of Geo Location Data Type on ArcGIS/Google Map/Leaf API

    LabAssignment 2

    For Those Who Did Part 2 of Lab2 in CIS611 as an Extra Lab, Create a Complete Inverted Index Table in the Slide 14 in Inverted Index Lecture Notes
    You don't need to Create a Linked List for Postings, You can include Postings in the Inverted Index Table.

    LabAssignment 2 Another Option on CLR User Defined Type, CLR User Defined Function, CLR Table Function and ArcGIS
    Example Procedure Instruction to Do Lab2 to Create CLR UDT and Table Valued Function

    Data:
    MasterAllUnionAddress.txt
    Crime Location data for Part 3
    NIJ (National Institue of Justices) Challenge Site

    CLR TVF Updated Info for 2017 SQL Server or higher

    Visualization Tools of GEO Spatial Data
    QGIS
    QGIS in Python

    Example to Visualize in QGIS
    Example of Lab2 GIS Data Processing in Python: Example Project to Guide How to Process Geo Spatial Data (generated from a machine) for Hot Spot Analysis From CIS 660 Project by Sarvesh Chande
    Lab2 GIS Data Processing Example of NIJ Geo Spatial Data Visulaization Using ArcGIS Map for Hot Spot Analysis NEW POST !
    Example of NIJ Geo Spatial Data Visulalization Using ArcGIS 3D Map for Hot Spot Analysis NEW POST !
    Example of GIS Data Processing in Java Script to Visualize in HTML and NIJ Data file in GeoJSON NEW POST !
    Example of Coordinate Conversion to Oregon North FIPS 3601 NAD83 NEW POST !
    Example of NIJ GIS Data Processing in a Plain Coordinate in Python for Clustering Analysis NEW POST !
    Example of NIJ GIS Data Processing in a Plain Coordinate in R for Clustering Analysis NEW POST !

    How to Create a Table with Bulk Insert with MS SQL Server
    Example of a Script to Create Multiple Tables with Bulk Insert in MS SQL Server
    How to Set Up ASP.NET with SQL Server
    Lab2 Guide: How to Debug CLR UDF, CLR UDT, CLR TVF

    Text Preprocessing Library in Python SpaCy:

    Liquistic Modules in Python SpaCy
    Lemmatizer in Python SpaCy
    Liquistic Modules for Tokenization, Stemming, Lemmatization in Python SpaCy
    How to Code Liquistic Modules like Lemmatizer in Python SpaCy
    Python Example for Basic Text Processing

    How to Connect to MS SQL Server in Python
    pyodbc: Examples of ODBC in Python
    Errors: Why Data Points are Mapped into Ocean?









  • Lab Assignment 3

  • Web Scrapping with DOM and XPath:

    Updated Lab3_1 on HTML Page Processing with DOM and XPath
    Infoplease site of State Union Addresses of US Presidents
    Correct page of Address of John Adams December 3 1799
    FAQ for Lab3_1 on Webscapping NEW POST !

    Extra Points if You Make it Automated for the Table Creation from your output file (CSV file) in your Database Server with CLR TVF/UDF Table Function (MS SQL Server) or Any Other Table Function/Collection (Oracle SQL Server) !


    XPATH Set Up with Others:

    Note that You don't Need to Do Any Extra Set Up to Use XPATH in your Java Script for HTML. All the RECENT Web Browsers Have XPATH Features in their Debugger by Default
    You May Need To Set UP XPATH for the Other Scripts/Languages to Use XPATH Methods in the Application Codes as below.

    Node.JS XPath Setup Guide and XML Parser in Java Script
    XPath Setup Guide for java Script
    XPath Setup with npm for Node JS in Java Script
    Setting Up DOM with XPath for Python


    Set Up DOM with XPath for HTML and XML:
    Java Script:
    How to Debug HTML with JAVA Script, DOM, XPath
    Java Script Debugger
    Node.JS XPath Setup Guide
    npm Node.JS XPath Setup Guide
    MDN Site for DOM with XPath in Java Script
    DOM with XPath Setup Guide for Java Script
    XPath Methods
    Google Crome XPath Helper Setup

    Python:
    Setting Up DOM with XPath for Python
    BeautifulSoup Documentation for HTML, XML for Python
    Webscapping with DOM, XPath for Python with BeautifulSoup (From the Stanford Class)
    Python XPath Guide

    Useful Site for MIME Type



  • XML Data Processing with DOM and XPath:

    Lab3_2 on XML Processing with DOM and XPath
    LabAssignment 3_2 - XML Input File

    Check the validity of XML Input. Add a proper XML doctype heading like in your XML input document and Validate your XML input and correct it if needed.

    LabAssignment 3 - DOM XML Parser Example Codes
    Lab3 Output Examples of Conversion Between XML and SQL Table
    Example of Stored procedure to Create SQL Tables from a Text file

    DOM XML Parser
    SAX XML Parser
    Lecture Note_5_5 On How DOM Parser and SAX Parser Work to Process XML data (Examples in Mobile App)
    Examples Conversion Between XML and Table
    FAQ on Lab3_XML and Lab3_JSON
    Wiki Data set in XML for your Project



  • JSON Data Processing:

    JSON Data Processing and Semi-Structure Database MongoDB
    Lab3_3: Yelp Business Data Processing with MongoDB

    Data Sets:
    Zip file for 100 business data JSON files (in valid JSON format)

    Zip file for 5 JSON files from Yelp data Challenge 2017 (They may not be in valid JSON format) Data is too big, can't put them here. Download directly from the Yelp site below !

    You can directly download the most recent data sets from Yelp site:
    Yelp Challenge Data Set

    Semi-structured Database with MongoDB:

    NoSQL Database Creation for Lab3_3

    Create Semi-Structured Database for the 100 Business Yelp data in JSON with MongoDB
    MongoDB:
    MongoDB Download
    MongoDB Enterprise Download
    MongoDB GUI Compass Download
    MongoDB Download and Installation
    Quick Set Up Instruction for MongoDB Download and Installation
    MongoDB Free Service

    MongoDB Intro
    MongoDB Site to start
    Mongo DB: Databases/Collections
    Mongo DB: CRUD Operations






  • Lab Assignment 4 on Inverted Index For EXTRA CREDIT !

  • Unstructured Data Processing:
    Building an Inverted Index from Unstructured Text Data
    Lab4 on Building an Inverted Index on Wiki Data Set
    How To Build Inverted Indexes:
    Class Note_18_1: Lecture Notes On Web Data Processing: Inverted Index in Google (Stanford)
    Example of an Inverted Index On Union Address Text Data

    Data Set and Tools:
    wiki Data set in XML and Html
    Wiki Text Data Sets Complete List and Tools

  • Web Application:
    Lectures on Web Application: See Server Side Programming Section
    Lab3 on Web App


    Lecture Notes for Database Programming with JDBC/ODBC:
    MySQL Applications Using Java & JDBC
    Database Programming with JDBC and Java
    Example of Stored Procedure in Database Server

    Example Codes for WebServer Side:

    PHP:
    Example Server Side Application Codes in PHP on WAMP Server:
    Trouble Shooting WAMP Server SetUp
    Lab3 Guide -- Building Web application with PHP and Mysql
    Lab Guide with Set Up instruction and Simple Example Codes for Building Web Application with PHP and MysqlNEW UPDATE !

    PHP Application Simple Code Examples: Search Music by Name
    source script of index.html
    source php script of search_musicapp.php
    source php script of search_musicapp1.php

    Simple Example Codes with PHP, Mysql and Bootstrap NEW UPDATE !
    Fourth Example Codes for Building Web Application with PHP and Mysql Calling StoredProcedureNEW POST !
    Second Sample Codes for Building Web Application with PHP and Mysql
    Third Sample PHP Codes for Building Web application with PHP and Mysql

    PHP Tutorials:
    Quick PHP Example Tutorial
    Database Programming with PHP
    Class Note: Database Programming Using PHP
    PHP Programming

    Building Web Service with JAVA in Eclipse with MS SQL Database Server and JDBC-ODBC:
    How to Set Up JDBC-ODBC in Eclipse to Connect to MS SQL Server
    Sample Codes to Build Web Application in Eclipse with JAVA/JDBC and MS SQL Server

    Building Web Service with ASP.NET with MS SQL Database Server and ODBC:
    How to Create Webserver site in local host in ASP .NET
    How to Create Webpage in ASP .NET
    Quick Tutorial: How to Create WebPage with Action in ASP .NET

    Sample Codes for How to Create Web Service in ASP .NET with SQL Server
    How to Connect to a Database in SQL Server From ASP .NET
    Set Up Example in SQL Server From Visual Studio in ASP .NET
    Sample Web Application with SQL Server and WebMatrix on ASP .NET FrameWork
    Example of Stored Procedure in Database Server
    Building Web application with SQL Server, GridView and MS IIS Manager in ASP .NET

    Java Script Framework - JQuery and Bootstrap:
    Quick JQuery and Bootstrap Tutorial
    JQuery Tutorial with Examples
    JQuery AJAX Tutorial with Examples
    Bootstrap Tutorial with Examples

    Java Script Framework - Angular JS
    Quick Angular JS Tutorial
    Angular JS Tutorial Site with Examples

    Node JS with MongoDB/PostgreSQL/SQL Server:
    Node JS:
    Node JS
    Node JS Download
    Node JS Package Manager Download
    MongoDB:
    Mongo DB
    MongoDB Download
    Set up Guide:
    Node JS with Mongo DB Setup Guide
    Node JS with PostgreSQL Setup Guide
    Sample Web Application Using Node JS with MS SQL Server and Mongo DDFrom a Master Project: Building a Recommendation System usng Amazon and Walmart Product and Review Data Sets


  • Extra Lab Assignment On Data Warehouse: For Those Who Has NOT Taken CIS611

    Guide for Installing MS SQL Server Data Tool (SSDT) - BI for Analysis Service and Importing Adventure Work Data WarehouseUpdated on Sep 20
    Instructions (ReadMe file) for Importing Adventure Works Data Warehouse 2014 to 2014 SQL Server
    Guide for Setting Up Adventure Work DW and Creating OLAP Cube
    Lecture Notes: Labs/Project Tutorials with DW and OLAP CUBE
    Fix for Empty Cube Errors






  • Lab Assignment 4_1 on Hadoop with MapReduce/ Project Set Up

    Lab4_1 on Installing Hadoop Updated with Extra Credit Task !
    For Those who Want to Set Up Your HDFS Cluster On EC2 Amazon Cloud, See the Cloud Section at the end of the Class Lecture Note Section for a Student Account.

    Hadoop Set Up Instruction Sites:
    Hadoop single node setup
    Hadoop Cluster setup
    Map Reduce Tutorial on Hadoop

    Data Set for MR Job:
    NASA HTTP Access Log File
    AWID:Wireless Network Server Log Data Set
    AWID Data Set Avaliable here as well:
    AWID: Wireless Network Server Log Data (around 20 GB zip)

    Lab Guides: Newest on the Top
    Hadoop Installation: How to Fix When Data Nodes are not Running (2018)
    Help Site on How to Fix When Data Node are not Running (2018)
    Procedure to How to Execute MapReduce in Eclipse to Run a Wordcount Job (2018)
    Procedure to Install Hadoop and Run a Wordcount Job on Window 2010 (2017)
    Procedure to Install Hadoop and Run a Wordcount Job on Window 2010 - Permission issue (2017)
    Procedure to Install Hadoop and Run a Wordcount Job on Mac (2017)
    Guideline4 for Passwordless SSH for Lab4_1 (2016)
    Guideline2 for Lab4_1 (2015)
    Guideline1 for Lab4_1 (2014)

    A good Instruction Site for Installing and Running Hadoop
    Installing Hadoop

    If you have a trouble installing Hadoop from the above site with not seeing the name node, You need to delete the temp files created in standalone mode and reformat the namenode.
    For setting up a passwordless SSH, see Ganesh's Lab4 below as well.
    Installing Hadoop and running a wordcount job by Ganesh VAVILAPALLI
    Video for Installing Hadoop shared by Prashant Patel
    Trouble Shooting Tips for Installing Haddop on VM
    Guideline3 for Lab4_1 on Mac (2013)


    MR Programming on Hadoop Lab:
    Lab4_2 on MR Programming on Hadoop:
    Implement Average Temp By Station in MR on Hadoop as in Lab Guide below. Follow the sample codes below.

    Simple MR Hadoop Lab Guides
    MR sample Codes - Average Temp By Zip Code
    MR Sample Codes - Sort By Station ID
    Zip file for station Data Set and
    Zip file for station Data Set and MR Codes







  • Lab Assignment 4_2 on NoSQL on Hadoop / Project Set Up

    NoSQL Systems on Hadoop Lab:

    Part 1: Due By the midnight April 12 !
    Part 2: Due By the midnight April 20 !
    Extra Credit By the end of May 4 !

    Lab4_2 on Using NoSQL Systems: Hive, PIG, Mongo DB, HBase, or Spark
    Example of Creating a Hive Table with Partitioning

    Note for MongoDB Join for Part2:

    There are no matching business id in the business ids from the list of 100 businesses and the review collection, so join results were empty.
    The reasons for this could be either 100 business data is too small to get any join result with the review data or the review data set in the new data set from Yelp 2019 has been changed. They keep changing or updating the Review data sets,
    so there is no matching id in the 100 business data set in the class web page which was collected in 2017. You are welcome to change one business id to have matching join results. Or Use the entire Business and Review json files in the new data sets from 2019 to have meaningful join results. The deadline has been extended to April 20.

    Data SetS:
    Video Game sales Data for HIVE/PIG
    JSON Files from Yelp Challenge
    NASA HTTP Access Log File
    AWID:Wireless Network Server Log Data Set
    AWID Data Set Avaliable here as well:
    AWID: Wireless Network Server Log Data (around 20 GB zip)

    Note that Hive has a bug that initialize your Name node whenever you start Hive again. You may lose your data so BACK UP Your Data ! Most of set up problems are coming from a Version Mismatch between conponents. It may not have documented correctly (remember these are open sources).


    NoSQL System Set Up and Basic CRUD Operations in Examples :

    MongoDB Join in Python Script as Client by Asanka K. Mananayaka
    Spark Basic Data Processing by Stacey Heideloff
    MongoDB Basic CRUD and Join Processing by Sankalp Godugu
    HBase Join with Hive by Paul Webster






    NOSQL System Set Up Guides for Lab4_2 (2018-2019)
    HBase with Hive Set Up guide by Paul Webster
    Hadoop/Cloudera/Hive Installation Guide

    NOSQL System Set Up Guides for Lab4_2 (2017)
    MongDB/Hive Installation Guide
    MongoDB/Hive Installation Guide, Permission Error Resolution
    MongoDB/Hive Installation Guide
    PIG/MongoDB Set Up Guide

    NoSql System Set Up Guides for Lab4_2: (2016)
    Hive Installation Guide
    MongoDB Installation Guide
    HBase and Cassandra Installation Guide
    HBase on Cloudera Setting up and Tutorial
    Spark Tutorial From Danielle Aring's Report
    VoltDB Commands and Stored Procedure Sample Codes

    Guides for Cloudrea:
    Cloudera Installation Guide

  • Cloud Computing Set Up

    You can create your Big Data Processing Infrastructure on Amazon Cloud for your Project
    Amazon Cloud Account for Students






  • Class Lecture Notes

    • Sections of the Lecture Notes:

  • Class_Notes: Introduction to Big Data Technologies and Big Data Processing Systems on Cloud
  • Class_Notes: Object Relational Mapping (ORM) for Big Data Processing Methods
  • Class_Notes: Web Application With Big Data Processing
  • Class_Notes: Big Data and Processing Techniques as Semi-Structured Models
  • Class_Notes: Big Data and Processing Techniques as Unstructured Texts
  • Class_Notes: Transaction and Concurrency Control
  • Class_Notes: Parallel Big Data Processing Systems on Cloud: NOSQL/NewSQL Systems




    • Class Lectures

  • Introduction to Big Data Technologies and Big Data Processing Systems on Cloud

    Class Note_1_2: Introduction to Big Data, Big Data Processing, and Big Data Processing Systems for Big Data Analytics
    Class Note_1_1: What is Big Data Analytics ? (in Two-Min Intro of Big Data Analytics)
    Overview of Big Data Processing Systems: NO SQL Systems


    Examples of Big Data Processing Projects Done in Big Data Lab:
    Intelligent Search Engine on Research Paper Repository and Wiki Pages
    Real-Time Twitter Sentiment Analysis System
    Intelligent Content Clustering Analysis on Research Paper Repository with Natural Language Processing
    Real Estate Property Information System with Recommemdation System
    Machine Learning Based Intelligent Full Text Search Engine with Natural Language Processing

    Important Subjects to Learn for Big Data Processing (From a SIGMOD Panel)

    Big Data, Data Scientist, Research in Big Data Analytics at CSU

    Examples of Big Data Processing Systems:
    Class Note_2_1: Example of Archtecture of Enterprise Big Data Processing Systems
    Class Note_2_2: Example of Enterprise Big Data Processing System on HDFS with the Data Warehouse Approach - LinkedIn Data Processing System
    Class Note_2_3: Example of Enterprise Big Data Processing System on HDFS: Yahoo PigLatin




  • Modern Database Programming for Web Applications with ORM (Object Relational Mapping)
  • Modern Database Programming for Web Applications

    Trigger:
    Class Note_6: Lecture Notes On Trigger
    Class Note_2_2: Example on How SQL Server Updates ROWs
    Trigger in MS SQL Server Specific Syntax
    Class Note_15_3: Compound Trigger Code Examples to avoid Table Mutation Errors in Oracle Server
    Class Note_15_4: Example of Instead of Triggerin SQL Server


    Object Relational Databases (ORDBMS)

    Class Note_15_2: Introdction To Object Relational DBMS
    Class Note_15_3: Lecture on Multimedia Databases
    Class Note_11: Lecture Notes on Processing Multimedia in Databases with JDBC and JAVA
    Example Codes to Insert and Retrieve Image Data in Oracle DB with JDBC and JAVA
    Example Codes to create, Insert and Retrieve Image Data in Oracle DB with InterMedia
    Example on How to Store and Retrieve Image Data in MySql with PHP


    Review on SP/Cursor/UDF/Table Function (From CIS430/530):
    Class Note_15: Database Programming: Embedded SQL/Dynamic SQL, Cursor, Stored Procedure, Table Function, UDF
    Class Note_15_1: Example of Stored Procedure in SQL
    Example of Table Function in SQL


    ORM: Object Relational Mapping

    Embedded/Dynamic SQL, Stored Procedure, Table Function, UDF, UDT (User Defined Type):

    Class Note_7: Lecture Notes On Current Database Programmability in C#

    Class Note_7_5: How ORM Features Can Be Used for Big Data Processing: Task Overview for Creating CLR Table valued Function with UDT (User Defined Type) in C# in .Net Framework

    CLR Examples:

    Class Note_7_1: Examples of CLR_UDF, CLR_UDT, CLR_UDA On Current Database Programmability in C#
    MSDN site For the Details on CLR
    Class Note_7_2: Sample Codes 2 to Create CLR Scalar Valued UDF in C# in .NET FrameWork
    Class Note_7-3: Sample Codes 3 to Create CLR Table Valued UDF in C# in .NET FrameWork
    Class Note_7_4: Sample Codes 4: How To Create CLR Table valued Function in C# in .Net Framework
    Class Note_7_5: Sample Codes 5 of How To Create CLR Table valued Function with UDT (User Defined Type) in C# in .Net Framework
    Linq Examples:
    Class Note_7-5: Lecture Notes On LINQ in C#
    MSDN: LINQ to SQL in C#
    MSDN: Starting with LINQ to SQL in C#
    Examples in Oracle SQL:
    Example of UDT in Oracle SQL Server

    Example of UDT and Collection (Table) in Oracle SQL Server




  • Web Application
    Web Application with JDBC/ODBC:

    If you don't know web application basics, See PHP Tutorial Section in the Class Notes of CIS 408 below
    Class Notes Section of CIS 408 Internet Programming Site

    Review on JDBC:
    Class Note_16_1: Building Web Applications with RDBMS and Java & JDBC
    Class Note_16_2: MySQL Applications Using Java & JDBC

    Review on PHP with ODBC:
    Class Note_9: Database Programming: Introduction of PHP
    Class Note_9_3: Chapter 15:Database Programming Using PHP
    Class Note_9_4: Web Programming Examples Using PHP

    PHP Examples on How to Process Form in PHP
    Examples of ODBC (Open Database Connectivity) in PHP

    Lecture Notes on Web Application with JDBC/ODBC:
    Class Note_16_3: Lecture Notes On Embedded SQL Using Java & JDBC
    Class Note_16_4: Example of Web Applications with MS SQL Server and Using C# & ODBC in ASP.NET


    Lectures on ODBC(Open Database Connectivity)/JDBC(Java Database Connectivity) to Connect To a Database Server:

    ODBC for Communication Between PHP Server and MySql Databse Server:
    For Example, MySql Server of WAMP Server to Listen on the Default Port:3308 on either on the Same Domain or Different Host within LAN

    PHP Basic Example Codes Using ODBC(Open Database Connectivity) mysqli to Connect to MySql ***
    All the Important ODBC MySqli Examples for PHP Server and MySQL Server Communication ***
    ODBC MySqli: Connection class ***
    ODBC MySqli: Statement class ***
    ODBC MySqli: Prepared Statement class ***
    ODBC MySqli for Stored Procedure in MySQL ***

    Three ODBC(Open Database Connectivity) APIs to Choose for PHP Server to Communicate To MySql Database Server


    From a Master Project: Building a Recommendation System usng Amazon and Walmart Product and Review Data Sets:
    Class Note_16_5: Example of Web Applications with MongoDB/ MS SQL Server Using Node JS/Angular JS


  • Web Application Basics

    Most of Lectures and Tutorials of Web Applications can be found here ! CIS 408 Internet Computing Class Notes

    HTTP Basics: How your Web Browser as a Client Communicates with a Webserver

    Web Application Architecture
    Lecture Note_5_1: Introduction to HTTP
    Example: From Form to HTTP
    HTTP Basics on MDN

    REST API:

    REST API Tutorials
    REST API Basic Guides
    MDN: XMLHttpRequest
    Lecture Note On Web Service with SOAP and REST API on Android

    Example Server Side Application Codes in PHP on WAMP Server:


    Building Web Service with ASP.NET with MS SQL Database Server and ODBC:
    How to Create Webserver site in local host in ASP .NET
    How to Create Webpage in ASP .NET
    Quick Tutorial: How to Create WebPage with Action in ASP .NET

    Sample Codes for How to Create Web Service in ASP .NET with SQL Server
    How to Connect to a Database in SQL Server From ASP .NET
    Set Up Example in SQL Server From Visual Studio in ASP .NET
    Example Codes 6: Web Application with Calling Stored Procedure in SQL Server and WebMatrix on ASP .NET FrameWork
    Example of Stored Procedure in Database Server
    Example Codes 7: Building Web application with SQL Server, GridView and MS IIS Manager in ASP .NET

    Building Web Service with JAVA in Eclipse with MS SQL Database Server and JDBC-ODBC:
    How to Set Up JDBC-ODBC in Eclipse to Connect to MS SQL Server
    Example Codes 5: to Build Web Application in Eclipse with JAVA/JDBC and MS SQL Server

    Processing Multimedia with Databases Server:
    Class Note_11: Lecture Notes on ODBC and PHP
    Example Codes to Insert and Retrieve Image Data in Oracle DB with JDBC and JAVA
    ODBC and C# ASP .NET
    Example on How to Store and Retrieve Image Data in MySql with PHP





  • Big Data Processing

    Motivations:

    Example of Enterprise Level Big Data Processing Infrastructures and Tasks From an Independent Study Report on Big Data Processing Infrastructure

    Examples of Data Pipeling for Query Execution Steps in Big Data Processing Systems

    Getting to Start with Examples of Intelligent Systems (AI) with Big Data, Big Data Processing and Big Data Analytics:

    LectureNotes_2_1: Overview of Question Answering System

    LectureNotes_2_2: Overview of IBM Watson: Question Answering System

    Lecture Notes_2_3: Overview of Sentiment Analysis System


    Examples of Big Data Projects:

    Example of Big data Project with Open IE (Information Extraction) Method: Building a Medical Question Answering System

    Project Example_1: Sharesci - Architecture of an Intelligent Document Search Engine: sharecsi
    Project Example_2: Overview of Intelligent Document Search Engine Using Machine Learning with Natural Language Processing

    wiki site of sharecsi for Documentation
    github site of an Intelligent Document Search Engine: sharecsi for Source Codes


    Big Data as Semi-Structured Data on Web Application Platforms on WWW

    See Web Application Basic on HTML, Java Script in the CIS 408 Internet Computing site
    CIS408 Class Lecture Notes Site

    Web Application Platforms on WWW

    Web Application Architecture

    World Wide Web Consortium (W3C) Reference Sites:

    HTML/HTML5
    World Wide Web Consortium (W3C) Standard for Markup Languages: HTML/HTML5/XHTML
    HTML/HTML5 Syntax by W3 Consortium
    XHTML1 XHTML2 is on the way !

    Big Data Format as Semi-Structured Data Model

    Semi-Structured Data Processing Techniques:


    1. Markup Languages:

    REVIEW:

    1. Markup Languages: HTML, XHTML, XML as Semi-Structured Data

    HTML/XHTML:

    HTML
    HTML Basic Summary
    Class Note_1_3: Overview of Web Application Components: HTML, XHTML, CSS, Java Script, CGI, PHP (Traditional Common Web Application Platforms) (
    XHTML Quick Tutorial
    Difference between HTML/HTML5 and XHTML

    HTML References:
    HTML Element List
    HTML Attribute LIST
    Global HTML Attribute LIST


    Webpage Processing with Semi-Structured Data Model:

    Semi-Structured Data Model - Document Object Model (DOM):

    DOM Lectures:

    Class Note_3_1: Example Picture of DOM Tree of Web Browser for HTML and Java Script
    Example of Document Object Model (DOM) for HTML and Java Script
    Class Note_3: Intro to Document Object Model (DOM) for HTML and Java Script

    Node Types of DOM Tree: (From W3C Schools Tutorial on HTML DOM)

    DOM Document Node Type for Document Element HTML
    DOM Element Node Types and Methods/Properties
    DOM Attribute Node Type and Properties
    DOM Text Node Type

    DOM References:
    Official Document Object Model (DOM) Specification for HTML/XHTML/XML by W3 Consortium

    HTML DOM Reference List
    HTML DOM Document and Method List
    HTML DOM Element and Method List
    HTML DOM Attribute and Method LIST

    MDN: HTML DOM Intro
    MDN: HTML DOM API
    MDN: Useful Example Sites of DOM Objects

    Code Examples:
    Example of DOM property InnerHtml
    Example of How to Find DOM Objects: getElementByTagName
    Example of How to Access DOM objects and values: childen element count
    Example of HTML DOM Object with button
    Element.className
    Example of How to Access DOM Atrributes: namednodemap: getnameditem
    Examples of HTML DOM Object creation
    Examples of HTML DOM Object Create and Append
    Example of How to Create Text Node
    How to create DOM Objects and Replace: replaceChild

    Form object and Submit Type
    How to Access Form object
    How to Create Form object
    Summary of Example Codes (Above) of DOM Objects: Input and Function to access value
    Summary of Example Codes (Above) on How to Use DOM objects and Functions to Manipulate DOM Objects


    Useful DOM Tree Query Features:

    Document.querySelector
    XPath in Java Script

    Class Note_1_4:HTML CSS Rules to Select Elements
    Class Note_2_1: Letcture Notes Java Script Basics (See Regular Expressions)
    JS RegExp
    RegExp: ignoreCase property of i modifier
    Document.querySelector with CSS Selector or Rex
    get elements by selector: querySelector with CSS Selector in Python
    XPath in Java Script
    Setting Up DOM with XPath for Python


    Webpage Processing Tool:
    DOM Example: Webscrapping with Python BeautifulSoup
    DOM Example: Python BeautifulSoup Documentation




    Reviw:
    Realtional Model and Constraints

    Design of Relational Database in E-R (Entity - Relation) Model
    Relational Database processing and Constarints


    Big Data Format as Semi-Structured Data:

    2. XML Lectures:

    Intro to XML Basics: Semi-structured Data and XML Data Model
    Class Note_10: Lecture Notes On Semi-structured Data Model and XML
    Class Note_12: Lecture Notes On XML Syntax and Query Data Model
    Class Note_13: Lecture Notes On Path Expression and XPATH
    Class Note_14: Lecture Notes On XQuery

    Big Data Processing Methods:

    Example of Semantic Web to Extract Web Contents: XML RDF Guide (Resource Definition Framework)

    Resource Definition Framework (RDF) in W3: Semantic Web to Extract Web Contents: XML RDF Guide


    Document Object Model (DOM) Tree Example for XML Data

    XML with DOM and XPath References:
    Complete List of DOM with XPath in www.w3.org

    XML DOM Parsers to Download for Programming in Other Languages:
    DOM XML Parser
    SAX XML Parser
    Which DOM Parser jar file Should I Download

    XML DOM Processng Examples in Progarmming:
    Example Codes of XML Processing with XML Parser in Java
    MDN: DOM Parser
    MDN Site for Code examples of DOM in Java Script
    MDN Site for Code examples of DOM XPATH in Java Script
    Examples of Conversion Between XML and Table

    Basic AJAX XML Processing Examples in Java Script:
    XML File Example: cd_catalog.xml
    AJAX XML File Processing Examples
    AJAX Examples - XML File Processing
    AJAX XML Processing Example


    DOM and XPath in Python:

    Selenium-Python: Setting Up DOM with XPath for Python
    Selenium-Python: XPath API to Locate Elements in Python
    Setting Up DOM with XPath for Python
    BeautifulSoup Documentation for HTML, XML for Python
    Webscapping with DOM, XPath for Python with BeautifulSoup (From the Stanford Class)
    Python XPath Guide

    Examples Conversion Between XML and Table




    DOM and XPath in Java Script:

    MDN Site for DOM with XPath in Java Script
    DOM with XPath Setup Guide for Java Script
    XPath Methods

    Set Up XPath in Java Script
    How to Debug HTML with JAVA Script, DOM, XPath
    Node.JS XPath Setup and XML Parser in Java Script

    MDN: DOM Parser
    MDN Site for Code examples of DOM in Java Script


    Examples of Big data Information Extraction (IE) techniques:
    Resource Description Framework (RDF) for Web Page Content Capturing
    Samentic Web Methods: XML RDF Guide (Resource Definition Framework)


    Other Refernces for XML Applications and Tools:
    XML XSLT Guide
    Lecture Note_5_5 On How DOM Parser and SAX Parser Work to Process XML data (Examples in Mobile App)
    Lecture Notes On RSS in Mobile
    Lecture Notes On Web Service in Mobile


    Automatic Table Creation in SQL Server:

    pyodbc(Open Database Connectivity) to Connect MS SQL Server in Python:

    Installation pyodbc to Connect to MS SQL Server in Python
    pyodbc to Connect to MS SQL Server in Python
    How to Connect to MS SQL Server in Python
    pyodbc: Examples of ODBC in Python

    How to Create a Table in a SQL Server from CSV/TSC file

    How to Create a Table with Bulk Insert with MS SQL Server
    Example of a Script to Create Multiple Tables with Bulk Insert in MS SQL Server



    Big Data Format as Semi-Structured Data:

    3. JSON (JavaScript Object Notation) as Semi-Structured Data:

    JSON Lectures:

    Class Note_10: Lecture Notes On Semi-structured Data Model and Comparisons to HTML, XML, JSON, and Relation

    Class Note_6: Quick JASON Tutorial
    What is JSON?
    JSON Syntax
    real application json data format: Yelp One Business Data Format


    Big Data in JSON Format from Real-life Applications:

    Yelp Data: Business.json
    Twitter Logging Structure in JSON
    Example: Twitter Data Analytics with MongoDB
    How to Exchange and Process JSON Data: JASON.parse and JASON.stringify
    JSON.parse
    Processing Date in JSON
    JSON Data Processing -- JSON.stringify

    JSON Object
    Accessing Properties of JSON Object
    Accesing and Changing JSON Object
    Delete Object in JSON
    JSON Array - Nested Array


    JSON Processing Examples:
    Intro to JSON Data Processing in Python
    Example of JSON Data Processing in Python

    JSON Data Processing with PHP in Server Side
    JSON Data Processing with HTML
    Lecture Note6_5_5: On How to Process JSON data (in Mobile App)

    Official Reference to JSON
    http://www.json.org/







    Semistructured Database System:

    NOSQL Database Semi Structured Data Processing
    MongoDB:

    Lecture Notes on Overview of MongoDB

    Mongo DB Setup:
    MongoDB Download
    MongoDB Manual for Getting Started
    Start your mongo Shell (client) and Connect to MongoDB Server
    MongoDB Cilent: Mongo Shell to start
    How to Import Data file to MongoDB
    Mongo import

    Mongo DB Intro:
    Introduction to Mongo DB
    Mongo DB: Databases/Collections
    Mongo DB: Document Format
    How to Create a MongoDB database and Collection
    Mongo DB Mapping Chart to Compare to SQL

    Mongo DB Basic CRUD Operations:
    Mongo DB: CRUD Operations
    Mongo DB Operations: InsertMany
    Mongo DB Operations: UpdateMany
    Mongo DB Operations: ReplaceOne

    Mongo DB Mapping Chart to Compare to SQL
    Mongo DB Query: Select Operation
    Mongo DB Select Operation on embedded documents
    Mongo DB Select Operation on arrays
    Mongo DB Select Operation on array of documents
    Mongo DB Select Operation on array of embedded documents

    Mongo DB Aggregation:
    Mongo DB: Aggregations
    Mongo DB Join Operator: lookup with unwind for array

    Mongo DB Resources:
    Class Note_22_3: MongoDB Resources
    Mongo DB Documentation
    Application Code Examples in Python with MongoDB and SQL Server

    Example Codes of JSON Data Processing in Python with Mongo DB


    Semistructured Database System - MongoDB: More Advanced

    Mongo DB Getting Started
    Mongo DB
    MongoDB Download
    Set up Guide:
    Node JS with Mongo DB Setup Guide
    Node JS with PostgreSQL Setup Guide
    Sample Web Application Using Node JS with MS SQL Server and Mongo DB (From a Master Project: Building a Recommendation System usng Amazon and Walmart Product and Review Data Sets)


    MongoDB Selection Query:
    Mongo DB Basic Selection Query: $find operator with Condition expressions
    Mongo DB Query (select): $find Example Explained

    MongoDB Query with Comparisons to SQLs:
    Mongo DB: Query find operator basic
    Mongo DB: CRUD Operations
    Mongo DB to SQL Basic Query Mapping Chart (Comparisons to SQL in Examples)

    MongoDB Expressions for Conditions:
    Mongo DB Condition Using $expr With Conditional Statements
    Mongo DB Logical Operators
    Mongo DB Condition with Logical Operator $and

    MongoDB Aggregate Pipelining:
    MongoDB: Intro to Aggregation Pipelining
    MongoDB Query/Aggregation Pipelining Operator Mapping Chart to SQL
    Class Note_24_1: Summary of MongoDB Query/Aggregation Ppelining Exampes with Comparison to SQL
    Complete List of MongoDB Aggregate Operators for Pipelining

    MongoDB lookup Operator for Join with Aggregation Piplelining :
    MongoDB $lookup operator for Join: Join returns null matching as well (Left Outer Join Semantics)
    MongoDB $lookup operator with $unwind for Array Valued Field as Join column
    MongoDB $lookup operator with $let for Multiple Fields for Join Columns
    MongoDB let operator

    MongoDB CRUD -- More Advanced:
    MongoDB UpdateMany Operation with Aggregation Pipelining and $set as $addfields, $unset Operators
    MongoDB Add Field with Aggregate operation Pipelining
    MongoDB Add Field with Aggregate operation Pipelining
    MongoDB Add Field with Aggregate operation Pipelining


    Examples of Big Data Project with MongoDB and Yelp Data

    Project Presenation
    Code Examples with MongoDB and Angular JS


    Examples of Big Data Applications from a Previous Research Team in Big Data Anaytics Lab

    sharesci: Intelligent Search Engine in Natural Language

    sharesci: Content Based Document Search Engine on wiki pages and research papers
    sharesci: Content Based Document Search Engine with Machine Learning

    sharesci github site for Source Codes
    sharesci wiki site in bigdata lab for documentation
    sharesci Project Report for Source Codes




    Lecture Sections:

  • Class_Notes: Introduction to Big Data Technologies and Big Data Processing Systems on Cloud
  • Class_Notes: Object Relational Mapping (ORM) for Big Data Processing Methods
  • Class_Notes: Web Application With Big Data Processing
  • Class_Notes: Big Data and Processing Techniques as Semi-Structured Models
  • Class_Notes: Big Data and Processing Techniques as Unstructured Texts
  • Class_Notes: Transaction and Concurrency Control
  • Class_Notes: Parallel Big Data Processing Systems on Cloud: NOSQL/NewSQL Systems



  • Unstructured Text Data Processing

    Big Data as Unstructured Text


    Unstructured Text data Processing techniques for Information Extraction (IE) Methods

    1. Inverted Index:

    Class Note_18_1: Lecture Notes On Web Data Processing: Inverted Index in Google (Stanford) Class Note_18_2: Lecture Notes On Information Retrieval for Google Search engine (Stanford)
    Cosine Similarity Example
    Class Note_18_3:Positioning Index for Phrase Query (Stanford)
    Sample Big Data Project: Building a Content Based Document(wiki Pages) Search Engine System
    Sample Big Data Analytic Project: Building a Content Based Document Ranking System Using Machine Leraning
    Example of Inverted Index Tables for Text Analysis in State of Union Addresses < Invered Index Example: Google NGram Viewer : Inverted Index for all N gram words



    2. Natural Languge Processing (NLP) Techniques:
    How to Build a Sentiment/Opinion Analysis Application with Text Analysis Techniques -- For Final Project !!
    Overview of Text Processing Tasks for Sentiment Analysis of User Review Texts :

    Class Note_18_1: Lecture Notes On Summarizing Opinions of Frequent Feature(Aspect) Words in Review texts
    Class Note_18_2: Lecture Notes On Research for Sentiment/Aspect Analysis Algorithms/Methods


    Context Aware NLP Methods:

    POS (Part of Speech) Tagging:

    Lecture Notes on POS (Part of Speech) Tagging (Stanford)
    English Grammar
    UPenn Treebank POS Tagging List


    NER (Named Entity) Tagging and Information Extraction (IE) :

    Lecture Notes on NER (Named Entity Recognizer) (Stanford)

    Software NER Tagging Parser:

    Stanford NLP NER (Named Entity) Tagging

    Software Open IE (Information Extraction) Parser:

    Stanford NLP Open IE (Information Extraction) for Triple Extraction


    Useful Natural Language Processing (NLP) Techniques to Use and Resource Sites for Download to Use for Text Analysis Applications:

    Core NLP Software Site of Stanford
    Core NLP online Run Test Site (Stanford)
    Useful Python Text Processing Lib: spaCy
    Python spaCy Example Codes for Text Processing Tasks
    NLP Parsers to download:

    Core Stanford Parser Download for POS and NER tagging (Github repo site as well)

    Avaliable Stanford NLP Softwares


    Stanford NLP POS (part Of Speech) Tagging
    Stanford NLP Parser
    Stanford NLP POS Tagger in JAVA


    WordNet Site (by Princeton) to downlaod NLP Software:

    WordNet by Princeton
    Download WordNet for Sentiment Analysis on Aspect/Feature Opinion Analysis



    Example of Text Analysis Tasks for Big Data Applications (Intelligent Systems):
    Overview of Text Processing Tasks for Review Analysis:
    Lecture Notes On Summarizing Frequent Feature(Aspect) words and Opinions in Review texts
    Lecture Notes On Aspect Analysis Approaches

    Example of Big data Project with Open IE Method: Building a Medical Question Answering System

    2. Passage Construction




  • Lectures on BIG Data Processing Systems

    Review in File System of RDBMS:
    Class Note_8: Chapter 17 Disk Storage System, File Structure, Hash Index
    Class Note_9: Chapter 18 Index Structure for File


  • Big Data Processing: MapReduce and Hadoop Distributed File System (HDFS)

    Class Note_18: Lecture Notes On Map Reduce (Google)
    Paper: Map Reduce by Google
    External Sort and External Hash Partition on Single Node (From CIS611 Notes)

    Class Note_18_1: Lecture Notes On Hadoop Distributed File System (HDFS)
    Another Word Count Map Reduce
    Hadoop Job Class

    Hadoop MapReduce Tutorial:
    Map Reduce Tutorial on Hadoop

    Hadoop Interface Input Format - Input Split: getSplits()
    Hadoop Interface for Writable
    Hadoop Interface for WritableComparable
    Hadoop Interface for OutputCollector
    Hadoop Interface for Map
    Hadoop Interface for Comparator to control Grouping
    Hadoop Interface for Combiner to Local Aggregation on the intermediate Output
    Hadoop Interface for Reducer

    Hadoop Parameters: io.sort.mb


    Hadoop Implementation Set Up:
    Hadoop Single Node Set Up
    Hadoop Fully Distributed Mode (Cluster) Set Up


    Code Examples of MapReduce on Hadoop:
    Simple MR Hadoop Lab Guides
    MR sample Codes - Average Temp By Zip Code
    MR Sample Codes - Sort By Station ID
    station Data Set
  • Transaction and Concurrency Control

    Review on Transaction in Relational Database System (RDBMS):
    Class Note_0: Review: CIS430/530 Lecture Notes On Relational DBMS: View, Trigger and Transaction
    Class Note_0: Review: Examples on Transaction with Multiple Concurrent Users

    Lecture Notes on Transaction and Concurrency Control:

    Class Notes_3: Overview On Relational DBMS Architecture, Query Processing, View and Transaction
    Class Note_4: Lecture Notes On Transaction
    Class Note_5: Lecture Notes On Concurrency Control




  • Quick Review: What are Data Warehouse and OLAP (Online Analytical Processing) ?

    For Reference Only for Those Who Did Not Take CIS 611 Enterprise Database Systems. DW and OLAP won't be covered in this course.
    Class Note_24: Lecture Notes On Overview of Data Warehouse with OLAP
    Class Note_24_1: Paper on Data Cube by Jim Gray (Microsoft), et al



  • Parallel Big Data Processing Systems:

  • NOSQL Systems:

    Class Note_22: Lecture Notes On Introduction to NO SQL

  • Hive
    Class Note_19: Lecture Notes On Hive (Facebook)

    Paper: Hive by Facebook

    Hive Documentations:
    Apathe Hive Site
    Apathe Hive Documentation
    Apathe Hive Tutorial

    Important Hive Tutorials:
    Apathe Hive Getting Started
    Hive Lanuage Manual: DDL, DML
    Hive Complex Data Types
    Examples for Hive Complex Data TypeS: Union, Map as a collection of key:value pairs
    Example of Hive DDL with Partitioning
    Hive DDL-Row Format, Storage Format,and SerDe
    Transform Clause to Provide User Customized Map and Reduce Script
    SerDe:
    Guide-Hive SerDe (Serializer and Deserializer)
    Hive Row Format with SerDe (Serializer and Deserializer)
    Hive Built in SerDe
    Code Example for Hive CSV SerDe
    Code Example for Hive JSON SerDe for Twitter data
    Code Example for Hive JSON SerDe for Twitter data (in doc)
    UDTF:
    Hive UDTF Tutorial
    Hive UDTF Example
    Hive File Format

    Sample Project Using Hive:
    Sample Project: Amazon Review Data Processing and Analytics Using Hive and JSON SerDe
    Sample Project: LinkeIn Data Processing and Analytics Using Hive
    Sample Project: Twitter Data Processing and Analytics Using Hive and MongoDB

  • Apathe Pig Latin:
    Class Note_20: Lecture Notes On Pig Latin (By Yahoo)
    Class Note_20_1: Lecture Notes On Pig Latin (By Yahoo)

    Class Note_20_2: Lecture Notes On MapReduce Join Algorithms
    Class Note_20_3: Lecture Notes On Advanced Partitioning Techniques

    Paper: Pig Latin By Yahoo
    Apathe Pig Latin Site to Start

    Sample Project Using PIG Latin:
    Sample Project: Nasa Network Server Log Data Processing and Analytics Using PIG Latin and R



  • Google's Big Table:
    Class Note_21: Lecture Notes On Big Table (By Google)
    Paper: BigTable by Google

  • FaceBook's HBase:

    Class Lectures:
    Class Note_23: Lecture Notes On HBase (FaceBook)
    Class Note_23_1: Lecture Notes More On HBase Architecture
    Class Note_23_2: Apathe HBase Hive Integration (by HortonWorks)
    Apathe HBase Hive Integration (in MapR)

    Quick Start Guides:
    Class Note_23_3: Quick Guide On HBase
    HBase Basic Tutorial to start
    HBase with Hive Set Up guide by Paul Webster
    HBase Tutorial with HBase Join with Hive by Paul Webster

    Apathe HBase Sites to Dive:
    Apathe HBase Site To Start
    Apathe HBase serDe
    Apathe HBase Hive Integration


    Major Prebuilt Platforms for Hadoop with NoSQL Stack:
    Hornworks:
    Hornworks Big Data Platform for Prebuilt Hadoop Stack with HBase Hive Integration
    Cloudera:
    Cloudera Big Data Platform for Prebuilt with Spark Hive Integration
    Cloudera Big Data Platform for Prebuilt Data Scientist Work Bench
    Cloudera Set Up for Spark Tutorial From Danielle Aring's Report

    Comparison: Hornworks vs Clodera vs MapR for Prebuilt Hadoop Stack Platform with HBase Hive Integration


    More on HBase:
    Apathe HBase Architecture
    More in Depth of Apathe HBase Architecture
    Apathe HBase Architecture

    Papers on HBase:
    Data Warehousing and Analytics Infrastructure at Facebook
    Big Data Processing System at FaceBook (From SIGMOD 2013)
    Paper on HBase
    Paper on HBase



  • MongoDB:

    Class Note_24: Lecture Notes On MongoDB
    Mongo DB: Databases/Collections
    Mongo DB: Databases/Collections
    Mongo DB: CRUD Operations
    Mongo DB Getting Started (See for Match an Element in an Array)

    MongoDB Selection Query:
    Mongo DB Basic Selection Query: $find operator with Condition expressions
    Mongo DB Query (select): $find Example Explained

    MongoDB Query with Comparisons to SQLs:
    Mongo DB to SQL Basic Query Mapping Chart (Comparisons to SQL in Examples)

    MongoDB Expressions for Conditions:
    Mongo DB Condition Using $expr With Conditional Statements
    Mongo DB Logical Operators
    Mongo DB Condition with Logical Operator $and

    MongoDB Aggregate Pipelining:
    MongoDB: Intro to Aggregation Pipelining
    MongoDB Query/Aggregation Pipelining Operator Mapping Chart to SQL
    Class Note_24_1: Summary of MongoDB Query/Aggregation Ppelining Exampes with Comparison to SQL
    Complete List of MongoDB Aggregate Operators for Pipelining

    MongoDB lookup Operator for Join with Aggregation Piplelining :
    MongoDB $lookup operator for Join: Join returns null matching as well (Left Outer Join Semantics)
    MongoDB $lookup operator with $unwind for Array Valued Field as Join column
    MongoDB $lookup operator with $let for Multiple Fields for Join Columns
    MongoDB let operator

    Mongo DB DDL/DML:
    MongoDB Index
    MongoDB Text Index Features
    Mongo DB: View for Read Only
    Mongo DB: BSON Document Format
    Mongo DB Distributed Cluster: Sharded Collection

    MongoDB Concurrency Control:
    Mongo DB: Update with Upsert = true
    MongoDB Update with upsert and Controlling Concurrency Probelms
    Mongo DB: Bulk Write

    MongoDB Other Operators:
    Mongo DB: text serach Operator
    MongoDB System variable
    Compelete List of Mongo DB Operators

    How to Discover MongoDB Scheme:
    How to Discover MongoDB Schema Information
    Mongo DB schema-analyzer
    Mongo DB schema-analyzer Tool: Variety

    Sample Project Using MongoDB:
    Sample Project: Twitter Data Processing and Analytics Using MongoDB

    Mongoose for connecting to MongoDB with Node JS Express:
    Mongoose To Set Up
    Mongoose Guide
    Node API for MongoDB:
    NodeJS: Integrating to MongoDB

    MongoDB Sites to Set Up:
    Class Note_22_3: MongoDB Site
    Mongo DB Resources
    MongoDB Download
    Mongo DB Getting started
    Mongo DB Documentation
    Mongo DB Glossary: _id



  • Cassandra:
    Class Note_25: Lecture Notes On Cassandra
    Apache Cassandra Site to start

  • Real Time Big Data Stream Processing System:

    Spark:

    Class Note_26: Lecture Notes On Spark
    Class Note_26_1: Lecture Notes On Spark: Spark Streaming/SparkSQL/SparkR

    Quick Tutorial to Start:
    Berkerley Spark Tutorial (in Scala or Java)
    Lecture Notes_27 On Storm

    Spark Tutorial with Text Analysis Project Examples:
    Spark Tutorial I for Set Up on Cloudera VM and Basic Text Analysis (From Danielle Aring's Report)
    Spark Tutorial II for Product Review Text Analysis (From Danielle Aring's Report)


    Prebuilt Data Analytic Platform with Cloudera:

    Cloudera Big Data Platform for Prebuilt with Spark Hive Integration
    Cloudera Big Data Platform for Prebuilt Data Scientist Work Bench
    Cloudera Set Up for Spark Tutorial From Danielle Aring's Report

    Sample Project Using Spark and MapReduce:
    Tutorial:Real Time Twitter Analysis with Spark and Kafka
    Analysis of State Union Address with We vs Me index
    Analysis of Amazon Review Data with Spark

    Useful Spark Sites:
    Apache Spark Site to start
    Apache Spark Documentation
    Spark Programming Guide
    Spark with Cloudera Site to start
    Launching Spark on Amazon EC2 Cluster

    Berkerley Spark Tutorial (in Scala or Java)
    Spark RDD
    How to Get Twitter Data

    Spark Twitter Collection in Scala
    Spark Pipeline
    Spark Pipelining Example
    Spark RDD Partitioning
    Spark Reduce and Fold Example

    Papers:
    Apache Spark for Stream DM
    Stream Data Analytics with SparkSQL
    Apache SparkSQL
    Apache SparkR
    Shark
    Shark SQL


    kafka: Asynchronous Big Data Stream Messanging System
    big data messaging with kafka
    kafka Set up Instruction Note that you need Java and Zoo Keeper for Kafka Set Up

    Sample Project Using Kafka:
    Real Time Twitter Analysis with Spark and Kafka


  • Parallel Big Data Processing Systems: New SQL Systems

  • VoltDB:
    Class Note_28: Lecture Notes On New SQL
    Class Note_29: Lecture Notes On VoltDB (New SQL)
    VoltDB Site to start


  • Big Data Processing Systems on Cloud Computing

    Class Note_30: Lecture Notes On Cloud Computing
    Mobile, Multimedia and Cloud Computing

    Amazon Cloud Service:
    Amazon Elastic Cloud Computing (EC2) for Web service
    How to Create Amazon Virtual Hosting (EC2) with a Web Server and Amazon Database Server (RDS)
    Amazon DynamoDB Database Service
    Amazon RDS (Relational Database Service)
    You can create your Big Data Processing Infrastructure on Amazon Cloud
    Amazon Cloud Account for Students

    Microsoft Cloud Service Azure
    Trial account for Microsoft AZURE Cloud
    AZURE Cloud
    Tutorial for MS AZURE Cloud
    Sample Project on Microsoft AZURE Cloud
    How to Create/Retrieve a Table in Microsoft AZURE Cloud
    How to Create/Retrieve BLOB data in Microsoft AZURE Cloud
    How to Create a SQL Database Server in Microsoft AZURE Cloud





    • Send email to me: sschung.cis@gmail.com