Big Data Basics

class: center, middle, inverse, title-slide

.title[
# Big Data Basics
]
.author[
### Justin Post
]

---

# Where are we?

- Foundation in Python through `JupyterLab`

Now:

- What is Big Data?
- What are common things we'll run into with Big Data?
- What issues are we going to try and tackle?
- What software can be useful?

---

# What is Big Data?

Useful definition:

- Big data = data that you can't handle 'normally'

---

# What is Big Data?

Useful definition:

- Big data = data that you can't handle 'normally'

&nbsp;  
&nbsp;  
&nbsp;  
&nbsp;

- Big Data usually requires learning of new tools

- [Want to feel overwhelmed?](https://mattturck.com/landscape/mad2024.pdf)

---

# What are Common Attributes of Big Data?

- Fifth **V** = [Value](https://www.ibm.com/think/topics/big-data-analytics)

---

# What are We Going to Consider?

- Basics of Big Data storage 
 

- Basics of Big Data querying 
 

- Basics of handling streaming data 
 

- Summarizing and modeling Big Data

---

# Data and Model Pipelines

In the end, we should have a reasonable idea of how data comes in, gets transformed/combined/etc., can be used to build models or predict an outcome!

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#img/Data-Processing-Pipelines-patterns-informatica.png" alt="Image from informatica.com" width="700px" />
Image from informatica.com
</div>

---

# Recap

- Big Data requires learning new tools & considering different algorithms

- Storage and retrieval of data is important

- Modeling and summarizing data can be done

- Should consider overall process/pipeline of data from start to finish