pyspark: Pandas-on-Spark
The video below discusses the use of pyspark and the pandas-on-spark framework. This utilizes a pandas style data frame and functionality.
I highly recommend watching the video using the ‘full’ Panopto player. There is a ‘pop out’ button in the bottom right of the video to enter this viewer.
The pyspark code used in the notes and the example done at the end of the notes is available in this notebook. You’ll need to download this .ipynb file and upload it to your JupyterHub environment. Make sure that the kernel used to run the notebook is a pyspark kernel!
Remember, if you are off campus you should log in to the VPN and then you can access our JupyterHub.
If you are not an NC State student, you can download docker and gain access to Spark with a Jupyter notebook interface reasonably quickly!
Notes
Use the table of contents on the left or the arrows at the bottom of this page to navigate to the next learning material!