Word count & spark
You can do this using Spark's Python API, PySpark. Here's a step-by-step guide on how to create a word count program:
- In your Python script, start by importing the necessary Spark modules:
from pyspark import SparkContext
sc = SparkContext("local", "WordCountApp")
- "local" indicates that you are running Spark in local mode.
- "WordCountApp" is the application name.
input_file = "input.txt"
text_data = sc.textFile(input_file)
Make sure to replace "input.txt" with the path to your input text file.
word_counts = text_data.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)
- flatMap splits each line into words.
- map converts each word into a key-value pair with a count of 1.
- reduceByKey combines counts for the same word.
output_file = "word_count_result"
word_counts.saveAsTextFile(output_file)
sc.stop()
Finally, run the script with:
spark-submit word_count.py