OSS TanbouOverview
Apache Spark is a unified analytics engine for large-scale data processing. On top of high-level Scala, Java, and Python APIs and an optimized engine, one project provides Spark SQL and DataFrames, pandas API on Spark, MLlib, GraphX, and Structured Streaming.
Features and best fit
Based on official documentation; not hands-on tested · Content checked:
Run batch, SQL, streaming, and machine learning on one engine
The current README combines an engine for general computation graphs with Spark SQL and DataFrames, a pandas-workload API, MLlib for machine learning, GraphX for graph processing, and Structured Streaming. The repository also documents Scala and PySpark shells, example programs, builds, and tests.
Sources: [1]
For unifying large data workflows across multiple nodes
It is a candidate for data teams bringing high-volume batch aggregation, interactive SQL, feature engineering, and stream processing onto a shared cluster foundation. Test partitions, skew, shuffles, memory, serialization, connectors, failure recovery, and cost with representative data.
Sources: [1]
Separate the development branch from release docs and supported languages
The README contains only basic setup and sends users to versioned documentation on the project site. Do not apply master build instructions directly to a production release; confirm Hadoop, Java, Scala, and Python compatibility for the selected version. The README marks the R API as deprecated. Spark uses the Apache License 2.0.
Official sources
- [1]Apache Spark README(2026-09-13)
- [2]Apache Spark license(2026-09-13)
Supplemental curator note
Selected for qualifying the common word Spark with Apache and expanding into a large long-tail surface across SQL, DataFrames, streaming, and ML.
Growth
Growth trends · Last 30 days
43,989 Stars
Trend data is still being collected.
Development activity
Last 90 days · weekly
- Commits (last 30 days)
- 464
- Open PRs
- 517
Development activity is still being collected.
Built with
Categories and tags
Categories
GitHub data
GitHub dataView detailed GitHub data
GitHub Topics
- python
- scala
- r
- java
- big-data
- jdbc
- sql
- spark
- Stars
- 43,989
- Forks
- 29,371
- Watchers
- 1,998
- Open issues
- 32
- Primary language
- Scala
- License
- Apache-2.0
- Repository last updated
- Sep 13, 2026
Explore next
- scikit-learn67,238 Stars
A Python ML library unifying preprocessing, classification, regression, clustering, model selection, and evaluation.
Python - pandas49,724 Stars
A Python DataFrame library for missing values, joins, aggregation, reshaping, time series, and tabular I/O.
Python - Apache Airflow46,842 Stars
A platform for defining DAGs in Python and scheduling, running, and monitoring batch, data, and ML workflows.
Python - Apache Kafka33,715 Stars
A distributed event-streaming platform for data pipelines, streaming analytics, and system integration.
Java - Apache Flink26,336 Stars
A stateful data framework for stream and batch processing with event time, windows, exactly-once guarantees, and back-pressure.
Java
Report incorrect information
Tell us if any listing information is incorrect or outdated.