Jump to content

User:Skptic/sandbox

From Wikipedia, the free encyclopedia
Nextflow
Original author(s)Paolo Di Tommaso
Developer(s)Seqera Labs, Centre for Genomic Regulation
Initial releaseApril 9, 2013; 11 years ago (2013-04-09)
Stable release
v19.01.0 / January 22, 2019; 5 years ago (2019-01-22)
Preview release
v19.01.0-edge / March 14, 2019; 5 years ago (2019-03-14)
Repositoryhttps://github.com/nextflow-io/nextflow
Written inGroovy, Java
Operating systemLinux, macOS
TypeScientific workflow system, Dataflow programming, Big data
LicenseApache License 2.0
Websitenextflow.io

Nextflow is an open-source scientific workflow system based on the Dataflow programming model. Originally developed at the Centre for Genomic Regulation and is now actively developed at Seqera Labs by the original authors. Nextflow provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Overview

[edit]

Apache Spark has as its architectural foundation the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.[1] The Dataframe API was released as an abstraction on top of the RDD, followed by the Dataset API. In Spark 1.x, the RDD was the primary application programming interface (API), but as of Spark 2.x use of the Dataset API is encouraged[2] even though the RDD API is not deprecated.[3][4] The RDD technology still underlies the Dataset API.[5][6]

Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.[7]

Spark facilitates the implementation of both iterative algorithms, which visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementation.[1][8] Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.[9]

Apache Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster), Hadoop YARN, or Apache Mesos.[10] For distributed storage, Spark can interface with a wide variety, including Alluxio, Hadoop Distributed File System (HDFS),[11] MapR File System (MapR-FS),[12] Cassandra,[13] OpenStack Swift, Amazon S3, Kudu, or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

Domain-specific language

[edit]

Spark SQL is a component on top of Spark Core that introduced a data abstraction called DataFrames,[a] which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language (DSL) to manipulate DataFrames in Scala, Java, or Python. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. Although DataFrames lack the compile-time type-checking afforded by RDDs, as of Spark 2.0, the strongly typed DataSet is fully supported by Spark SQL as well.

The Nextflow domain-specific language is a superset of the Apache Groovy programming language. It includes processes 

greetings = Channel.from("Hello", "Ciao", "Hola", "Bonjour")

process hello_world {

    input:
    val x from greetings

    output:
    file "${x}.txt" into grettings_txts

    script:
    """
    echo "${x} World!" > ${x}.txt
    """
    
}

Containers

[edit]

Spark Streaming uses Spark Core's fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD transformations on those mini-batches of data. This design enables the same set of application code written for batch analytics to be used in streaming analytics, thus facilitating easy implementation of lambda architecture.[15][16] However, this convenience comes with the penalty of latency equal to the mini-batch duration. Other streaming data engines that process event by event rather than in mini-batches include Storm and the streaming component of Flink.[17] Spark Streaming has support built-in to consume from Kafka, Flume, Twitter, ZeroMQ, Kinesis, and TCP/IP sockets.[18]

In Spark 2.x, a separate technology based on Datasets, called Structured Streaming, that has a higher-level interface is also provided to support streaming.[19]

Execution Engines

[edit]

Spark MLlib is a distributed machine-learning framework on top of Spark Core that, due in large part to the distributed memory-based Spark architecture, is as much as nine times as fast as the disk-based implementation used by Apache Mahout (according to benchmarks done by the MLlib developers against the alternating least squares (ALS) implementations, and before Mahout itself gained a Spark interface), and scales better than Vowpal Wabbit.[20] An overview of Spark MLlib is exist.[21] Many common machine learning and statistical algorithms have been implemented and are shipped with MLlib which simplifies large scale machine learning pipelines, including:

History

[edit]

Spark was initially started by Matei Zaharia at UC Berkeley's AMPLab in 2009, and open sourced in 2010 under a BSD license.[23]

In 2013, the project was donated to the Apache Software Foundation and switched its license to Apache 2.0. In February 2014, Spark became a Top-Level Apache Project.[24]

In November 2014, Spark founder M. Zaharia's company Databricks set a new world record in large scale sorting using Spark.[25][23]

Spark had in excess of 1000 contributors in 2015,[26] making it one of the most active projects in the Apache Software Foundation[27] and one of the most active open source big data projects.

Given the popularity of the platform by 2014, paid programs like General Assembly and free fellowships like The Data Incubator have started offering customized training courses[28]

Version Original release date Latest version Release date
Old version, no longer maintained: 0.5 2012-06-12 0.5.1 2012-10-07
Old version, no longer maintained: 0.6 2012-10-14 0.6.2 2013-02-07[29]
Old version, no longer maintained: 0.7 2013-02-27 0.7.3 2013-07-16
Old version, no longer maintained: 0.8 2013-09-25 0.8.1 2013-12-19
Old version, no longer maintained: 0.9 2014-02-02 0.9.2 2014-07-23
Old version, no longer maintained: 1.0 2014-05-26 1.0.2 2014-08-05
Old version, no longer maintained: 1.1 2014-09-11 1.1.1 2014-11-26
Old version, no longer maintained: 1.2 2014-12-18 1.2.2 2015-04-17
Old version, no longer maintained: 1.3 2015-03-13 1.3.1 2015-04-17
Old version, no longer maintained: 1.4 2015-06-11 1.4.1 2015-07-15
Old version, no longer maintained: 1.5 2015-09-09 1.5.2 2015-11-09
Old version, no longer maintained: 1.6 2016-01-04 1.6.3 2016-11-07
Old version, no longer maintained: 2.0 2016-07-26 2.0.2 2016-11-14
Old version, no longer maintained: 2.1 2016-12-28 2.1.3 2018-06-26
Old version, no longer maintained: 2.2 2017-07-11 2.2.3 2019-01-11
Old version, yet still maintained: 2.3 2018-02-28 2.3.3 2019-02-15
Current stable version: 2.4 2018-11-02 2.4.0 2018-11-02
Legend:
Old version, not maintained
Old version, still maintained
Latest version
Latest preview version
Future release

See also

[edit]

Notes

[edit]
  1. ^ Called SchemaRDDs before Spark 1.3[14]

References

[edit]
  1. ^ a b Zaharia, Matei; Chowdhury, Mosharaf; Franklin, Michael J.; Shenker, Scott; Stoica, Ion. Spark: Cluster Computing with Working Sets (PDF). USENIX Workshop on Hot Topics in Cloud Computing (HotCloud).
  2. ^ "Spark 2.2.0 Quick Start". apache.org. 2017-07-11. Retrieved 2017-10-19. we highly recommend you to switch to use Dataset, which has better performance than RDD
  3. ^ "Spark 2.2.0 deprecation list". apache.org. 2017-07-11. Retrieved 2017-10-10.
  4. ^ Damji, Jules (2016-07-14). "A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets: When to use them and why". databricks.com. Retrieved 2017-10-19.
  5. ^ Chambers, Bill (2017-08-10). "12". Spark: The Definitive Guide. O'Reilly Media. virtually all Spark code you run, where DataFrames or Datasets, compiles down to an RDD
  6. ^ "What is Apache Spark? Spark Tutorial Guide for Beginner". janbasktraining.com. Retrieved 2018-04-13.
  7. ^ Zaharia, Matei; Chowdhury, Mosharaf; Das, Tathagata; Dave, Ankur; Ma,, Justin; McCauley, Murphy; J., Michael; Shenker, Scott; Stoica, Ion (2010). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (PDF). USENIX Symp. Networked Systems Design and Implementation.{{cite conference}}: CS1 maint: extra punctuation (link)
  8. ^ Xin, Reynold; Rosen, Josh; Zaharia, Matei; Franklin, Michael; Shenker, Scott; Stoica, Ion (June 2013). "Shark: SQL and Rich Analytics at Scale" (PDF). {{cite journal}}: Cite journal requires |journal= (help); Unknown parameter |conference= ignored (help)
  9. ^ Harris, Derrick (28 June 2014). "4 reasons why Spark could jolt Hadoop into hyperdrive". Gigaom.
  10. ^ "Cluster Mode Overview - Spark 1.2.0 Documentation - Cluster Manager Types". apache.org. Apache Foundation. 2014-12-18. Retrieved 2015-01-18.
  11. ^ Figure showing Spark in relation to other open-source Software projects including Hadoop
  12. ^ MapR ecosystem support matrix
  13. ^ Doan, DuyHai (2014-09-10). "Re: cassandra + spark / pyspark". Cassandra User (Mailing list). Retrieved 2014-11-21. {{cite mailing list}}: Unknown parameter |mailinglist= ignored (|mailing-list= suggested) (help)
  14. ^ https://spark.apache.org/releases/spark-release-1-3-0.html
  15. ^ "Applying the Lambda Architecture with Spark, Kafka, and Cassandra | Pluralsight". www.pluralsight.com. Retrieved 2016-11-20.
  16. ^ Shapira, Gwen (29 August 2014). "Building Lambda Architecture with Spark Streaming". cloudera.com. Cloudera. Retrieved 17 June 2016. re-use the same aggregates we wrote for our batch application on a real-time data stream
  17. ^ "Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming" (PDF). IEEE. May 2016.
  18. ^ Kharbanda, Arush (17 March 2015). "Getting Data into Spark Streaming". sigmoid.com. Sigmoid (Sunnyvale, California IT product company). Retrieved 7 July 2016.
  19. ^ Zaharia, Matei (2016-07-28). "Structured Streaming In Apache Spark: A new high-level API for streaming". databricks.com. Retrieved 2017-10-19.
  20. ^ Sparks, Evan; Talwalkar, Ameet (2013-08-06). "Spark Meetup: MLbase, Distributed Machine Learning with Spark". slideshare.net. Spark User Meetup, San Francisco, California. Retrieved 10 February 2014.
  21. ^ Harifi, Sasan; Byagowi, Ebrahim; Khalilian, Madjid (2017). Tan, Ying; Takagi, Hideyuki; Shi, Yuhui (eds.). "Comparative Study of Apache Spark MLlib Clustering Algorithms". Data Mining and Big Data. Lecture Notes in Computer Science. Springer International Publishing: 61–73. doi:10.1007/978-3-319-61845-6_7. ISBN 9783319618456.
  22. ^ "MLlib | Apache Spark". spark.apache.org. Retrieved 2016-01-18.
  23. ^ a b Clark, Lindsay. "Apache Spark speeds up big data decision-making". ComputerWeekly.com. Retrieved 2018-05-16.
  24. ^ "The Apache Software Foundation Announces Apache&#8482 Spark&#8482 as a Top-Level Project". apache.org. Apache Software Foundation. 27 February 2014. Retrieved 4 March 2014.
  25. ^ Spark officially sets a new record in large-scale sorting
  26. ^ Open HUB Spark development activity
  27. ^ "The Apache Software Foundation Announces Apache&#8482 Spark&#8482 as a Top-Level Project". apache.org. Apache Software Foundation. 27 February 2014. Retrieved 4 March 2014.
  28. ^ "NY gets new bootcamp for data scientists: It's free, but harder to get into than Harvard". Venture Beat. Retrieved 2016-02-21.
  29. ^ "Spark News". apache.org. Retrieved 2017-03-30.
[edit]
  • No URL found. Please specify a URL here or add one to Wikidata.

Category:Apache Software Foundation Category:Big data products Category:Cluster computing Category:Data mining and machine learning software Category:Free software programmed in Scala Category:Hadoop Category:Java platform Category:Software using the Apache license Category:University of California, Berkeley Category:Articles with example Scala code


Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=User:Skptic/sandbox&oldid=1086842710"