Jump to content

NSynth: Difference between revisions

Add links
From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
No edit summary
No edit summary
Line 13: Line 13:
}}
}}


'''NSynth''' (Neural Synthesizer) is an audio synthesizer introducing a different approach to audio synthesis, using deep neural networks to generate sounds at the level of individual samples. The [[algorithm]], outlined in a paper in April 2017<ref name="auto">{{Cite arXiv|eprint=1704.01279|class=cs.LG|first1=Jesse|last1=Engel|first2=Cinjon|last2=Resnick|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|last3=Roberts|first3=Adam|last4=Dieleman|first4=Sander|first5=Douglas|last6=Simonyan|first6=Karen|last7=Norouzi|first7=Mohammad|year=2017|last5=Eck}}</ref>, generates new sounds through a [[neural network]] based [[synthesis]], employing a [[WaveNet]]-style [[autoencoder]] to learn its own temporal embeddings from four different sounds.<ref>{{Cite web|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|url=https://research.google/pubs/pub46119/|website=research.google}}</ref>. Google then released an open source hardware interface for the algorithm called NSynth Super<ref>{{Cite web|title=Google's open-source neural synth is creating totally new sounds|url=https://www.wired.co.uk/article/google-ai-nsynth-algorithm-music-creativity|website=wired.co.uk}}</ref>, used by musicians such as [[Grimes]]<ref>{{Cite web|title=73 &#124; Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll|url=https://www.preposterousuniverse.com/podcast/2019/11/18/73-grimes-c-on-music-creativity-and-digital-personae/|website=www.preposterousuniverse.com}}</ref> and [[Yacht (band)|YACHT]] <ref>{{Cite web|title= Music and Machine Learning (Google I/O'19) |url=https://www.youtube.com/watch?v=pM9u9xcM_cs|website=youtube.com}}</ref> to generate experimental music using AI. The research and development of the algorithm was part of a collaboration between [[Google Brain]], [[Magenta (research team) | Magenta]] and [[DeepMind]].<ref>{{Cite web|title=NSynth: Neural Audio Synthesis|url=https://magenta.tensorflow.org/nsynth|website=Magenta}}</ref>
'''NSynth''' (a [[portmanteau]] of "Neural Synthesizer") is a software [[algorithm]], outlined in a paper in April 2017<ref name="auto">{{Cite arXiv|eprint=1704.01279|class=cs.LG|first1=Jesse|last1=Engel|first2=Cinjon|last2=Resnick|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|last3=Roberts|first3=Adam|last4=Dieleman|first4=Sander|first5=Douglas|last6=Simonyan|first6=Karen|last7=Norouzi|first7=Mohammad|year=2017|last5=Eck}}</ref>, that generates new sounds through a [[neural network]] based [[synthesis]], employing a [[WaveNet]]-style [[autoencoder]] to learn its own temporal embeddings from four different sounds.<ref>{{Cite web|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|url=https://research.google/pubs/pub46119/|website=research.google}}</ref>. Google then released an open source hardware interface for the algorithm called NSynth Super<ref>{{Cite web|title=Google's open-source neural synth is creating totally new sounds|url=https://www.wired.co.uk/article/google-ai-nsynth-algorithm-music-creativity|website=wired.co.uk}}</ref>, used by musicians such as [[Grimes]]<ref>{{Cite web|title=73 &#124; Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll|url=https://www.preposterousuniverse.com/podcast/2019/11/18/73-grimes-c-on-music-creativity-and-digital-personae/|website=www.preposterousuniverse.com}}</ref> and [[Yacht (band)|YACHT]] <ref>{{Cite web|title= Music and Machine Learning (Google I/O'19) |url=https://www.youtube.com/watch?v=pM9u9xcM_cs|website=youtube.com}}</ref> to generate experimental music using AI. The research and development of the algorithm was part of a collaboration between [[Google Brain]], [[Magenta (research team) | Magenta]] and [[DeepMind]].<ref>{{Cite web|title=NSynth: Neural Audio Synthesis|url=https://magenta.tensorflow.org/nsynth|website=Magenta}}</ref>


== Neural Audio Synthesis ==
== Development ==


=== Dataset ===
=== Dataset ===
Line 22: Line 22:


=== Model ===
=== Model ===

A spectral autoencoder model and a WaveNet autoencoder model are publicly available on GitHub <ref>{{Cite web|title=NSynth: Neural Audio Synthesis|url=
https://github.com/magenta/magenta/tree/main/magenta/models/nsynth|website=GitHub}}</ref>. The baseline model uses a spectrogram with fft_size 1024 and hop_size 256, MSE loss on the magnitudes, and the Griffin-Lim algorithm for reconstruction. The WaveNet model trains on mu-law encoded waveform chunks of size 6144. It learns embeddings with 16 dimensions that are downsampled by 512 in time<ref name="auto">{{Cite arXiv|eprint=1704.01279|class=cs.LG|first1=Jesse|last1=Engel|first2=Cinjon|last2=Resnick|title=Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders|last3=Roberts|first3=Adam|last4=Dieleman|first4=Sander|first5=Douglas|last6=Simonyan|first6=Karen|last7=Norouzi|first7=Mohammad|year=2017|last5=Eck}}</ref>.


{{Infobox synthesizer
{{Infobox synthesizer

Revision as of 09:57, 2 November 2022

NSynth neural synthesizer
Original author(s)Google Brain, Deep Mind, Magenta
Initial release6 April 2017; 7 years ago (2017-04-06)
Repositorygithub.com/magenta/magenta/tree/main/magenta/models/nsynth
Written inPython
TypeSoftware synthesizer
LicenseApache 2.0
Websitemagenta.tensorflow.org/nsynth

NSynth (a portmanteau of "Neural Synthesizer") is a software algorithm, outlined in a paper in April 2017[1], that generates new sounds through a neural network based synthesis, employing a WaveNet-style autoencoder to learn its own temporal embeddings from four different sounds.[2]. Google then released an open source hardware interface for the algorithm called NSynth Super[3], used by musicians such as Grimes[4] and YACHT [5] to generate experimental music using AI. The research and development of the algorithm was part of a collaboration between Google Brain, Magenta and DeepMind.[6]

Development

Dataset

The NSynth dataset is composed of 305,979 one-shot instrumental notes featuring a unique pitch, timbre, and envelope, sampled from 1,006 instruments from commercial sample libraries.[7] For each instrument the dataset contains four-second 16kHz audio snippets by ranging over every pitch of a standard MIDI piano, as well as five different velocities [8]. The dataset is made available under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. [9]

Model

A spectral autoencoder model and a WaveNet autoencoder model are publicly available on GitHub [10]. The baseline model uses a spectrogram with fft_size 1024 and hop_size 256, MSE loss on the magnitudes, and the Griffin-Lim algorithm for reconstruction. The WaveNet model trains on mu-law encoded waveform chunks of size 6144. It learns embeddings with 16 dimensions that are downsampled by 512 in time[1].

NSynth Super
The NSynth Super front panel: a metal box with a bright colored screen input.
NSynth Super Front Panel
ManufacturerGoogle Brain, Google Creative Lab
Dates2018
Technical specifications
Synthesis typeNeural Network Sample-based synthesis
Input/output
Left-hand controlPitch bend, ADSR
External controlMIDI

Nsynth Super

Nsynth Super is an audio synthesizer released by Google in 2018.[11] It is designed to provide musicians with an accessible interface to the Nsynth algorithm.[1] Design files, source code and internal components are released under an open source Apache License 2.0[12], enabling hobbyists and musicians to freely build and use the instrument.[13][14] The instrument includes features from notable artists, such as Grimes and Yacht, using Nsynth Super in their music productions.[15]

Hardware

I'm baby four dollar toast deep v subway tile small batch affogato. Celiac bitters cray post-ironic, DSA coloring book sustainable whatever. Cronut trust fund lo-fi, ugh flexitarian dreamcatcher paleo. XOXO mumblecore listicle man braid lomo poke blog. Photo booth cornhole mukbang edison bulb, put a bird on it 3 wolf moon

Influence

google IO / Grimes etc baby four dollar toast deep v subway tile small batch affogato. Celiac bitters cray post-ironic, DSA coloring book sustainable whatever. Cronut trust fund lo-fi, ugh flexitarian dreamcatcher paleo. XOXO mumblecore listicle man braid lomo poke blog. Photo booth cornhole mukbang edison bulb, put a bird on it 3 wolf moon

References

  1. ^ a b c Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". arXiv:1704.01279 [cs.LG].
  2. ^ "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". research.google.
  3. ^ "Google's open-source neural synth is creating totally new sounds". wired.co.uk.
  4. ^ "73 | Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll". www.preposterousuniverse.com.
  5. ^ "Music and Machine Learning (Google I/O'19)". youtube.com.
  6. ^ "NSynth: Neural Audio Synthesis". Magenta.
  7. ^ "NSynth Dataset". activeloop.ai.
  8. ^ A bot will complete this citation soon. Click here to jump the queue arXiv:1907.08520.
  9. ^ "The NSynth Dataset". tensorflow.org.
  10. ^ "NSynth: Neural Audio Synthesis". GitHub.
  11. ^ "Google built a musical instrument that uses AI and released the plans so you can make your own". CNBC. 13 March 2018.
  12. ^ "googlecreativelab/open-nsynth-super". April 1, 2021 – via GitHub.
  13. ^ Nast, Condé. "Google's open-source neural synth is creating totally new sounds". Wired UK.
  14. ^ "Open NSynth Super". hackaday.io.
  15. ^ "73 | Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll". www.preposterousuniverse.com.

Further reading

Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". arXiv:1704.01279 [cs.LG].



References

Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=NSynth&oldid=1119583626"