Talk:Hopfield network

Incomplete descriptions of Running

According to the description under the section Running, a node must be picked, after which the behavior is deterministic. This is a rather incomplete description. How is the behavior of the node defined after it is picked? I suppose it is updating or replacing the value s_i with it's activation a_i (according to the definition given earlier), but it would be better if this is stated explicitly. —Preceding unsigned comment added by 140.78.124.18 (talk) 17:06, 29 June 2009 (UTC)[reply]

I will re-write the running section, and explicitly detail the process Razvan Marinescu (talk) 12:08, 12 January 2013 (UTC)[reply]

Further Correction - symmetric weights

My error - I misread the context of his statement. The condition of symmetric weights guarantees that following the update rule makes energy a monotonically decreasing function, which guarantees convergence to local minima, however, non-symmetric weights do not seem to impare the use of the network as a content-addressable memory system. — Preceding unsigned comment added by Corvi42 (talk • contribs)

With non-symmetric weights, the attractors can be periodic or chaotic, which makes it less usable for retrieving a memory. Dicklyon (talk) 16:10, 28 March 2014 (UTC)[reply]

Correction - connections need not be symmetric!

If you refer to the origina Hopfield paper ( citied at the bottom of the page ) he discusses the performance of networks with the "special case" of symmetric weights, but says that the network performs just as well with non-symmetric weights. Specifically he says: "The flow in phase space produced by this model algorithm has the properties necessary for a content-addressable memory whether or not Tij is symmetric" (Hopfield, 1982, p. 2556) — Preceding unsigned comment added by Corvi42 (talk • contribs)

That's right. He says in the 1982 paper (and repeats in the ones from 1984 and 1986) that the weights should be more or less symmetric in order to converge. --Ben ^T/_C 14:27, 5 July 2007 (UTC)[reply]

Connection between Hopfield Net and Ising model?

Hello!

I had some classes this week which involved the definitions of Hopfield networks and Ising model, and came here to look for further information/links.

There is a link in this article to Ising model, but nothing is written in the article body that explains the connections between the two concepts, maybe someone could fill that gap in?

(I'll try after I've studied enough to understand the connection myself).

Cheers — Preceding unsigned comment added by 157.82.246.144 (talk • contribs)

The Ising model is a model of ferromagnetism. Atoms are bipolar (i.e. either positive or negative) and they have connections and local interactions of atoms can lead to some state transitions on a global level. They are the theoretical foundation of Hopfield Networks and Hopfield specifically mentions them and changes the atoms to McCulloch-Pitts neurons, i.e. he gives them a threshold. --Ben ^T/_C 14:33, 5 July 2007 (UTC)[reply]

See reference to Little in the main article. mcyp (talk) 20:38, 18 March 2021 (UTC)[reply]

Definitions?

The relation between the a[i]'s and the s[i]'s is not clear. Are the a[i]'s just the updated values of the s[i]'s? In that case, why not call them both s[i]?

Another terminological matter: The article says

Hopfield nets can either have units that take on values of 1 or -1, or units that take on values of 1 or 0.

and goes on to give the updating rules in the two cases. This seems like to much attention to a trivial matter of scaling. I would suggest choosing one convention or the other for the article and then mentioning that the other convention is also used. --Macrakis 16:21, 15 August 2006 (UTC)[reply]

I also don't find it that important, whether they are ${0,1}^{n}$ or ${-1,1}^{n}$ . But I find it important that units can be also continuous. Bipolar units are only one particular case studied in Hopfield's papers. --Ben ^T/_C 14:37, 5 July 2007 (UTC)[reply]

I agree this is quite ambiguous, and the case with 0 and 1 is actually mathematically more complicated (for example, I believe the Energy function would first of all be different and even more complicated ... -1 and 1 provides some desirable symmetry that symplifies mathematical calculations and proofs of convergence, training rules). I will change that to -1 and 1, explicitly say that the article uses this convention .... and only mention the other convention. Razvan Marinescu (talk) 12:18, 12 January 2013 (UTC)[reply]

Energy formula

Currently energy is written as:

E=-{\frac {1}{2}}\sum _{i<j}{w_{ij}{s_{i}}{s_{j}}}+\sum _{i}{\theta _{i}\ s_{i}}

I feel this is incorrect. Either removing 1/2

E=-\sum _{i<j}{w_{ij}{s_{i}}{s_{j}}}+\sum _{i}{\theta _{i}\ s_{i}}

or summing over all $i$ and $j$

E=-{\frac {1}{2}}\sum _{i,j}{w_{ij}{s_{i}}{s_{j}}}+\sum _{i}{\theta _{i}\ s_{i}}

would fix the problem. But I'm not so confident to modify the main text. I'd appreciate if somebody could check it. — Preceding unsigned comment added by 61.213.69.186 (talk • contribs)

i agree, and i've changed it, (before looking here) I T.A a neural networks course... you can easily see this be derivating E w.r.t S_j to get h_j — Preceding unsigned comment added by 128.139.226.37 (talk • contribs)

right. --Ben ^T/_C 14:38, 5 July 2007 (UTC)[reply]

energy

I have an argument on energy function. Sometimes threshold is a more complicated function and we cannot easily incorporate it into Energy function. I mean as I have seen in "Associative memories - the Hopfield net", it should not contain this term:

\sum _{i}{\theta _{i}\ s_{i}}

Am I right? — Preceding unsigned comment added by Amshali (talk • contribs)

Name of article

Would it not be more encyclopedic for this article to be entitled Hopfield _Network_, rather than Hopfield _Net_? The reasons should be obvious. Opinions?65.183.135.40 (talk) 06:21, 6 March 2008 (UTC)[reply]

Agree. Posted a request. Qwertyus (talk) 12:12, 6 June 2011 (UTC)[reply]

Associative memory: Terminology and cross-refs

I am confused and the Wikipedia is currently not in a very helpful state regarding this:

Is "Content-addressable memory" synonymous to "associative memory" (as indicated here) or is it a term specifically describing a kind of memory hardware (as indicated in the article by that title)? Is a Hopfield network also an "auto-associative memory"?

For right or wrong, I felt an urge to make the following modifications:

1) I modified the crossref to "Content-addressable memory" to point to the more general "associative memory" (disambig); this I did because the article on CAM is too specific and (for the time being) not appropriate to reference from here. Depending on the correct answer to my confusion question, perhaps it is the CAM article that should be generalised to make it compatible with the ANN context?

2) I also added See also: Associative memory and Auto-associative memory.

195.60.183.2 (talk) 18:19, 17 July 2008 (UTC)[reply]

Training and Limitations

As it stands right now, the article doesn't say how one can train Hopfield nets to make particular inputs become local minima. It also doesn't say whether you can have two Hopfield nets with the same local minima but different "basins of attraction".

Also, I assume there is some limit to the number of minima that a network can be trained to (I would guess something like the logarithm of the number of nodes, or something similar) but the article doesn't say anything about this. If there is in fact no limit, it would be good to mention that.

120.18.115.209 (talk) 06:39, 15 October 2009 (UTC)[reply]

Binary treshold unit = Perceptron

Perhaps the Binary treshold unit section can be merged with perceptron —Preceding unsigned comment added by StookWagen (talk • contribs) 10:39, 2 December 2009 (UTC)[reply]

A perceptron and a hopfield net differ by the shape of their network: the perceptron is feed-forward whereas hopfield nets are recurrent. So it would probably be missleading to link the two of them. --Toukip (talk) 04:28, 16 November 2010 (UTC)[reply]

Also, the Hopfield net can use any kind of nonlinearity, not just a threshold. So saying a Hopfield net is a symmetric recurrent interconnection of perceptron units is not general enough. Dicklyon (talk) 16:19, 28 March 2014 (UTC)[reply]

Picture

Arows in the picture (A Hopfield net with four nodes) have wrong direction. --Bojan PLOJ (talk) 11:37, 6 March 2010 (UTC)[reply]

Indeed, the picture seems very wrong to me, the connectivity does not seem right and it is clearly not the prototypical graph that is used for an Hopfield network. I'll try to modify that when I have time... Piero le fou (talk) 14:12, 28 March 2014 (UTC)[reply]

The arrow direction is not the problem. The problem is that where lines converge, it's not clear that it means a summing node; some of the arrowheads mean weights, and some don't. It's an unusual way to draw it; usually weighting and summing are localized to the neuron. Dicklyon (talk) 15:44, 28 March 2014 (UTC)[reply]

The main problem with the illustration

is that it's completely unintelligible to the average interested Wikipedia reader. There is no discussion of the meanings and uses of the various symbols in it, not even a simple key to name those different symbols. yoyo (talk) 02:06, 26 November 2016 (UTC)[reply]

Initializing the Network

I tried using this article (and some other sources) to create a hopfield net, but I found, in addition to the lack of notes on how to train the network, it is very unclear how the network ought to be initialized. Perhaps psuedocode describing the algorithms involved would be helpful? Chekhov.seagull (talk) 19:29, 8 March 2010 (UTC)[reply]

I can write about this. A Hopfield network is first of all trained with patterns that fix the weights. For initialisation, the user needs to set the values of the units to the input pattern. Razvan Marinescu (talk) 12:08, 12 January 2013 (UTC)[reply]

Inputs/outputs?

Is it correct to say in a Hopfield net, unlike more general recurrent NNs, all nodes are both input and output nodes? It would be nice if the article said something explicit about this. 71.95.146.73 (talk) 18:36, 30 July 2011 (UTC)[reply]

This is true, but by no means the only difference. What make Hopfield Networks special are also the following features:

They are are reccurent and converge to attractors
The — Preceding unsigned comment added by Mrazvan22 (talk • contribs)

New Results

What about this recent work: http://arxiv.org/abs/1206.2081 113.190.172.15 (talk) 14:40, 19 September 2014 (UTC)SOC[reply]

Training - Normalization not needed?

Currently the rule for learning is: $w_{ij}={\frac {1}{n}}\sum _{\mu =1}^{n}\epsilon _{i}^{\mu }\epsilon _{j}^{\mu }$ where $n$ is the number of learnt patterns.

Why would the normalization (dividing by $n$ ) needed? Many don't use it, including Hopfield himself in the original 1982 paper (eq. 2 there).

It is only needed to avoid 'very large' weights when storing 'a lot' of memories. (a lot compared to the capacity of the actual computer's memory).

If true, I feel that the normalization can be removed for distilling the essence (and essential parts) of the algorithm, and be mentioned as an alternative, and not as the definition.

Capacity

Using ~0.5N^2 weights (floats) to remember ~0.138N^2 bits does not seem to give an advantage (at least in size) maybe a discussion addressing performance would be helpful. — Preceding unsigned comment added by 84.75.58.165 (talk) 15:49, 7 August 2017 (UTC)[reply]

External links modified

Hello fellow Wikipedians,

I have just modified 2 external links on Hopfield network. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:

When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.

This message was posted before February 2018. After February 2018, "External links modified" talk page sections are no longer generated or monitored by InternetArchiveBot. No special action is required regarding these talk page notices, other than regular verification using the archive tool instructions below. Editors have permission to delete these "External links modified" talk page sections if they want to de-clutter talk pages, but see the RfC before doing mass systematic removals. This message is updated dynamically through the template {{source check}} (last update: 5 June 2024).

If you have discovered URLs which were erroneously considered dead by the bot, you can report them with this tool.
If you found an error with any archives or the URLs themselves, you can fix them with this tool.

Cheers.—InternetArchiveBot (Report bug) 17:12, 6 November 2017 (UTC)[reply]