it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
It could be represented with a series of lookup tables, especially quantized LLMs. A series of inputs results in a specific output that gets passed to the next set of nodes. Repeat 7 billion times, and the final output from the last set of nodes is a set of token probabilities.
There are a finite number of possible inputs if the network is quantized. (And even if the network isn’t quantized, it just makes the possible input space much larger.) Especially if there’s been connection pruning for input nodes that don’t have a significant impact on the output.
Let’s say you have five inputs. For each possible input from node 1, you have a volume. Input is 3? Go grab volume 3. Node 2 relates to chapter; value is 10? Flip to chapter 10. Each chapter is a series of tables. Node 3 is for which table to look at, node 4 is row, node 5 is column. Voila, your five inputs have become a single output value.
Anything that is immutable and has a finite number of inputs and outputs can be represented by a lookup table.
That doesn’t sound anything like a neural network. Did you know that the chance of having a particular order of a deck of 52 cards shuffled 7 times is 8x10^67. What do you think the number of possible configurations is for a neural network of hundreds of millions of nodes?
? A neural network is not a lookup table.
it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
the linked paper is very good: https://arxiv.org/pdf/2309.10668
That’s like saying the neural network in your head has compressed your knowledge of language and just expands the relevant parts when needed.
yup
It could be represented with a series of lookup tables, especially quantized LLMs. A series of inputs results in a specific output that gets passed to the next set of nodes. Repeat 7 billion times, and the final output from the last set of nodes is a set of token probabilities.
I don’t understand how a network of weighted nodes could be represented by a lookup table.
There are a finite number of possible inputs if the network is quantized. (And even if the network isn’t quantized, it just makes the possible input space much larger.) Especially if there’s been connection pruning for input nodes that don’t have a significant impact on the output.
Let’s say you have five inputs. For each possible input from node 1, you have a volume. Input is 3? Go grab volume 3. Node 2 relates to chapter; value is 10? Flip to chapter 10. Each chapter is a series of tables. Node 3 is for which table to look at, node 4 is row, node 5 is column. Voila, your five inputs have become a single output value.
Anything that is immutable and has a finite number of inputs and outputs can be represented by a lookup table.
That doesn’t sound anything like a neural network. Did you know that the chance of having a particular order of a deck of 52 cards shuffled 7 times is 8x10^67. What do you think the number of possible configurations is for a neural network of hundreds of millions of nodes?
I never said it would be a reasonable size, just that it could be represented by a lookup table.
🤔 a lookup table with more entries than molecules in the galaxy - probably still not big enough