Discuss AI Here

[Svenska]

But I posit that if you don't have the ability to do simple logical analysis, you don't have intelligence.

I never argued about them having "intelligence", my discussion point was always about being useful. When comparing with humans as a baseline, I specifically excluded experts in the fields. In my opinion, AI and LLMs already are useful in some areas and it looks like their usefulness will increase in the near future. The business model aspect I don't care about.

Nothing prevents an LLM working as part of a bigger system. Humans use computer programs to support their thought processes, why can't LLMs do the same? Teach them to use equation solvers and you get a more efficient solution than teaching the LLM to solve equations.

If you're a programmer you can replace the LLM with another program running on that machine that can do things, such as logical analysis, that the LLM software is incapable of, but then you're no longer running an LLM. You can't train an LLM to do logical analysis any more than you can use Wordstar 3.0 to brighten a JPEG image.

LLMs are just heavily data-driven algorithms. Nothing prevents them from being an (inefficient) equivalent to a JPEG brightener. The opposite is true: multi-layer perceptrons (which are the foundation of current-generation LLMs) are equivalent to other programs. You can build Turing machines with them, therefore you can also build a logic analyzer with them.

I don't see why you think that whould be impossible.

LLMs are not deterministic: that's a key part of their design.

Well, no. We intentionally inject randomness into the algorithm to create varying output. Nothing requires us to do that, it's an active decision by the environment. By default, LLMs are deterministic. I've done performance measurements using llama.cpp on different machines, and of course I use a fixed seed. If ChatGPT's API does not allow you to do that is their business decision, not a fault with LLMs in general.

It's not going to change the situation that nobody can tell you why an LLM produces a particular answer.

It's not impossible, just infeasible. The matrices are too large. On the other hand, nobody can really tell you why your Windows machine behaves the way it does today either. Clarke's third law applies here as well.

 

[Kouji Matsui]

I've found an interesting thread...

I'm going to share my thoughts here because I don't hear many reasons that are close to my opinion in terms of insights about AI (LLM) systems around me.

When ChatGPT was starting out with a bang, I was quite skeptical about AI systems (I still have my doubts). I understand the concepts and failures of “expert systems” in the 20th century. In addition, I understand the AI of that generation, that is, the realization by the perceptron network, and the basic structure, but I know the fact that it could not evolve.

It has been 30 to 40 years since then, but it is true that ChatGPT is blowing away the impression of that time. While thinking that it might be an illusion, I felt “something like consciousness” and felt surprise and a kind of fear. Well, if we kept on playing with it, we could see that it was still just a toy system...

Not long after that, GPT4 was released. The basic responses had improved a lot, and we were surprised and scared again. As for the fear, by this time I had a clear understanding that it was not so-called Skynet, and that it was undesirable for this to be the monopoly of a single company.

Of course, I also understand that it costs a lot of money to run an AI system (using a large number of ultra-expensive GPU systems for learning and operating a prompt system), but I was worried about the future because OpenAI had made such a good start.

There are also other aspects to this, but now I'd like to return to the technical side of things.

After GPT4 was released, people like me who were interested in the limits of this AI system and wanted to explore them began to appear. Among them, there was someone who was giving a very interesting session. This person is a mathematician and is well known in Japan for being able to write humorous documents.

https://gist.github.com/hyuki/5f6795852060fbf2ff6021b6856aa00a (in Japanese)

This talk is about how ChatGPT can provide insight into what is causing a contradiction when given a contradictory state due to a combination of graph structures. Watching this session, it seems like an ineffective AI system for a while, but the impact when it finds a contradiction is great.

I think this is not a partial copy of a paper from somewhere, and it is a result that cannot be obtained unless the inference is correctly guided. Watching this session, I couldn't help but feel the potential of AI technology. At the same time, I think that a kind of “prompt engineering” is also important here, where we have to carefully choose what we say during the session in order to get the desired result.

(This is also true when discussing with other humans.)

This is a bit of a tangent, but as a personal opinion on the future of AI systems: I think it will be faster for the world to be disrupted by unintentional information manipulation through AI systems simply being connected to social networking services before Skynet has a human-like ego... (Maybe that will happen soon)

Having said that, as I continued to use GPT4 myself, I eventually realized that it was also capable of answering questions based on a large amount of information sources with high accuracy, but would only give garbage-like answers to those that were not.

For example, I can understand and write C/C++/C#/F#/TypeScript. So, when I ask it to answer the same subject code in these languages, it outputs obviously random code in the case of F#.

Similarly, if I ask them to write code that is very narrowly constrained, such as old x86 boot code (loaded from the 512-byte boot sector), they will still produce code that needs a lot of reworking, and they will present it as if it were perfect. And the more I give them feedback on the code, the more quickly they forget about it and make the same mistakes again.

I got fed up with this, so when I got an undesirable answer, I would go back and ask the question again, and then have them answer it with a more conservative and careful content. Without even thinking about it, the information on “old boot code for x86” is so different from the information sources on “TypeScript” that I think this result is unavoidable. Even during the session, they kept mentioning old boot code for Linux.

More recently, the GPT4o1 model was released. This o1 model also surprised me in a different way, in that it appeared to be iterating (repeating) the inference process in a hierarchical manner.

The iterations appeared to be a process of checking for any inconsistencies before giving a wrong answer as much as possible. The interesting thing was that when it came to seeking answers to very complicated questions, it seemed that the process of generating an answer --> it's wrong --> generating an answer --> it's wrong, would repeat itself over and over again without ever coming to an end. Eventually, in the middle of this process, something like a 'fluctuation' would occur in the decision-making, and this would become the breakthrough that led to the true answer.

If each reasoning process could become 'smarter', it seemed that the level of practicality would increase. Moreover, this is similar to two or three people repeatedly discussing things, and it may be ideal for making advanced decisions. However, at the same time, if a certain level of context and consensus (?) is not shared, it seems that the exchange of conclusions will be meaningless. I think that these concerns have been resolved because this method has been adopted, but it is interesting.

I am interested in creating my own computer language, and I have been asking questions about the type system of the implementation and implementing it using the o1 model, but for relatively simple type systems, the implementation and corresponding test code also output valid results, including the validity of the theory (I have specified TypeScript just to be sure). However, when I gradually delve into complex type systems, I sometimes see the reason for the answer at a certain point.

The reason for this is that in the field of research into type systems, Haskell and OCaml are the mainstream languages used, and they are being pulled in the direction of the topics (i.e. papers) being discussed in these languages.

In computer language science, the superficial syntax is replaced by an abstract data structure called AST during the input process, and it deviates from the familiar syntax. However, the data structure ends up looking exactly like Haskell, and the flow of the type system discussion ends up concluding with a deep insight into the OCaml type system, and it easily moves away from the discussion of the “original” type system that I am thinking about.

For this reason, I have the feeling that ChatGPT and other AI systems are “quite” influenced by the amount and quality of information that was used as input for the learning. I feel that TypeScript is more fluent than F# in expressing results, and that it is necessary to make considerable efforts in terms of what we say in order to draw out discussions about theories that they have never heard of before.

 

[cjs]

...the superficial syntax is replaced by an abstract data structure called AST during the input process, and it deviates from the familiar syntax.

Tell that to a Lisp programmer. :-)

 

[Zare]

I never argued about them having "intelligence", my discussion point was always about being useful. When comparing with humans as a baseline, I specifically excluded experts in the fields. In my opinion, AI and LLMs already are useful in some areas and it looks like their usefulness will increase in the near future. The business model aspect I don't care about.

Nothing prevents an LLM working as part of a bigger system. Humans use computer programs to support their thought processes, why can't LLMs do the same? Teach them to use equation solvers and you get a more efficient solution than teaching the LLM to solve equations.

This is quite reasonable and pragmatic view to have.

Interestingly defined "thought process" - I see technology like search engines and LLMs having a profound impact here. Google did.
In their case, 20 years after, people with brains now have more brains, because they can easily lookup the next detail they are after, people with less brains now have even less, because they don't even need to memorize an address or a simple piece of data. Google's impact on the data sources wasn't profound, the number of invalid sources has risen due to the reward system in search indexing, but the valid ones are still there.

It's interesting to talk about the 3rd IMO...everyone is talking about how experts and layman could use it, what about the "logistic chain". I mean we don't try to boot dirty physical technologies that would destroy nature nowadays, do we? Discussion about how LLMs gather data and what is the impact of that system on the data sources even if the sources aren't IP protected is IMO a discussion worth having.

So my input to the discussion would be, why should I or we, people that put their mind on forum, approve of this model? Of our thought process being harvested for automatic usage, by a corporation probably?

What is actually already happening is that sources are diminishing because LLM's direct interpretation doesn't allow the economic WWW reward model of visits and page hits. Good posters on Stack Overflow have their own personal benefits behind the work they do, Stack Overflow has business benefits of providing the platform to them, and it all depends on the user actually landing on SO.

In near future they will be able to generate decent quality deepfake/CGI combo videos that depict a procedure, like cooking a meal, superseeding YT cooking channels real people do. This won't kill the best channels of course, but it will raise the entry level of a "human standard". Nobody will make videos that LLM is able to make, if you make one, it needs to be better. And when you publish it, who says LLM will not be able to harvest your sauce in a law-bypassing manner? The transformative and automatic nature enables them to derive a 100th derivative iteration from the original source, enabling them to retain a resulting characteristic in end product while losing track of the original in the mix? A picture's wow factor might be in the play of lights/contrast, LLM might steal that in a way author cannot claim he has been copied.

I know you said bussiness aspect is off discussion but I'm afraid we cannot decouple them yet. The entire "AI" craze is bussiness-driven. It's not like MP3s or streaming that people just jumped into and now the industry needs to adapt. It's the industry players pushing it, under the narrative of a wild, organic growth, that other industry players need to adapt to or they will be left behind.

 

[cjs]

I never argued about them having "intelligence", my discussion point was always about being useful.

Ok, you never made that sufficiently clear up to this point. It sounds as if you're essentially in agreement with me then, so I don't know why we've been having such an argument.

LLMs are just heavily data-driven algorithms. Nothing prevents them from being an (inefficient) equivalent to a JPEG brightener.

It sounds as if you're talking about a completely different trained network that wouldn't be an LLM any more.

At any rate, if you're saying that LLMs are the same old trained networks we've had all along, just producing a superficially more impressive result (at much greater cost than we've spent on them before), we're also in agreement.

It's not impossible, just infeasible. The matrices are too large. On the other hand, nobody can really tell you why your Windows machine behaves the way it does today either. Clarke's third law applies here as well.

Clarke's third law applies in neither of these cases, except perhaps to people who don't understand computers or computer programming. To someone who does understand these; there's nothing even the slightest bit magical about Windows and the only thing magical about LLMs is how what goes on in the brains of humans to let them be so convinced and entranced by them. (As has happened since ELIZA.)

And yes, people can really tell you about why your Windows machine behaves like it does. In many cases they can simply tell you how they programmed it. There are more complex and mysterious cases, of course, where you may have to go so far as to give them the machine to examine, but all of these are in principle debuggable and are often debugged.

The LLM is an entirely different situation from even the most complex human-written code. Even for small networks, while you might be able to trace through the network to see what information went to a node, how it was transformed, and what came out and where that went, that does not mean you know why, in any reasonable sense of the word, the LLM came up with some output. If you knew why, you would be able to tell me what to change on the training inputs to change that output, and what else it would effect.

This is why we say LLMs are "trained," and not "programmed"; what they will do is essentially a crapshoot as far as any individual input to output mapping, which is exactly unlike regular programming.

 

[Agent Orange]

Ask yourself if you think AI will ever be able to "reason"? Will it be able to make decisions based on morality? Who lives who dies etc? Like HAL-9000.

 

[jlang]

Ask yourself if you think AI will ever be able to "reason"? Will it be able to make decisions based on morality? Who lives who dies etc? Like HAL-9000.

Not with the current LLM systems. That path won't lead to AGI ever.
HAL had his own problems to deal with.

 

[Eudimorphodon]

The LLM is an entirely different situation from even the most complex human-written code. Even for small networks, while you might be able to trace through the network to see what information went to a node, how it was transformed, and what came out and where that went, that does not mean you know why, in any reasonable sense of the word, the LLM came up with some output. If you knew why, you would be able to tell me what to change on the training inputs to change that output, and what else it would effect.

This is why we say LLMs are "trained," and not "programmed"; what they will do is essentially a crapshoot as far as any individual input to output mapping, which is exactly unlike regular programming.

Here's a five year old blog entry that's about applying "deep learning" models to the problem of artificial vision, but the situation it describes is essentially the same as how language-focused LLMs work, and why you can't meaningfully compare the process they use to "reasoning" as we (mostly fail to) understand it in biological systems. A trained model can, when it's had enough labeled training data to work with and gone through (Carl Sagan voice) "Billions and Billions" of cycles of sorting and reinforcement achieve some pretty amazing results in terms of producing "statistically correct" answers, at least for questions that are closely constrained to examples within the dataset it was trained on. But there is *nothing* like "reasoning" going on here.

The example in the blog entry is about creating a program that can tell the difference between a cat and a dog, and how the "deep learning" approach will almost certainly end up creating a transformation matrix that depends heavily on attributes that don't have anything to do with the actual "things" cats and dogs. IE, in the end this model still has absolutely no idea what a "cat" or a "dog" are, it just has a mountain of statistical data about the kind of pixel patterns are more common in pictures of those two kinds of things, and, as the blog points out, because they don't *know* what anything in the pictures actually *are* they can be easily tricked into thinking a cat is a dog simply by sprinkling a few "dog pixels" into the photo you're asking it about. The theoretical example he uses is putting a set of dog tags on a cat, but in the real world attacks on these models can be far more subtle than that. Here's an old paper about breaking a computer vision program trained to recognize street signs into completely wiffing on stop signs just by pasting a few rectangular stickers to the sign. You might object that the example there is from 2018, but these are fundamental problems with the approach. Five years later it still doesn't take much to poison Stable Diffusion so it'll make you a cat when you ask for a dog, or a cake instead of a hat.

(On the subject of advisarial attacks against deep learning models, I absolutely applaud the efforts of researchers who are creating tools like Glaze and Nightshade, which apply subtle transformations to human generated imagery which, like a piece of tape on a stop sign, humans usually wouldn't even notice but actively damage AI image models if they're inserted into their training data. Everyone should be using these tools if they're putting their work on the Internet. Full stop.)

I mean, clearly there are a lot of AI fans in this thread and, sure, I get it, the epistemological limits of LLMs aren't going to matter to them a whole lot as long as the output they get from it is "useful". For tasks like rote computer programming your chances of being "statistically correct" with a model like this are probably especially good because by definition programming is a "constrained" subject (IE, computer languages operate with formal syntax that even pretty rudimentary pattern matching systems should be able to pick up on) and, well, ChatGPT has ingested the entirety of GitHub to pirate from and play madlibs with. BUT... as is painfully clear from the OP of this thread, the fact that it chucks out pretty usable code a lot of the time does not mean that it "understands" it in the sense that a human does. ChatGPT doesn't "know" what a carry flag is or does; it's an object in the weighted word cloud that it operates from and it only knows what words are likely to appear near it.

There's been a number of hurtful comments about babies and bricklayers in this thread, but here's the thing about ignorant humans: if they have reasonably normal IQs the process of teaching them something is *completely and utterly different* from training an LLM. If you want to teach a child to tell the difference between a cat and a dog you're not going to have to show them a thousand labeled pictures of both only to be disappointed when the kid sees one of these animals from the wrong angle and declares it to be a "street lamp", it's only going to take a (remarkably small) number of examples to get this toddler's inherent symbol processing engine to tune internal classifiers that are remarkably accurate at distinguising the concept of "cat-ness" from "dog-ness". Obviously untrained humans don't have a similar instinctual understanding of computers like they do for animals, but if I take a bricklayer of average intelligence and pay him enough money to make it worth his time he's going to be be able to be walked through it, and the training process likewise is not going to involve sitting them down to read an entire library's worth of 8080 documentation and code and expecting their brain to form a vast statistically weighted word cloud from it. Humans (and animals, for that matter) use *completely* different symbolic and semantic models for learning; teaching that bricklayer how to code is going to largely involve references/analogy to concepts they already know, and by the end of it they're going to be conceptualizing the flag register in the CPU as a thing that has a state, and this state is affected by the operation of the CPU in predictable ways. (Which if you now continue to gainfully employ the bricklayer writing 8080 code he'll continue to incrementally refine his mental model of that thing based on discrete processing of facts and experiences, verses having to digest even *vaster* word clouds to adjust the weighting of his responses.) There's a "mental image" there that an LLM just doesn't have, and, because of the inherent limits of the approach, will *never* have...

... Which again, the response will be "but they work well enough to be useful so it's still the future". And, well, sure, I imagine this crap isn't going away, there are things it can do. But it's also true, right now, that this current hype cycle is incredibly dependant upon models made out of vast quantities of stolen copyrighted material and propped up by massive infusions of venture capital, which ultimately is very unlikely to be sustainable. Maybe the people pushing this are going to win the lawsuits about the intellectual property theft, and hey, maybe they'll even get their nuclear reactors to power it with, but at a certain point maaaaybe it's worth taking a step back and asking if this is actually making anyone's life better in ways that couldn't be acheived far more efficiently and equitably. But whatever.

 

[cjs]

Brilliant post, @Eudimorphodon, and especially thanks for those links. I have a couple further comments.

...by the end of it they're going to be conceptualizing the flag register in the CPU as a thing that has a state, and this state is affected by the operation of the CPU in predictable ways.

Further, looking at my example in the lead post, if they have any reasonable amount of intelligence at all, regardless of their level of understanding of the 8080, when I explain to them that POP PSW affects the carry, they will immediately stop using it in a place where they need to have preserved the previous value of the carry; the option to use that at the end of the routine in question will be entirely off the table. That's something that Copilot was entirely unable to do; even when I finally got it to agree in English that POP PSW affected the carry, it still kept putting that at the end of the code and claiming that it didn't affect the carry.

For tasks like rote computer programming your chances of being "statistically correct" with a model like this are probably especially good because by definition programming is a "constrained" subject...

It seems very likely that LLMs will indeed help programmers write considerably more code as time goes on. The sad thing about this is that it's not improving the world, it's making it worse, and probably making programmers worse. Programs turning into massive amounts of boilerplate code has long been a problem; being able to generate boilerplate with less effort is an anti-solution: it just leaves you with more of the problems caused by boilerplate code in the first place, such as being unable to find bugs in all that mess, and making it hard to understand what the code is doing.

And LLMs seem to me simply incapable of fixing this problem, because the problem itself is about removal of statistically common things with small variations, replacing them with far less common things (essentially, data compression). Which is pretty much the opposite of what an LLM does: give you variants of common things.

 

[Eudimorphodon]

That's something that Copilot was entirely unable to do; even when I finally got it to agree in English that POP PSW affected the carry, it still kept putting that at the end of the code and claiming that it didn't affect the carry.

That aligns with some of the conversations I’ve had with Copilot myself, and I assume what’s happening here is something along the lines of whatever dynamic weights the chat engine pieces together from its limited and ephemeral context window are just hopelessly overmatched when compared to the baked in training. Maybe in a few months OpenAI will scrape this conversation and through sheer luck will end up tipping a scale that makes it slightly more likely to give a right answer. But… yeah, it’s still not going to “know” what it’s talking about any more than it does know.

It seems very likely that LLMs will indeed help programmers write considerably more code as time goes on. The sad thing about this is that it's not improving the world, it's making it worse, and probably making programmers worse.

The really terrifying thing about letting LLMs and deep learning write code is that people will be crazy enough to trust their lives to it. Some people already are.

 

[cjs]

The really terrifying thing about letting LLMs and deep learning write code is that people will be crazy enough to trust their lives to it. Some people already are.

Well, this one I don't feel quite so bad about because it's no more than a slightly worse version of what we have already. I've worked for plenty of managers who don't actually care if the code works: their priorities were that they checkbox on their spreadsheet saying "feature done" gets checked off on time, followed by (though not as important) that the feature appears to work at a casual glance. I'm sure some companies doing safety-critical systems are at least in part doing their code the same way.

For most serious work on safety critical systems, LLMs won't be a problem because the code is carefully reviewed to see that it matches the (also careful) design, so it doesn't really matter where the code comes from: poor code will get fixed. You don't trust the source, you trust the review process.

 

[Zare]

A senior reviewing that poor code shouldn't be happy to find out a junior used a LLM.
If you come up with a bad solution, you had a through process behind it, that process should be evaluated and fixed too. "Why did you think this is a proper way?" Through that talk you can fix yourself and the code.

In LLM case, "Why did you think this is a proper way" yields the "I didn't, ChatGPT told me" honest answer.

 

[Svenska]

So my input to the discussion would be, why should I or we, people that put their mind on forum, approve of this model?
Of our thought process being harvested for automatic usage, by a corporation probably?

I don't think it matters whether we approve of specific models (or AI in general). That decision has largely been taken by others already, for all kinds of purposes. Given the very important personal impact into lives of, for example, disabled people, even governments will be reluctant to blanket-ban the whole thing.

Make your thought process public and you have to live with the fact that it might be read by others. Make it machine-readable and you must expect it to be read by machines. That's a contract we've been forced upon since search engines - Google specifically - have started to harvest the world.

I strongly disapprove of machine-only governmental processes. Sure, I do quite trust both my banks and my governments (they hold sway over me as a person either way, but at least most EU countries have strong laws to deal with that). But even in a decentralized, well-planned and working governmental structure, the data (which is both value and danger) ends up in the hand of very few companies. Those I don't trust, because they have strong inventives to violate or change regulations in their favor, and there are far less enforceable checks in the first place.

Currently, I am more concerned about the "everything must be an app" mentality than AI in general. The latter will be integrated into the former, which merges them. On it's own, AI/LLMs are just technology and they will be used, abused and misused, like most other things. The scale and speed are concerning.

In near future they will be able to generate decent quality deepfake/CGI combo videos that depict a procedure, like cooking a meal, superseeding YT cooking channels real people do.

Yes. Not much we can really do about it. For now, we have AI wars; Eudimorphodon gave some great examples. Long term, we will see legislation to deal with the fallout. After all, copyright or intellectual property law in general came and changed with technology. People will sort it out eventually, and apart from voting, there's little to be done by me on a bigger scale.

And when you publish it, who says LLM will not be able to harvest your sauce in a law-bypassing manner?

The EU tries to balance different perspectives to become a blueprint for the world to follow. It has proven to be able to do that recently (be it the enforcing of Micro USB or USB-C, or the 5+ year support for phones). Whether it will succeed, time will tell. Right now, we're in a wild west situation with companies and countries wildly differing in how they see the world.

I don't expect any global solution for at least 10 years.

I know you said bussiness aspect is off discussion but I'm afraid we cannot decouple them yet. The entire "AI" craze is bussiness-driven.

The reason I don't want to discuss the business aspect is twofold: First, there is no meaningful any any of us can influence the outcode. We can all hate it together, but that won't change the result. Might as well focus on something else. Second, the people who are driving AI now are the same people who drove (and probably laundered money through) crypto before. To me, these are not people with a great ethically or morally sound compass, but they have the money. The US as arguably the largest influencer in the world values money above all [note: this is my distinctly European perspective; my last visit to the mainland US predates 9/11], so I have quite low expectations of the near future.

Even if the AI hype train ends without great results - like crypto - these people will move on to destroy something else. They now have the money and the compute to do so.

Ok, you never made that sufficiently clear up to this point.

The reason I specifically don't talk about whether AI/LLMs are "intelligent" is because intelligence itself is weakly defined. For most people and even professionals, it's more a case of "I know it when I see it" and as we see with LLMs, that doesn't work well.

I might make hopefully be clear that I don't see the current AI hype as particularly good. Not because LLMs are bad, but because we are using them in places we really shouldn't. Having to convince a badly-trained chatbot that it is incompetent in order to get any meaningful support is just plain bad, but that is what we will see a lot more of. I do hope that at least some companies are smart to not go all-in (amount of hope is limited), or that some smart companies will emerge after the inevitable crash (amount of hope is also limited). Time will tell.

It sounds as if you're talking about a completely different trained network that wouldn't be an LLM any more.

Throughout the discussion, I have conflated LLMs and other forms of AI on purpose. I don't see them as different technologies; it's the same technology adapted for different use cases. The network architectures are different, sure, but both a PDP-11 and a G4 PowerPC are "computing devices". Vintage computing has the advantage that the past is largely static and dominant, and that we largely are domain experts in that area - but not in AI.

Nobody prevents you from training an LLM using JPEG atoms as tokens and brightened images as input data. If trained well, it will output brightened JPEGs - but would it still be an LLM?

At any rate, if you're saying that LLMs are the same old trained networks we've had all along, just producing a superficially more impressive result (at much greater cost than we've spent on them before), we're also in agreement.

I think we are getting substantially more impressive results, but I do generalize to other somewhat hyped AI areas. With specialized hardware to run inference on (modern smartphones for example), they will become substantially cheaper to run.

I've seen some interesting takes on using analog computing technologies to reduce the costs further, for example. Some 15 years ago I took classes an neural networks, and one important takeaway with classic multi-layer perceptrons is that more but low-quality neurons can provide equivalent results. Biological neurons are far from the quality we expect from digital systems, so there's precedent in reality.

And as I mentioned before, computers were incredibly rare and expensive to run for somewhat better results than a room full of people doing math. That has changed substantially since, so I don't see this as long-term issues.

To someone who does understand these; there's nothing even the slightest bit magical about Windows and the only thing magical about LLMs is how what goes on in the brains of humans to let them be so convinced and entranced by them.

There is certainly no magic in Windows. But it's such advanced technology (read: complex pile of poo) that it might as well be.

And yes, people can really tell you about why your Windows machine behaves like it does. In many cases they can simply tell you how they programmed it. There are more complex and mysterious cases, of course, where you may have to go so far as to give them the machine to examine, but all of these are in principle debuggable and are often debugged.

In theory, everything is debuggable. In practice, it's not - I've worked with a great deal of complex Android-based stuff and while a lot of understanding can be gained (sometimes at great cost), a fair share of issues are simply intractable to deal with for various reasons. Hand it to the supplier and hope that they can deal with their own technology. Experience says that they are not always able to.

Multiply that by the amount of interconnected dependencies across global networks and not a single person in the world can tell you why. The systems we are currently building in the tech sector are of such complexity to exceed any single human mind. Whether that's intentional or not, I don't know.

The LLM is an entirely different situation from even the most complex human-written code. Even for small networks, while you might be able to trace through the network to see what information went to a node, how it was transformed, and what came out and where that went, that does not mean you know why, in any reasonable sense of the word, the LLM came up with some output. If you knew why, you would be able to tell me what to change on the training inputs to change that output, and what else it would effect.

I somewhat agree. But I see many purely technical systems where even their designers look at them like mostly black boxes. (I certainly do for some of my own creations from years ago, when I was even more inexperienced than I am now.)

On an institutional level, things are different - after the original designers leave or die, there simply might be nobody left who even remembers the why in the design. Answering any question of behaviour tends to get as hand-wavy as discussions about LLM results: You can trace the values through the system, but nobody can explain why.

 

[Svenska]

Ask yourself if you think AI will ever be able to "reason"? Will it be able to make decisions based on morality?

Probably about as good as humans under stress. Statistics show that humans under stress tend to revert to their training or, if that is not directly applicable, take basically random decisions. (At least that's what I know from pilots.)

The ability to reason in LLMs is currently a big research area, with some promising results (and a lot of failures, as expected of early research), so my answer to your first question is "I don't see why not". As for morality... when I look at the current state of the world, I see a lot of conflicting moralities out there. We might want to start agreeing on that ourselves first, then start asking whether a computer can do it.

I mean, clearly there are a lot of AI fans in this thread and, sure, I get it, the epistemological limits of LLMs aren't going to matter to them a whole lot as long as the output they get from it is "useful".

I don't see myself as AI fan, just realistically seeing the wave coming. (Just to be clear.)

There's been a number of hurtful comments about babies and bricklayers in this thread, but here's the thing about ignorant humans: if they have reasonably normal IQs the process of teaching them something is *completely and utterly different* from training an LLM.

Yes. LLMs are taught results, not processes. The training efficiency is also very low in its use of data. Also, the really large systems are usually only sent once through their training datasets. I don't even want to think about how often I repeated some tasks until I was able to do them reliably...

There's a "mental image" there that an LLM just doesn't have, and, because of the inherent limits of the approach, will *never* have...
... Which again, the response will be "but they work well enough to be useful so it's still the future".

Cooks don't need to know the details on how the chemicals they use decompose in the process of making food... the mental image of any single person is also limited. It's just a lot wider in general - which is, incidentally, why we use schooling to [try to] give children an education while their brains are still flexible.

Maybe the current approaches won't be able to get the mental image, and maybe we will figure out that the current training algorithms just are unfixable and end up with another AI winter. Right now, it doesn't look like it yet - but we are inside a hype train. Compared to the blockchain/bitcoin hype, I see real applications, so it might not fizzle out the same way.

The really terrifying thing about letting LLMs and deep learning write code is that people will be crazy enough to trust their lives to it.

Amen to that. Covid vaccinations have shown that humanity, if it wants to, can roll out changes very fast and very wide, and we are now rolling out basically unproven, somewhat-working technologies at large scale. Terrifying is a good word for that.

 

[Kouji Matsui]

Just before ChatGPT was really starting to get recognized, I was saying things like the following:

* The key to getting it to give the right answer is whether or not we can accurately define the problem. The people who are (overly) excited about this technology are under the mistaken impression that they can just talk about a problem and it will come up with an answer.
* So, what does “accurate” mean? It depends greatly on the questioner's position, situation, and environment. The answer given when "the boss" asks about a problem is probably very different from the answer given when "the wife" asks the same question. When most of the contextual information is lost, the answer is simply a combination of some information obtained from a huge amount of learning, and it is only a plausible answer.
* Even so, the answer seems correct at first glance. Well, it might actually be an accurate answer by chance. But it is more than enough for the majority of ordinary people to believe that this is the truth.
* What is going to happen from now on is that everyday life will be disrupted by insufficiently true answers. This may be that you are directly confused by the answers you get from AI systems, or it may be that such false information spreads in a way that is difficult to distinguish, becoming a large, invisible swell in everyday life that affects our lives.

Well, my opinion wasn't understood at all...

This morning I saw a news report about how the safety of the peer review system is being threatened by the spread of fake papers generated by AI systems. What's more, not only is the content garbage, but there are also reports of fake references to papers, the background to journals doing this to increase the amount of articles they publish, and methods of using AI systems to automatically detect this... I was shocked to learn that this problem has already started to occur.

 

[cjs]

This morning I saw a news report about how the safety of the peer review system is being threatened by the spread of fake papers generated by AI systems. What's more, not only is the content garbage, but there are also reports of fake references to papers, the background to journals doing this to increase the amount of articles they publish, and methods of using AI systems to automatically detect this... I was shocked to learn that this problem has already started to occur.

That news report sounds kind of off to me. Certainly LLM-generated papers are becoming yet another issue in academic publishing, but it's a minor one compared to the other issues that have been besetting it over the last couple of decades, the main one being predatory publishers.

 

[cjs]

The reason I don't want to discuss the business aspect is twofold: First, there is no meaningful any any of us can influence the outco[m]e.

Of course there is. You have a vote, and you have the ability to talk to your lawmakers. That the EU has such different laws and regulation from the U.S. (including the examples you've given yourself) demonstrates that people have a very meaningful influence over businesses. Google and Apple don't operate very differently in the EU just because they feel like it.

In theory, everything is debuggable.

No. There is a qualitative difference between code written with a purpose and a trained neural network such that's it's reasonable to say that the NN is not debuggable in the way the code is. If I ask you to create an NN that will never "decide" that a flag is preserved over a POP AF instruction, you can't do that, except to wrap it in traditional code.

And it's made all the worse by the fact that LLMs deliberately introduce randomness in their outputs. Sure, you can if you want to get rid of that, but the designers of LLM don't do that for good reasons: they won't produce anywhere near as convincing (or, for many purposes, good) results if you do that. A good part of the "creativity" that LLMs display are due to that randomness.

In practice, it's not - I've worked with a great deal of complex Android-based stuff and while a lot of understanding can be gained (sometimes at great cost), a fair share of issues are simply intractable to deal with for various reasons. Hand it to the supplier and hope that they can deal with their own technology.

That you cannot debug some compiled code given to you by someone else is not surprising, but that's not at all what I'm talking about. I am of course talking about debugging with the original source, not random compiled systems that someone has given you.

I somewhat agree. But I see many purely technical systems where even their designers look at them like mostly black boxes. (I certainly do for some of my own creations from years ago, when I was even more inexperienced than I am now.)

Again, the point is not that you choose not to go back and understand systems that you once understood before, but that the trained neural networks were never understood by their own creators in the way that someone who's written code understands it. (With the caveat that they made an effort to produce understandable code, and wanted to understand it. Certainly there's plenty of traditional code out there that people have written that they don't properly understand, but that's not because they can't, it's because they don't want to, for reasons economic or otherwise.)

The ability to reason in LLMs is currently a big research area, with some promising results (and a lot of failures, as expected of early research), so my answer to your first question is "I don't see why not".

It seems to me pretty clear why not: reason and randomness are not compatible. Much of reasoning involves designing rules to follow and then following those rules, and LLMs cannot follow rules.

The "promising results" are surface show that are clearly not heading towards reason. In the ChatGPT 3 era, LLMs could often (but not always) give the correct answer to addition problems with small numbers but not large ones. That right there says that there's something vastly different from reason going on: if you reason out the rules for addition of numbers they always work. They managed to "fix" that problem within a year or so such that it was often giving you the correct answers for much larger numbers too, but there's clearly no reasoning on the part of the LLM involved there: it still occasionally gives wrong answers for addition of smaller numbers, depending on the context in which they're presented. You can tell it a procedure for figuring out the answer, and it cannot follow that procedure. And I see nothing in the design of the system that would give it any ability to follow procedures reliably; that's simply not something that neural networks can do.

I don't even want to think about how often I repeated some tasks until I was able to do them reliably...

This is again the fallacy of asserting the consequent. You seem to keep going back to, "I've found a situation where a humans displayed the same problem as an LLM," but that doesn't change at all that there are lots of cases where humans, e.g., learn something after being trained on three or four examples and LLMs never do that.

 

[Kouji Matsui]

That news report sounds kind of off to me. Certainly LLM-generated papers are becoming yet another issue in academic publishing, but it's a minor one compared to the other issues that have been besetting it over the last couple of decades, the main one being predatory publishers.

This story is about a case of a so-called “predatory publishers", where papers are generated by AI systems and then disguised as submissions from unrelated people to fill the pages of the journal. My head hurts...

If a bogus paper is published under the name of a non-existent person, then it's easy enough to ignore that journal, but if a paper is published under the name of a real person, then it's very difficult to track down who wrote it.

https://twitter.com/x/status/1855277008194052577

 

[cjs]

This story is about a case of a so-called “predatory publishers", where papers are generated by AI systems and then disguised as submissions from unrelated people to fill the pages of the journal. My head hurts...

This is not actually predatory publishing per se; it's not harming a scholar who wants/needs a publication by attracting it to a place that doesn't give the benefits of publication in a reliable journal. That said, my guess would be that it is a predatory publication trying to make it look as if more reliable authors are publishing there. (I can't see any gain to doing this otherwise.)

If a bogus paper is published under the name of a non-existent person, then it's easy enough to ignore that journal, but if a paper is published under the name of a real person, then it's very difficult to track down who wrote it.

I think in this case it doesn't matter who actually wrote it; the harm is that it purports to be written by a scholar who didn't write it, thus affecting is reputation with anybody who discovers it. As for the journal itself, it's very easy to ignore it anyway, which is part of the problem of predatory publishing.

 

[Zare]

I'm following a musician that got copyright struck on every platform over his own music. Some Chinese AI guy and their recording company stole bits from everyone, made AI albums noone listenes at, and then registered all the "music" in those tracks.

The real author had settled that with Youtube swiftly because he's also a big Youtuber, but now with Spotify, Apple Music, and so on...it's going to last a lot.

He is a prolific musician in real life, was involved with lot of notable bands in his niche of music, etc. He's constantly recording and making stuff. Now he has to waste time catching a computer program. This is bullshit.

Things can be taken care of simply, it needs to be a lawful misdemeanor to try and patent/trademark/copyright non-human generated stuff. Period.