Maybe it's just because both our industry as well as the internet always going to the extrema, but I'd love to see a nuanced take on the subject at some point.
My experience so far is middling; I found that it can be extremely helpful when working on novel things; greenfield projects it can lay a solid foundation for. If I'm learning some new framework it can shit out a first version that helps me understand how the thing works.
However, as soon as things get a bit more complex, they immediately start shitting the bed. Working in existing codebases of more than like 25 files and they become useless. Hell, even asking some models to document a 150 line batch install file made it hallucinate all kinds of stuff that didn't exist.
Furthermore, you really do have to talk to them like they're autistic idiot-savants. Precise steps, boundaries, everything. But even then they often just go off into the deep end. Even when I explicitly tell it to do one thing, one thing only and not change anything else, over 50% of the time it starts changing everything.
It's another tool, and like with all tools you need to know how and when to use it, I guess. As an industry we're not going to finger out the right ways if half the world refuses to use a powerdrill because well gosh darn it, hammers have always been good enough, and the other half swears up and down you don't need cement anymore because you could just as easily screw bricks together.
I had a fun day with replit yesterday. I signed up to try it out after someone at work mentioned it.
I gave it a really simple challenge: a note taking app. Allow the user to write a sentence of plain text, and add it to an array with a time stamp. Show the text on screen. Save it in local storage.
I could hand code this in about ten minutes.
Replit went off and built a truly impressive UI that looked like something that would take me a couple of evenings to pull together. It was professional and polished, but it didn't work. Hitting enter to add the item did nothing.
I chatted to it and explained. It confidently did a few revisions but didn't fix it.
It started adding logging and asked me to paste in the logs. After each subsequent paste it would say some variation of "ah I see the problem" and revise again.
After the third time, I didn't reply with the console logs and instead suggested we change tack. Replit completely ignored my input and carried on talking as if I was following instructions "yes I see the issue now, ok I've fixed it"
It was just a string of hallucinations, pretending to take user input but it fact not doing so. In the end I started asking it completely off topic questions but it had an imaginary convo with me instead. I gave up.
I've had much more success with non agentic LLMs like Gemini or Claude but they all are great at the initial response and slowly degrade add the conversation continues
>they all are great at the initial response and slowly degrade add the conversation continues
I've noticed this too. I give the llm 3 or 4 prompts before a new session is opened and code is repasted. My work flow is a lot like
1. Here's a schema, what would you change? I assume nothing, because my schemas are perfect.
2. Given this schema, make me some structs
3. Given these structs, make me some endpoints
1.(New session) Given these endpoints that return data like this, make me a ui
1. while true: (New session), An llm wrote me this ui, what would you change?
If I try to steamroll ahead from schema to ui, I find it starts having trouble with context, like hallucinating functions it meant to write but didn't, or just straight up dropping functions it wrote before.
It makes me think back to "attention is all you need". Most of the LLMs seem to have quite limited attention, at least while coding. I suspect replit probably had similar issues while talking to itself, before it even gave me the first response.
Imo there is nothing wrong with generating code with AI, it's the effort spent on supervising the quality of product that matters.
But that requires you to have certain levels of knowledge on that domain to begin with, which is not something you can just "vibe" your way out, at least for now.
I've been a software engineer for over 30 years and I have to say using AI and vibe coding has made me a 10x programmer. It's all about prompting and debugging, knowing how to fix issues is still a necessary skill and how to build/architect things from a prompt perspective. I've released 8 npm modules in the past 2 weeks alone just for some silly stuff I was using.
They said 'maybe'. I'm gonna say they are right, because there is no need for 99%+ of npm modules (and indeed, by far most are not used by anyone except the org author). Especially with AI, most npms are trivially generated faster than you can find and install them and usually less influenced/opinionated by the usually terrible star vying programmer that uploaded it in the first place. If AI trivially generates your npm (so you can make 4 per week next to whatever you were doing), why would I need it instead of having AI trivially generating a specific (without the features I don't need) version for what I am doing at the moment?
Coding since 1986, and what I see is the dream of CASE tooling finally coming true, eventually as AI matures, assuming we aren't getting through yet another AI cycle that will stabilize until the next wave, I see ourselves writing requirements documents in a UML kind of way (in spirit, doesn't need to be boxes and arrows) for AI minions to actually do the work with one third of current team sizes.
The other two thirds can search for something else, just like the factory workers when robots took over most factory roles.
The remaining one third, are the folks doing the jobs robots can't yet do, or doing their maintenance.
Software factories are no different from physical ones, in this regard, a few iluminaries will be able to profit from AI, everyone else needs to re-train themselves.
> I see ourselves writing requirements documents in a UML kind of way
That's what I've had most success with. Chat to create a requirements/spec document. Then convert it into a work plan. Then feed the stages into an agent for implementation. Works great and the extra "let me know which areas should be more refined" does wonders.
This was the dream. How it was supposed to be. Speak and it becomes. But developers found a way to gatekeep the whole process by inventing thousands of different languages/wheels over time.
And they held the keys. Now that they are steadily working most of themselves out of a job, they are still stuck with their hubris-tainted glasses, unable to move on.
6 months ago I had no idea how Github works. Now I have thousands of commits and I have learnt to git reset --hard as second nature now :-) Hell, yesterday I figured out by chance, how to only revert a particular file. So yeah, baby steps.
I am "building" every day, talking to my AI Coder friend. We are having a lot of fun. I don't go full Agent Mode. I am always on Ask Mode. (Because - hard lessons - you know)
I am learning about Lighthouse (97% aggregate for my Next.js app)
Yesterday I made my site an "App", because someone asked if "there's an app for that"
Now I know about PWA and all it's requirements. Cool.
The point is. I am having fun building stuff. And that would not have been possible for me a year ago.
Ask yourself. If you as a coding genius had to develop a program without any internet access, how far would you get?
So stop crying that the bar has been lowered. Raise the bar. Be the guy that can outsmart the AI when it tries to take over the world. There will be few of you needed, but the need will be critical. We will always need a few Myron Aubs. Just not millions of them.
This is the same conceit for rapid application development and just do the prototype in production.
It just has a lot more legs to it with llms.
I'm glad you're having fun.
But the world doesn't run on prototype code. Maybe llms will help being the quality of code in the last 20 years collectively up a notch.
In other words, it would be great if llms took all that "just good enough code" we wrote in the last 2 to 3 decades and made it actually good code.
I strongly suspect this is not going to be the case. Instead, the world will be absolutely tsunamied by an avalanche of not even quite good enough code
> If you as a coding genius had to develop a program without any internet access, how far would you get?
As an engineering student transferred to applied math in the early 1980s I was able (with no real substantial internet access, literally getting floppy discs by mail, etc) to write an OS, an original commercial event tracking system for fleet management of mining vehicles, a real time marine seismic capture, storage, view and analysis system (basic PC, real time OS (that a friend wrote), Window Manager, dedicated hardware, lots of assembler, signal processing, etc).
And a whole lot more. (Bit of work on a sheep shearing robot, lot of early GIS mapping routines)
(Oh, some interesting stuff for CAYLEY/MAGMA that has recently been used to break quantuum encryption candidates .. odd just how long some fundementals last)
It seemed normal at the time to be able to do this, both a circle of friends and myself had a ball building stuff - much of it based on books, talking in person to others, and just diving in and playing about.
It's perhaps easier to get by without AI and real time always connected internet than you seem to think.
Just someone in the void between Jerome Gauntlett, Charles Leedham-Green, Len Beadell, George Havas, Graham Priest, William Tutte, Jack Wong Sue, and a pantheon of others ...
We can agree I think, that there are languages that are replicating almost exactly what others have done before, coming off as more of a vanity project, than anything else, while some are truly revolutionary/evolutionary?
Maybe it's just because both our industry as well as the internet always going to the extrema, but I'd love to see a nuanced take on the subject at some point.
My experience so far is middling; I found that it can be extremely helpful when working on novel things; greenfield projects it can lay a solid foundation for. If I'm learning some new framework it can shit out a first version that helps me understand how the thing works.
However, as soon as things get a bit more complex, they immediately start shitting the bed. Working in existing codebases of more than like 25 files and they become useless. Hell, even asking some models to document a 150 line batch install file made it hallucinate all kinds of stuff that didn't exist.
Furthermore, you really do have to talk to them like they're autistic idiot-savants. Precise steps, boundaries, everything. But even then they often just go off into the deep end. Even when I explicitly tell it to do one thing, one thing only and not change anything else, over 50% of the time it starts changing everything.
It's another tool, and like with all tools you need to know how and when to use it, I guess. As an industry we're not going to finger out the right ways if half the world refuses to use a powerdrill because well gosh darn it, hammers have always been good enough, and the other half swears up and down you don't need cement anymore because you could just as easily screw bricks together.
Perhaps this one from James Bach. "Is There Something in Your “I”?" https://www.satisfice.com/blog/archives/487881
I had a fun day with replit yesterday. I signed up to try it out after someone at work mentioned it.
I gave it a really simple challenge: a note taking app. Allow the user to write a sentence of plain text, and add it to an array with a time stamp. Show the text on screen. Save it in local storage.
I could hand code this in about ten minutes.
Replit went off and built a truly impressive UI that looked like something that would take me a couple of evenings to pull together. It was professional and polished, but it didn't work. Hitting enter to add the item did nothing.
I chatted to it and explained. It confidently did a few revisions but didn't fix it.
It started adding logging and asked me to paste in the logs. After each subsequent paste it would say some variation of "ah I see the problem" and revise again.
After the third time, I didn't reply with the console logs and instead suggested we change tack. Replit completely ignored my input and carried on talking as if I was following instructions "yes I see the issue now, ok I've fixed it"
It was just a string of hallucinations, pretending to take user input but it fact not doing so. In the end I started asking it completely off topic questions but it had an imaginary convo with me instead. I gave up.
I've had much more success with non agentic LLMs like Gemini or Claude but they all are great at the initial response and slowly degrade add the conversation continues
>they all are great at the initial response and slowly degrade add the conversation continues
I've noticed this too. I give the llm 3 or 4 prompts before a new session is opened and code is repasted. My work flow is a lot like
1. Here's a schema, what would you change? I assume nothing, because my schemas are perfect.
2. Given this schema, make me some structs
3. Given these structs, make me some endpoints
1.(New session) Given these endpoints that return data like this, make me a ui
1. while true: (New session), An llm wrote me this ui, what would you change?
If I try to steamroll ahead from schema to ui, I find it starts having trouble with context, like hallucinating functions it meant to write but didn't, or just straight up dropping functions it wrote before.
It makes me think back to "attention is all you need". Most of the LLMs seem to have quite limited attention, at least while coding. I suspect replit probably had similar issues while talking to itself, before it even gave me the first response.
Imo there is nothing wrong with generating code with AI, it's the effort spent on supervising the quality of product that matters.
But that requires you to have certain levels of knowledge on that domain to begin with, which is not something you can just "vibe" your way out, at least for now.
I've been a software engineer for over 30 years and I have to say using AI and vibe coding has made me a 10x programmer. It's all about prompting and debugging, knowing how to fix issues is still a necessary skill and how to build/architect things from a prompt perspective. I've released 8 npm modules in the past 2 weeks alone just for some silly stuff I was using.
Maybe there was no need for 8 more npm modules ... is_odd ...
Finally the treadmill of endlessly reinventing JavaScript frameworks and libraries is available for us Joe Schmoes too.
How do you know? I'd have to see those modules first to have an opinion.
They said 'maybe'. I'm gonna say they are right, because there is no need for 99%+ of npm modules (and indeed, by far most are not used by anyone except the org author). Especially with AI, most npms are trivially generated faster than you can find and install them and usually less influenced/opinionated by the usually terrible star vying programmer that uploaded it in the first place. If AI trivially generates your npm (so you can make 4 per week next to whatever you were doing), why would I need it instead of having AI trivially generating a specific (without the features I don't need) version for what I am doing at the moment?
https://www.npmjs.com/org/profullstack
Counter-point: there's plenty of need for modern-es6/typescript-first/promise-aware/tree-shakable npm libs.
Coding since 1986, and what I see is the dream of CASE tooling finally coming true, eventually as AI matures, assuming we aren't getting through yet another AI cycle that will stabilize until the next wave, I see ourselves writing requirements documents in a UML kind of way (in spirit, doesn't need to be boxes and arrows) for AI minions to actually do the work with one third of current team sizes.
The other two thirds can search for something else, just like the factory workers when robots took over most factory roles.
The remaining one third, are the folks doing the jobs robots can't yet do, or doing their maintenance.
Software factories are no different from physical ones, in this regard, a few iluminaries will be able to profit from AI, everyone else needs to re-train themselves.
> I see ourselves writing requirements documents in a UML kind of way
That's what I've had most success with. Chat to create a requirements/spec document. Then convert it into a work plan. Then feed the stages into an agent for implementation. Works great and the extra "let me know which areas should be more refined" does wonders.
[flagged]
I bet it can code an entire app faster than you can. If you already know what you want, AI is a huge productivity boost.
Writing code takes me little to no time
Thinking about what I want is what takes me most of my time
I am an engineer and, as such, are paid to think, not "just" use my hands
This was the dream. How it was supposed to be. Speak and it becomes. But developers found a way to gatekeep the whole process by inventing thousands of different languages/wheels over time.
And they held the keys. Now that they are steadily working most of themselves out of a job, they are still stuck with their hubris-tainted glasses, unable to move on.
6 months ago I had no idea how Github works. Now I have thousands of commits and I have learnt to git reset --hard as second nature now :-) Hell, yesterday I figured out by chance, how to only revert a particular file. So yeah, baby steps.
I am "building" every day, talking to my AI Coder friend. We are having a lot of fun. I don't go full Agent Mode. I am always on Ask Mode. (Because - hard lessons - you know)
I am learning about Lighthouse (97% aggregate for my Next.js app)
Yesterday I made my site an "App", because someone asked if "there's an app for that"
Now I know about PWA and all it's requirements. Cool.
The point is. I am having fun building stuff. And that would not have been possible for me a year ago.
Ask yourself. If you as a coding genius had to develop a program without any internet access, how far would you get?
So stop crying that the bar has been lowered. Raise the bar. Be the guy that can outsmart the AI when it tries to take over the world. There will be few of you needed, but the need will be critical. We will always need a few Myron Aubs. Just not millions of them.
Happy building (Even if you are not "coding")
This is the same conceit for rapid application development and just do the prototype in production.
It just has a lot more legs to it with llms.
I'm glad you're having fun.
But the world doesn't run on prototype code. Maybe llms will help being the quality of code in the last 20 years collectively up a notch.
In other words, it would be great if llms took all that "just good enough code" we wrote in the last 2 to 3 decades and made it actually good code.
I strongly suspect this is not going to be the case. Instead, the world will be absolutely tsunamied by an avalanche of not even quite good enough code
A fair and measured response. Thank you.
> If you as a coding genius had to develop a program without any internet access, how far would you get?
As an engineering student transferred to applied math in the early 1980s I was able (with no real substantial internet access, literally getting floppy discs by mail, etc) to write an OS, an original commercial event tracking system for fleet management of mining vehicles, a real time marine seismic capture, storage, view and analysis system (basic PC, real time OS (that a friend wrote), Window Manager, dedicated hardware, lots of assembler, signal processing, etc).
And a whole lot more. (Bit of work on a sheep shearing robot, lot of early GIS mapping routines)
(Oh, some interesting stuff for CAYLEY/MAGMA that has recently been used to break quantuum encryption candidates .. odd just how long some fundementals last)
It seemed normal at the time to be able to do this, both a circle of friends and myself had a ball building stuff - much of it based on books, talking in person to others, and just diving in and playing about.
It's perhaps easier to get by without AI and real time always connected internet than you seem to think.
Is that you, Myron?
Just someone in the void between Jerome Gauntlett, Charles Leedham-Green, Len Beadell, George Havas, Graham Priest, William Tutte, Jack Wong Sue, and a pantheon of others ...
> But developers found a way to gatekeep the whole process by inventing thousands of different languages/wheels over time.
Yep, definitely would have been better if developers hadn't invented higher level languages and LLMs would be forced to output assembly.
We can agree I think, that there are languages that are replicating almost exactly what others have done before, coming off as more of a vanity project, than anything else, while some are truly revolutionary/evolutionary?
No, I do not agree. Which languages are vanity projects in your eyes?
I'm sure you could have learned how to use git and github before AI existed.
AI Coding motivated me. Maybe that's the point.
[flagged]