I truly feel that flagship model LLMs’ comprehension and general everyday competencies have become less capable. Ethan Mollick1 and others show many wonderful things that are fascinatingly capable, even as one-shot prompts. Yet I find most of the important work I'm doing and advising others to do is not of the one-shot nature; it is slow, iterative, and requires much exploration. But, this even holds to some incredibly basic auto-suggested changes and corrections.

I find myself2 continuously being in dismay when corrections that were previously streamlined are now clunky and require more revising. I understand several performance benchmarks show substantial improvement, but I don't have much to say in favor of the general use for most LLMs as becoming more cognitively fluent or more adapted to the user (or to 'me', in particular.)

When I structure it to do what it should do at a high level, it works well; if you provide the structure or give it a blank canvas with defined constraints; or, if the standard is merely that something interesting comes out and the specifics are negotiable, these outputs can be genuinely striking.

What I haven't seen improve is the underlying robustness; nothing I've experienced is particularly improved in what I'd say is actual cognitive functioning or becoming less-brittle in its contextual awareness. It is still, as it was listed some time ago, "an intern that needs much guidance on how to do things" - that intern is now a fantastic coder, and can pull sophisticated-sounding information and citations; there is some element of brilliance there.

Yet for the kind of work that I find most critical and most important and requires the most actual consideration, or deep work, I am left feeling its almost taken a step back in the last 12 months, rather than improved.3

I know this is probably a controversial hot take-like post, or something you'd hear from a card-carrying member of the AI Hater Club; that's for you to determine.

What I'd add is that I would prefer it weren't true.

Hand-holding a model through a critical conceptualization is time I would rather spend elsewhere, and I keep being reminded that the limits of the intellectual partnership are what they are.

Easy and Hard problems for in AI for Science

My reference of the day: This piece by Ruairidh Battleday and Sam Gershman gets at some of it - the difference between the "hard problem" (not consciousness!) and "easy problem."

In other words, if you're research or business aims are "easy problem" related - this is fantastic times for many AI related implementations, for sure.

But, indeed, "Solving the hard problem is beyond the capacities of current algorithms for scientific discovery because it requires continual conceptual revision based on poorly defined constraints"; even though this paper was made perhaps over a year ago and published earlier this year proper, I would say this still holds.

PS: We had the opportunity to cover this in the Cognition Futures (JOPRO + Orthogonal Research and Education Laboratory) Reading Group recently, which will be posted on our YouTube soon.

Subscribe now

Note: This post is adapted from my original LinkedIn post

One of the most important things to consider, when cold emailing or replying to emails, and in general:

How much rapport and grace do you have with the person?

If you are already solidly “in” with them, and they have a sense of respect or investment in you -- it’s often better to just get the message to them, especially if they are in a space or carry themselves in a way where they are familiar to and open with the development of others.

Exceptions: difficult personalities, hierarchies

The obvious counters to this are: the personalities who need something very precise and in a particular format to reply to; they should be handled with care, and picking your battles or your asks is a different ballgame.

Or, if someone is vested (by choice or force) in particular hierarchical communication structures, being savvy of what those limitations are will help you -- even if it’s a matter knowing the rules so they can be broken given the appropriate time or method.

The Key: make a clear and concise request

But I see a lot of early career folks wondering if/when/how to send emails, and most of the time it’s better to get the conversation started than hem-and-haw or fuss about perfect language; if someone wants to have the conversation with you or give you the time, they will.

So a simple, concise request to receive that conversation / attention / response from them is sufficient.

Expanded context to consider

There is a particular flow and inertia to all of this, and there is a divide between people who have had to manage projects , plan research, or do programming -- that those who do not are often unfamiliar with.

Many things come down to figuring out if (as an applicant or joiner-of-projects) you can find a way to get in that flow and part of the caravan of activities that the host (org, or person directly) is managing; and on the other side, the manager of the activities needs to assess whether or not you can fit in, keep up, and are worth the investment.

Understanding where you are in all of this is another key to understanding communications, requests, timing, and potentials for involvement.

Subscribe now