in fact, it may be up to chance for us (or the self-improving AI) to do so right now, because neuroscience is nowhere near that far in explaining our thinking.

It's essentially what AI are doing right now, massive amounts of trial and error to hope the AI finds some solution that makes sense and goes along with it. The AI is an "optimizer" in that it continuously tries to find a better solution rather than thinking through the problem as a person might with a big set of previous experience. We just kind of hope it stumbles on the right answer with a whole lot of repetition and a bit of smart direction-picking.

Do you think it's possible to create an AI that's whole purpose would be to crunch through data (with the reward being more data connections made)?

The problem is that two sets of data can look very different, with very different inputs, and it's almost impossible to make a single program that can deal with more than one or two "types" of data. Also we have to have a massive set of meaningful data in order to do something like this, which is not something easy to come by.

That's the problem with neural networks is that it has a set of inputs then a set of nodes that represent the connections and ideas from those inputs. Put a new set of info from a different situation in and those nodes are going to suddenly become meaningless.

I think the big thing here will have to be an AI that can manage other AI that deal with sets of data. One whose job is to create/identify information and find the best program to solve that problem.

And all this training is expensive and hard to do, as well. Neural network/machine learning rigs are set up with a whole bunch of high end graphics cards, and eat up a whole lot of processing power. The bigger the network the more exponentially costly it is to run it. You can kind of see this problem having been solved in the brain as well, where it has billions of neurons but only a few percent are used at a time for any individual input.

The brain doesn't even exist in a space where it has to have a processor go through each neuron to simulate it before it can get some sort of output.