Neural Network Architecture Construction

Introduction

This article is going to discuss neural network construction from a different perspective than is usual in conventional approaches. This approach, which will be referred to as Neural Architecture, is intended to explore construction of neural networks using neurons as explicit building blocks rather than anonymous elements trained en mass. Simple Python programs will be used to demonstrate the concept for simple boolean logic functions.

The approach of this article is intentionally named Neural Architecture because it is meant to parallel the way in which a traditional architect systematically constructs a fine building: by developing well-known patterns of construction elements, which may be re-used to create ever more sophisticated structures. The conventional approach to neural network development is to define a network as consisting of a few layers in a multilayer-perceptron type of topology with an input layer, output layer, and one or two hidden layers. Then a training algorithm such as backpropagation is applied to develop the interconnection weights. Sometimes a more sophisticated approach is taken such as using a cascade or recurrent topology but for all intents and purposes, the end result is a standard topology of a few highly-connected layers. This approach was a major breakthrough in the field because it led some people to start thinking outside the box of symbolic reasoning that dominated Artificial Intelligence at the time. It has also been successfully used in a variety of pattern recognition and control applications that are not effectively handled by other AI paradigms.

However, these applications would not generally be considered to represent higher levels of intelligence or cognitive processing. For example, suppose a neural network is developed that can successfully recognize human faces under a variety of conditions. This is a highly useful application and well within the realm of conventional neural networks. However, that is where the capability of the network leaves off–at recognizing the facial image. Aside from generalizing facial features, it can offer nothing more in terms of reasoning about those facial features. Further, it is asserted that the standard approach to neural network development is not suitable for realizing these higher levels of intelligence.

One of the fundamental problems is the limited manner in which we approach the neural architecture. To illustrate this problem, we will return to the building architecture analogy. In this way, our standard approach to neural architecture can be likened to designing a building using bricks. An architect who always thinks in terms of bricks will not likely progress beyond a certain level of sophistication, because as a component, a brick only offers one purpose: to support other bricks. Instead, an architecture progressively develops more sophisticated, proven structures based on the brick (or other primitive components) which can be re-used to develop higher-level components. A house is conceived, not in terms of bricks and wood, but rather in terms of walls, doors, and rooms. A sophisticated architect might even find these components mundane and instead think in more abstract terms of “spaces”, energy and flow of human traffic.

This is the notion of “patterns”, and in fact these (architectural) patterns were exactly the inspiration for the field of software patterns. The same thinking can be applied to neural networks: a neuron by itself only serves the function of exciting other neurons. And conventional neural net learning algorithms are geared toward categorization or other mapping operations. As a proponent of neural networks, one believes that arbitrarily complex intelligence processes can be realized, and we have the human brain as pretty convincing support of that belief. However, to continue progress in this direction, it is likely that we have to develop more sophisticated abilities as neural architects and develop useful, proven neural patterns similar to the way that building architects have done over time, and in the way software architects are now doing.

Enough philosophy– we will now be taking a fresh look at how patterns can be constructed starting with simple neural elements, and specifically we will start with boolean logic elements. Architecting using boolean logic does not immediately offer an advantage over using regular logic gates, but illustrates how crisp logic or symbolic elements can arise from fuzzy neural processing elements. It will also provide a conceptual foundation for future articles.

Neural networks

• Neural network: information processing paradigm inspired by biological nervous systems, such as our brain
• Structure: large number of highly interconnected processing elements (neurons) working together
• Like people, they learn from experience.
• Neural networks are configured for a specific application, such as pattern recognition or data classification, through a learning process
• In a biological system, learning involves adjustments to the synaptic connections between neurons.

The first step in the architecture process is to define the primitive building block, and if you haven’t fallen asleep at this point, you have no doubt figured out that this will be a neuron. The neuron model we will use is a version of the tried-and-true model used for software neural networks, also known as the perceptron. As illustrated in , the perceptron has multiple inputs and one output. The mathematical model of the perceptron is given by:

a = squash(Σ(iiwi))

where:

• is input i to the perceptron
• is the weight for input i
• is the activation (output)
• and

squash(x) = {

1 if x >threshold

0 otherwise

The nature of the perceptron has been discussed many times elsewhere, including in Matthews, so we won’t dwell on it here. But basically, the perceptron calculates a weighted sum of the inputs and then subjects it to a nonlinear “squashing” function–in our case, this is a simple threshold operation. The nonlinear threshold operation is part of what makes a neural net exhibit interesting behavior. Otherwise it would amount to matrix operations.

Nature of the game

Now that we have the model for a basic neuron defined, we can now proceed to define basic logic gates by simply working out two things:

• the weight values
• the threshold

For our discussion we will assume that weights can be positive (excitory) or negative (inhibitory) and be in the range between -1 and 1. The threshold will also be assumed to be in the range -1 to 1.

If we cast this in terms of signals, then it equates to the requirement that both inputs have to be sufficiently high to produce an output. So, we will set our threshold to a high value of 0.8. Next we will set the weights for the two inputs at 0.5 each. If one input is one (=1) then the neuron activation (output) will be given by

a

= squash( 1 * 0.5 + 0 * 0.5)

= squash( 0.5 )

= 0 since 0.5 <0.8 {the threshold}

Therefore if either or both inputs is 0, the output is zero. If both are 1, then the weighted sum will be greater than the threshold. Listing 1 show the Python program that demonstrates the neural AND gate, and will be used as a template for other logic functions. The author prefers Python for prototyping, because it results in compact code that looks very close to pseudocode.

Program to test Inverter

We have shown how a simple set of logic gates may be implemented using neurons. In theory, this is all we would need to implement a universal digital computer of arbitrary complexity, because that is all a microprocessor does. In fact, it can be shown that all the logic functions can be implemented with one type of gate: either all NAND gates or NOR gates. However, if that was our goal, there would be no advantage to this approach over using standard logic gates. The point is that a neural approach can be used for designing intelligent structures–that is explicitly architecting (there’s that word again) structures–as well as evolving them using conventional neural network training paradigms. With neural architecture, we can start with simple logic operations and build them up into more interesting logical structures. As an example, consider our first example, the simple AND gate. This could be used to digitally perform an operation on bits, but it can also be used to perform a logical AND operation on two rules, for example: IF rule X AND rule Y THEN fire (output is active). Furthermore, by using neurons, the inputs are not at all limited to binary signals, they could be analog signals (rule X is sort of true) or even the output of an entire neural network.

Conclusion

The neural architecture approach can be seen to have the following key advantages or possibilities:

• Can develop structures built on a universal neuron building block.
• The same basic building block can be considered as a neuron (standard perceptron) or a logic gate.
• The approach allows crisp structures to interface with fuzzy ones, combining the nondeterministic and evolutionary properties of neural networks along with the advantages of explicit, structured design.
• Can develop recognizable higher level building blocks, a important requirement for realizing continuously more complex architectures.