Computational Mechanisms of Pulse-coupled Neural Networks a Comprehensive Review
Sensors (Basel). 2020 May; xx(10): 2764.
An Improved Pulse-Coupled Neural Network Model for Pansharpening
Xiaojun Li
oneKinesthesia of Geomatics, Lanzhou Jiaotong Academy, Lanzhou 730070, China; nc.utjzl.liam@iljx
2National-Local Articulation Engineering science Enquiry Heart of Technologies and Applications for National Geographic State Monitoring, Lanzhou 730070, China
Haowen Yan
iFaculty of Geomatics, Lanzhou Jiaotong University, Lanzhou 730070, Mainland china; nc.utjzl.liam@iljx
twoNational-Local Articulation Engineering Research Eye of Technologies and Applications for National Geographic State Monitoring, Lanzhou 730070, Red china
Weiying Xie
3Country Key Laboratory of Integrated Service Network, Xidian University, Eleven'an 710071, Cathay; nc.ude.naidix@eixyw
Lu Kang
4State Key Laboratory of Resources and Environmental Information System, Plant of Geographical Sciences and Natural Resources Research, CAS, Beijing 100101, Prc; nc.ca.sierl@lgnak
Yi Tian
vGuangdong Provincial Cardinal Laboratory of Robotics and Intelligent System, Shenzhen Institutes of Advanced Technology, CAS, Shenzhen 518172, Prc; nc.ca.tais@1nait.iy
Received 2020 Apr 8; Accepted 2020 May 11.
Abstract
Pulse-coupled neural network (PCNN) and its modified models are suitable for dealing with multi-focus and medical image fusion tasks. Unfortunately, PCNNs are difficult to directly apply to multispectral image fusion, specially when the spectral fidelity is considered. A key problem is that well-nigh fusion methods using PCNNs usually focus on the option mechanism either in the infinite domain or in the transform domain, rather than a details injection mechanism, which is of utmost importance in multispectral image fusion. Thus, a novel pansharpening PCNN model for multispectral epitome fusion is proposed. The new model is designed to acquire the spectral fidelity in terms of human visual perception for the fusion tasks. The experimental results, examined by dissimilar kinds of datasets, show the suitability of the proposed model for pansharpening.
Keywords: multispectral image, pansharpening, pulse-coupled neural network, high-resolution image, epitome fusion
1. Introduction
Multispectral (MS) prototype is of great importance in typical remote sensing applications, such every bit land utilization [one], urban interpretation [2], urban monitoring [three], and scene understanding [4] tasks. All the same, owing to the physical constraints of the optical satellite sensor and its advice bandwidth, the spectral diversity and the high spatial resolution cannot exist obtained at the same fourth dimension. In other words, the finer spectral resolution is obtained at the price of the coarser spatial resolution. Thus, pansharpening—which combines the narrow-band multispectral epitome with a broadband loftier spatial resolution panchromatic (PAN) image to acquire a loftier-resolution multispectral prototype—is desirable. The contributions of the pansharpening technology for remote sensing tasks mainly include change detection [5], feature extraction [half dozen], land classification [7] and scene interpretation [viii]. Detailed reviews of the pansharpening methods can be plant in [nine,x,xi,12].
Virtually popular methods for pansharpening fall into two master groups, which are the component substitution (CS) methods and the multiresolution analysis (MRA) methods [9]. The CS methods mean that one of a component extracted from the multispectral image is substituted by using the PAN image. It consists of the methods, such as the intensity-hue-saturation method (IHS) [13,14], the primary component analysis method (PCA) [15,16], and the Gram–Schmidt (GS) method [17]. The MRA methods provide effective multiresolution decomposition tools to obtain spatial details from the high-resolution epitome, which could be injected into the resampled multispectral images. These algorithms include the decimated wavelet transform (DWT) [xviii], the undecimated wavelet transform (UDWT) [nineteen], the "à trous" wavelet transform (ATWT) [20], the Laplacian pyramid (LP) [21], and the morphological pyramid [22]. More recently, several deep learning approaches have been proposed to address the fusion trouble, which achieves the better fusion results through studying the grooming dataset. It consists of methods such as the convolutional neural networks method [23] and the pansharpening convolutional neural networks method [24]. Other pansharpening methods accept been too proposed for the preprocessed steps or the misaligned information, such as the enhanced back-projection model [25] and the improved fusion model [26].
Equally a basic remote sensing problem, pansharpening is getting more attending forth with the rapid development of the high-resolution satellite paradigm. The platforms of Google Earth and Microsoft Bing are the virtually representative applications of pansharpening [27]. The chief purpose of pansharpening is to inject the details from the higher resolution image into the lower ones and then as to obtain the high-resolution multispectral paradigm. Classical methods produce the injection coefficients either through global estimation or locally used rectangular windows. Apparently, global injection algorithms will inevitably upshot in more spectral distortion, considering all the extracted details are treated without distinction. Local estimation of the injection coefficients performs improve than global ones by considering the correlations of the neighborhood pixels. However, the local estimation areas for the injection process are e'er restricted in a stock-still square window [28], which will provide frustrating results equally the foreground and the background pixels announced in the same square. The mixed state of affairs in a given square window is specially common in high-resolution remote sensing images, since the tiny objects are more likely available for the loftier-resolution images. Thus, the injection coefficients should be estimated in an irregular region rather than a square one. The pulse-coupled neural network (PCNN), which derives from the synchronization oscillation miracle of visual cortex neuron, has the ability to generate the irregular clustering regions. The synchronization oscillation phenomenon means that the neurons (respective to the pixels in an paradigm) of a similar status will release the pulses at the same fourth dimension.
The PCNN has shown great ability in image processing and design recognition applications [29]. More than specifically, one of the most diffused uses of PCNN and its improved versions is in dealing with the image fusion problem, which includes m-PCNN [30], memristive PCNN [31], memristor-based PCNN [32], and shuffled frog leaping PCNN [33]. Though these models have laudable ability in the multi-focus image fusion and medical image fusion, they are difficult to apply directly in pansharpening. All these PCNN fusion models suggest that the input channels are similar and parallel, which lacks the considerations of details injection and spectral preservation. Obviously, the degree of spectral distortion is of the utmost importance in the pansharpening cess of the remote sensing images. Therefore, the issue is unable to be direct addressed by traditional PCNNs. Cheng proposed a new adaptive dual-aqueduct PCNN for fusing the infrared and visual images [34], which exhibits proficient fusion performance. Ma et al. also tried to address the fusion bug for infrared and visible images by using neural networks, such as the DDcGAN method [35], the FusionGAN method [36], and the particular-preserving adversarial learning method [37]. These models have attained very satisfactory results. Nonetheless, the application does not take the spectral channels into account. Shi et al. presented a novel PCNN based algorithm for remote sensing image fusion [38], which tin achieve practiced fusion results for the multi-source remote sensing images. Nevertheless, the fusion algorithm mainly aimed for epitome enhancement, and spectral fidelity is not under consideration.
To settle the effect, a pansharpening PCNN (PPCNN) model for multispectral prototype fusion is proposed in the paper. Information technology can adaptively inject spatial details to the multispectral images in each iteration. Benefiting from the characteristic of the synchronous pulse emitting phenomenon of PPCNN, the particular injection interpretation tin can exist manipulated among the pixels which not only have similar values, but also similar neighborhoods. In addition, the proposed model is proved to coincide with the human visual perception, which results in the ameliorate visual inspection of fusion results. In the newspaper, the performance of the PPCNN has been well tested with five different high-resolution datasets. The results have been well assessed by both the spectral and spatial quality evaluation benchmark. The results of the experiments indicate that the proposed multispectral fusion method is effective for pansharpening.
The residual of the commodity is organized as follows: Section ii reviews the standard PCNN model in brief, and then proposes a novel PPCNN model. Section 3 is devoted to presenting a pansharpening method based on the proposed PPCNN model. Section iv gives the experimental method. Experimental results and discussions are exhibited in Department 5, and Section vi summarizes the work.
2. PCNN and PPCNN Models
To suggest the new PPCNN model, the standard PCNN model is briefly introduced. Adjacent, the proposed PPCNN is introduced. The improvements on the standard PCNN model are based on the applied demands of the pansharpening applications. The analysis and the implementation of the new model are likewise given in the section.
two.one. Standard PCNN Model
The encephalon, as is well-known, perceives the outside world through a complicated neural network of visual cortex neurons. The information of the real world is transmitted by thousands of neurons in the network. Each neuron consists of the jail cell body, the synapse, the axon and the dendrites. As shown in Figure 1, the cell torso receives the electrical impulses from the synapses of other neurons via its dendrites. The membrane potential of the cell body rises as the neuron continues to be stimulated by other neurons. The electrical impulse is generated when the membrane potential is larger than its threshold. The threshold of current neuron changes in a nonlinear way later stimulation. On the other hand, the neuron also transmits electrical impulses to other neurons via the axon part.
As inspired past the structure of visual cortex neuron, Johnson et al. proposed the PCNN model [39]. PCNN, which imitates the mechanism of mammalian visual cortex, is a laterally continued neural network of 2-dimensional neurons with no training. As shown in Figure two, each neuron of the standard PCNN model is divided into iii sequential parts: the receptive field, the modulation field, and the pulse generator. It can be mathematically described as follows [39,40]:
(1)
(2)
(3)
(4)
(5)
In the PCNN model, the position of the neuron is denoted as a two-dimension symbol ij (ij means that the position of the neuron is at row i, cavalcade j), so as to be conveniently applied for an image. The neuron kl is defined as the neighboring neuron of the current neuron ij. The neuron ij receives the electric impulses from the synapses of neighboring neurons kl, which simulates the functions of the dendrite part to receive the local stimulus via linking synapse G (or W). Later on the electrical impulses are gathered past the prison cell, the inputs are distinguished into two channels. One channel is the feeding input F and the other is the linking input L. The difference between them is that the feeding input F is influenced by the external stimulus I. In the modulation part, the internal activity U is designed to imitate the membrane potential of the cell body. The internal activity U is obtained by the coupling of F and weighted 50, where the weighting gene is typically denoted as β. The pulse generator will produce the output pulse Y, if the dynamic threshold E is less than the internal activity U. Furthermore, VF , VL and VE indicate normalizing parameters. The parameters αF , αL and αEast denote the exponential decay coefficients, which imitate the exponential attenuation characteristics over time.
2.ii. Proposed PPCNN Model
In order to make the PCNN model advisable for pansharpening, an improved PCNN model is proposed, which is called PPCNN hereafter. For the convenience of the fusion task, the new model should be designed to have ii external inputs, and its characteristic of synchronous pulse emitting should remain unchanged. The neuron structure of the improved model is shown in Figure iii. Compared with the standard PCNN, the PPCNN model has at least four advantages: (1) ii external stimuli I and P are considered simultaneously for the convenience of fusion operations, (2) each neuron ij of PPCNN has its own weighting factor βij rather than a uniform 1 in the standard PCNN model, (3) the internal activity U, which is composed of the sum of the feeding part F and the weighted linking part Fifty, is considered as the terminal fusion result, and (4) fewer parameters in the new model. The PPCNN model can be mathematically described using the following expressions:
(6)
(seven)
(8)
(9)
(10)
2.three. Implementation of PPCNN Model
Since the PPCNN model is mainly for the fusion tasks, the output part Y in the standard PCNN turns into the intermediate variable in the PPCNN model. Appropriately, the internal activeness U represents the fusion effect. Before the implementation of PPCNN is described, a couple of necessary symbols used should be divers. Symbol × indicates the multiplication betwixt a constant and the matrix. Symbol • multiplies each element of one matrix with the respective one in the other matrix. Symbol ⊗ denotes the convolution between 2 matrices. Specifically, each neuron has its own βij in the PPCNN model, designed as follows:
(11)
(12)
where the symbol Cov(X, Y) stands for the covariance functioning between X and Y. Std(X) indicates the standard deviation of X. I refers to the input multispectral image. Pan denotes the panchromatic prototype. An instance of β image is shown in Figure iv.
The implementation procedure of PPCNN model includes the following steps:
-
Initializing matrices and parameters. Y[0] = F[0] = Fifty[0] = β[0] = U[0] = 0, and E[0] = FiveE . The initial value of the iteration number n is ane. P is denoted as the spatial detail epitome. I and P are all normalized between 0 and one. Other parameters (VF , VL , 5East One thousand, Due west, and αE ) are determined by experience based on different applications.
-
Compute F[n] = VF × (Y[n − 1] ⊗ M) + I, L[north] = 5L × (Y[n − ane] ⊗ W)+P, and U[n] = F[northward − ane] + β[n − 1]•L[n − 1].
-
If Uij [n] > Eastij [n], then Yij [n] = ane, else Yij [n] = 0.
-
Update E[n + i] = e- αE × E[n] + VDue east × Y[n].
-
Stop iteration until all neurons have been stimulated, and the internal action U is the final fusion result, else let n = n + one and render to the Step (2).
two.4. Analysis of PPCNN Model
For the PPCNN model, Equation (8) can be rewritten beneath:
(thirteen)
The part Iij + βij •Pij of the Equation (13) is similar to the traditional pansharpening method, except that βij changes in each iteration. It means that the proposed model has the injection machinery which is beneficial for pansharpening awarding. In improver, some skilful characteristics have been inherited from the standard PCNN in the new model, such as the phenomenon of the synchronous pulse emitting and the exponential attenuation of the threshold. As exhibited in Equation (13), the influences to the electric current neuron are not only from the external stimulus (I and P), but too from the neighborhoods, which guarantee the synchronous pulse emitting machinery. In other words, the neurons with similar neighborhoods and stimuli will emit the pulses at the aforementioned fourth dimension. Once the stimulus is a high-resolution epitome, the synchronous pulse emitting mechanism volition make the interpretation of boundaries and textures of the object more than accurate.
Another observation is that the threshold of the neuron decays exponentially according to Equation (x). Since the initial value Y[0] = 0 and E[0] = VEastward , Eij at the kickoff iteration is
Suppose that the neuron ij fires at the n 0th iteration first, and fires over again at the nl thursday iteration, where l is the iteration number of the firing consequence. When the iteration number n is less than north 0, we have
Otherwise, while north is greater than n 0, we get
(xvi)
where L is the full firing times of the neuron ij.
According to Equation (ix), the output pulse of the current neuron ij is generated when the threshold Due eastij is merely less than the internal activeness Uij . Thus, we have the firing time as follows:
(17)
Since Uij is the accumulation of the external stimuli (multispectral and panchromatic images), Equation (17) obeys the Weber–Fechner law, i.e., at that place exists a logarithmic relationship between the firing time and the external stimuli. Therefore, the proposed model is coincide with the man visual perception. The exponential attenuation curve of the proposed model is shown in Effigy 5. Information technology is found that PPCNN processes the lower stimulus more precisely and the higher ones more coarsely. The characteristic is in accordance with the human visual system that people are very sensitive to contrast variation in the night areas.
Compared with the detail injection mechanism of the most pansharpening methods and the pulse-emitting machinery of standard PCNN model, the PPCNN model has the advantages of both by introducing the details injection mechanism into the standard PCNN model. In improver, as indicated in Equation (12), βij [n] automatically changes based on the current firing environs in the iteration, which guarantees that the statistical calculations are computed among the neurons with the like state. Furthermore, PPCNN is proved to coincide with the human visual arrangement. These improvements are believed to be propitious for multispectral remote sensing epitome fusion.
three. PPCNN Based Pansharpening Approach
For pansharpening applications, at that place is a 1-to-i corresponding relationship betwixt the neurons and the input prototype. Thus a neuron of the PPCNN model is a mapping from a specific pixel in the remote sensing epitome. Before the model is applied, the input multispectral and the Pan images should be normalized betwixt 0 and 1 so equally to simplify the parameter settings of the network. Permit PI and MIk exist the input PAN prototype and the interpolated multispectral images, respectively. The normalized ones tin be divers as:
(18)
where m indicates the kth band of the multispectral images, and the total band number is One thousand. Symbol max indicates the maximum value of all pixels. Consequently, Iyard and PN are the corresponding normalized versions of MIchiliad and PI. The architecture of the PPCNN pansharpening approach is shown in Effigy half-dozen, which consists of the following steps:
-
Interpolate the multispectral prototype to the PAN size by employing the fifty-fifty cubic kernel to acquire the input MIk [41].
-
Obtain Ik and PN co-ordinate to Equations (19) and (20).
-
Execute the histogram matching betwixt PN and Ik to obtain the matched version PNk of the PAN image.
-
Obtain the reduced-resolution version P N50 of PN by using a low-pass filter, which obeys the modulation transferring function of the given satellite [42]. Calculate Pk = PNk − P NLk, where one thousand = 1, …, K.
-
Set k = i.
-
Update the internal activity Uk later implementing the PPCNN model, where the external stimuli are Ithousand and Pk , respectively.
-
If k < K, let k = k + 1 and return to Step (6), otherwise, end the process.
-
Obtain the fusion event R past the changed normalization transformation Rg = φ × Ug .
4. Experiments
In this section, several experiments are designed to evaluate the effectiveness of the proposed model. Obviously, fusing the remote sensing images with loftier spatial resolution is more difficult; thus, the experimental datasets were selected mainly on the loftier-resolution images. In this department, the evaluation criteria for assessing the fusion results are firstly briefly reviewed. Subsequently, several experimental datasets are prepared to testify the broad capability of the PPCNN model in the fusion applications of remote sensing images. Considering the proposed PPCNN model is a laterally connected feedback network with no training, the parameter settings of PPCNN are besides given in the section.
4.1. Quality Evaluation Criteria
The experimental results need to be evaluated with quantitative statistical criteria, which focus on both the spectral and spatial quality of the fusion results. Here, the measures used include the spectral angle mapper (SAM) [43], the relative dimensionless global error in synthesis (ERGAS) [44], the quaternion index (Q4) [45,46] and the spatial correlation coefficient (SCC) [47]. They are mathematically described as follows:
(21)
(22)
(23)
where the symbol < > represents the inner product and the symbol‖‖indicates the fifty ii-norm. RMSE is the abbreviation of root mean square error. In addition, r stands for the spatial resolution ratio. The symbols σ and μ represent the variance and mean value, respectively.
SAM measures the global spectral accuracy, while ERGAS can correspond the radiometric distortion between the basis-truth image and the fusion upshot. Q4 is used for overcoming the limitations of the root mean square error (RMSE) [nine], which quantifies both spatial and spectral quality. SCC tin can evaluate the spatial correlation between two images. It should be noticed that the platonic values of SAM, ERGAS, Q4 and SCC are 0, 0, 1 and 1, respectively.
4.2. Datasets
The performance of the PPCNN model has been well tested with five datasets. All the datasets represent different landscapes and are captured by iv unlike loftier-resolution satellite sensors. The kickoff dataset is captured by the WorldView-2 sensor over an urban area of Washington. The dataset mainly consists of buildings, wood, and the river. The second dataset is the landscape of the mountainous area of Sichuan province in China. Information technology is captured by IKONOS-2 satellite sensor. The third dataset, which is captured from QuickBird satellite, indicates a suburban region of Boulder city in the Us. The fourth dataset, named Lanzhou dataset, is captured past the GF-ii sensor of China. The Lanzhou dataset represents a mountainous suburban area of Lanzhou city in northwest China, which is composed of the Yellow River, buildings and mountains. The 5th dataset is a dam area of Xinjiang province in China, which is also acquired using a GF-2 sensor. All the multispectral image of the datasets consists of four channels, i.east., greenish, blue, red, and nigh infrared bands. For convenient comparing, all the original images in the datasets are degraded based on the Wald's protocol [48] so equally to obtain the reference images. As a result, the original multispectral images can be used as the ground-truth images for accurate evaluation. Figure 7 shows the v datasets of the pansharpening experiments. The detailed parameters of all datasets are shown in Table 1.
Table 1
Experiments | Datasets | Satellites | Size | Spatial Resolution | Region |
---|---|---|---|---|---|
Pan and MS | Washington dataset | WV-2 | PAN: 2048 × 2048 MS: 512 × 512 | 0.5 m/2 k | Washington, D.C., USA |
Sichuan dataset | IKONOS-2 | PAN: 2048 × 2048 MS: 512 × 512 | 1 1000/4 yard | Sichuan, China | |
Boulder dataset | QuickBird | PAN: 2048 × 2048 MS: 512 × 512 | 0.vii m/ 2.8 m | Boulder, The states | |
Lanzhou dataset | GF-two | PAN: 2048 × 2048 MS: 512 × 512 | 0.81 k/3.24 one thousand | Lanzhou, Prc | |
Xinjiang dataset | GF-2 | PAN: 2048 × 2048 MS: 512 × 512 | 0.81 m/3.24 m | Xinjiang, Red china |
4.3. Initialize PPCNN Parameters
Since no training is needed in the PPCNN model, the settings of parameters and matrices are discussed hither. Among which, linking synapses K and W with the matrix [0.707, ane, 0.707; i, 0, 1; 0.707, 1, 0.707] are obtained by means of the Euclidean distance between two neurons. VEast is set to exist a big value to ensure that each neuron will be stimulated only once. The initial value of iteration number n is 1. The selection of FiveF and VL are analyzed in Figure eight. They are discussed equally the PPCNN is carried out with the Washington dataset. Effigy 8 gives the SAM, ERGAS, Q4 and SCC results of different VF and FiveL , when αEastward equals to 1.1. Information technology is found that VL offers little influence on the results. Another observation is that Q4 and SCC get the maximum values when the VF is larger than 0.1. SAM and ERGAS get the minimum values if the VF is larger than 0.ane. Hence, the 5F could be set as an arbitrary value when it is larger than 0.one. Figure 9 shows the SAM, ERGAS, Q4 and SCC curves of different αE . We tin can see from Effigy 9 that the optimal value of αE is 1.ane.
5. Experimental Results and Discussions
In this section, the fusion experiments between the Pan image and the multispectral images are carried out. The performance of the proposed model will be well assessed, and the discussions will be made in detail. In the experiments, a couple of classical and land-of-the-art algorithms are presented as comparative methods, i.e., the GS method [17], the PCA method [xv], the Brovey transform (BT) method [49], the IHS method [13], the ATWT method [20], the additive wavelet luminance proportional (AWLP) method [47], the CBD method [9], the revisited AWLP (RAWLP) method [50], the full calibration regression (FSR) method [51] and the MOF method [22]. The pansharpening results obtained from the PPCNN algorithm and other methods are presented and discussed using five dissimilar datasets.
The experimental results with the Washington dataset are shown in Effigy 10. Figure xa shows the depression-resolution multispectral image of the Washington dataset. Effigy 10b gives the reference ground-truth multispectral image. The fusion outcome obtained from the proposed PPCNN method is shown in Figure 10c. The resulting fused multispectral images produced from GS, PCA, BT, IHS, ATWT, AWLP, CBD, MOF, RAWLP and FSR methods are shown in Figure 5d–thousand, respectively. From the Washington dataset obtained by WorldView-ii, it tin can exist noticed that the proposed PPCNN method, the ATWT method, the AWLP method, the CBD method, the MOF method, the RAWLP method and the FSR method perform better in spatial details and spectral preservation, while the contours in the PCA method and the BT method are blurred and not sharp enough. Wrong colors of modest roof are obtained by the BT and the IHS methods. Tabular array two provides the quantitative comparison results for the Washington dataset, where the best upshot is highlighted in assuming. It can be found that the PPCNN arroyo performs better than the others with less spectral distortion and better item preservation.
Table 2
Criteria | PPCNN | GS | PCA | BT | IHS | ATWT |
Q4 | 0.8884 | 0.7822 | 0.7438 | 0.7764 | 0.7841 | 0.8626 |
SAM (°) | 7.7527 | 8.4126 | ix.6313 | 8.1936 | viii.6529 | 7.9265 |
ERGAS | 4.9416 | half-dozen.5617 | 7.5797 | vi.7480 | six.6710 | 5.5633 |
SCC | 0.8375 | 0.8349 | 0.7933 | 0.8372 | 0.8175 | 0.8317 |
Criteria | AWLP | CBD | MOF | RAWLP | FSR | |
Q4 | 0.8600 | 0.8696 | 0.8651 | 0.8731 | 0.8795 | |
SAM (°) | 7.9247 | 8.5065 | 7.9248 | 7.7967 | 8.3575 | |
ERGAS | v.7873 | v.4875 | 5.4889 | 5.3147 | v.0657 | |
SCC | 0.8187 | 0.7812 | 0.8309 | 0.8369 | 0.8271 |
Figure 11 shows the pansharpening results of Sichuan dataset. Equally shown in Effigy 11, the colors of the BT and IHS methods are not correctly synthesized. The AWLP method looks a little blurry. CBD method shows nice right colors every bit a whole, but another information computed by the method is redundantly added, especially in white expanse of Figure 11j. It has also been found that visual quality is generally adequate for other 7 methods. Tabular array 3 demonstrates that the PPCNN method obtains the all-time fusion result for Sichuan dataset.
Table iii
Criteria | PPCNN | GS | PCA | BT | IHS | ATWT |
Q4 | 0.8959 | 0.8354 | 0.8368 | 0.7086 | 0.5658 | 0.8772 |
SAM (°) | i.3857 | 1.7843 | 1.7942 | 2.7454 | four.2208 | 1.5599 |
ERGAS | 1.1769 | 1.4884 | ane.4911 | one.8771 | 2.8338 | 1.2873 |
SCC | 0.9552 | 0.9525 | 0.9520 | 0.9085 | 0.7201 | 0.9509 |
Criteria | AWLP | CBD | MOF | RAWLP | FSR | |
Q4 | 0.8714 | 0.8762 | 0.8810 | 0.8917 | 0.8926 | |
SAM (°) | ii.9800 | one.4627 | ane.5765 | 1.3983 | i.3927 | |
ERGAS | ane.7519 | 1.3027 | 1.2960 | i.1865 | 1.1819 | |
SCC | 0.9343 | 0.9515 | 0.9476 | 0.9540 | 0.9542 |
From Figure 12, the experimental results for the Boulder dataset also show that the BT and IHS methods produce more spectral distortion, not simply in the farmland and tree area, simply also in the white roof surface area. For the Boulder dataset, wrong colors of small objects are also obtained using the CBD method, such as the false red lines in the white roof. Every bit shown in Table 4, the PPCNN method outperforms others in the quantitative comparison.
Table 4
Criteria | PPCNN | GS | PCA | BT | IHS | ATWT |
Q4 | 0.9126 | 0.8511 | 0.8513 | 0.8291 | 0.8084 | 0.9057 |
SAM (°) | 0.9747 | 1.3354 | i.5113 | 1.1640 | one.4448 | 1.0470 |
ERGAS | 0.8049 | 1.0992 | 1.0346 | 1.1104 | one.2317 | 0.8957 |
SCC | 0.9557 | 0.9411 | 0.9411 | 0.9467 | 0.9328 | 0.9432 |
Criteria | AWLP | CBD | MOF | RAWLP | FSR | |
Q4 | 0.9050 | 0.8972 | 0.9026 | 0.9062 | 0.9097 | |
SAM (°) | 1.2062 | one.0305 | 1.1257 | ane.1903 | i.1528 | |
ERGAS | 0.9731 | ane.0052 | 0.9763 | 0.9312 | 0.9441 | |
SCC | 0.9346 | 0.9369 | 0.9465 | 0.9393 | 0.9392 |
For the experiments with the Lanzhou dataset, it is obvious from Figure thirteen that the large spectral distortion is introduced by the GS, PCA, BT and IHS methods, specially in the Yellow River area of Lanzhou city. In addition, part of the small island is missing in Figure 13j obtained by CBD methods. Another ascertainment is that the tree belt looks a picayune blur in Figure 13j. Table 5 compares the PPCNN algorithm with others through the Lanzhou dataset. It shows that the proposed approach outperforms the others in the quantitative evaluation.
Tabular array v
Criteria | PPCNN | GS | PCA | BT | IHS | ATWT |
Q4 | 0.9107 | 0.8230 | 0.7763 | 0.8126 | 0.8097 | 0.8980 |
SAM (°) | 1.4304 | i.7992 | ii.3502 | 1.4307 | i.6756 | 1.4545 |
ERGAS | 1.6697 | two.2794 | 2.8436 | 2.2208 | two.2514 | 1.7911 |
SCC | 0.8944 | 0.8713 | 0.8516 | 0.8717 | 0.8777 | 0.8844 |
Criteria | AWLP | CBD | MOF | RAWLP | FSR | |
Q4 | 0.8961 | 0.7957 | 0.8835 | 0.9000 | 0.8982 | |
SAM (°) | 1.4480 | ii.2117 | 1.4583 | i.6917 | one.4550 | |
ERGAS | 1.7683 | 3.2100 | 1.9758 | 1.7373 | 1.7963 | |
SCC | 0.8771 | 0.8021 | 0.8829 | 0.8822 | 0.8878 |
Figure xiv shows the pansharpening results of the Xinjiang dataset. Since the Xinjiang dataset does not take a lot of loftier-resolution tiny textures, the visual quality is acceptable for all algorithms. Nevertheless, Tabular array six demonstrates that the PPCNN algorithm also performs the best.
Table 6
Criteria | PPCNN | GS | PCA | BT | IHS | ATWT |
Q4 | 0.9253 | 0.8650 | 0.8479 | 0.8514 | 0.8621 | 0.8956 |
SAM (°) | 0.9940 | 1.2560 | 1.3230 | 1.0468 | ane.0672 | 1.1253 |
ERGAS | one.1784 | 1.6340 | 1.7619 | i.5944 | 1.6054 | 1.3765 |
SCC | 0.9195 | 0.8814 | 0.8778 | 0.8820 | 0.8918 | 0.8904 |
Criteria | AWLP | CBD | MOF | RAWLP | FSR | |
Q4 | 0.8836 | 0.8479 | 0.8802 | 0.9181 | 0.8864 | |
SAM (°) | 1.1277 | one.5497 | 1.1718 | 1.0129 | 1.1674 | |
ERGAS | one.3733 | 2.0711 | i.5450 | ane.2622 | ane.4706 | |
SCC | 0.8830 | 0.8444 | 0.8896 | 0.9142 | 0.8847 |
In conclusion, we tested the effectiveness of the PPCNN method with v datasets of dissimilar landscapes and sources. The experimental results of all datasets are summarized in Figure 15. In general, the results of the BT and IHS methods exhibited more spectral distortion. In some cases, GS and PCA methods are not good at spectral preservation either. More than specifically, the contours of small objects obtained by these 4 methods and the AWLP method are sometimes blurred and not precipitous plenty. In addition, the CBD method produces spectral distortion in some small objects. Hence, the PPCNN, ATWT, MOF, RAWLP and FSR methods obtain good visual results. In greater item, information technology can be noticed that the proposed PPCNN method produces images with better spatial detail quality and spectral quality than other methods co-ordinate to the quantitative comparison.
vi. Conclusions
The newspaper presents a novel PPCNN model and applies information technology to pansharpening approaches. The PPCNN model has two external stimuli rather than a single ane in the standard PCNN, which guarantees it more convenient for fusion tasks. In improver, the internal activity of PPCNN is designed to have the role of details injection performance, which makes information technology easier to reserve the spectral allegiance. Five datasets with different characteristics, caused by four different loftier-resolution sensors, were used to evaluate the model. The efficient performance of the PPCNN model is thoroughly examined through urban, suburban, mountainous and other complex landform datasets, which demonstrates that the PPCNN approach performs better with regard to both detail and spectral preservation.
In fact, the PPCNN model imitates the visual cortex neurons to transmit the input images to produce the synchronous electrical pulses with no grooming. On the other mitt, deep learning models, such as CNN and RNN, imitate the mechanism of brain by training. If the output pulses of the PPCNN model are treated equally the input of the CNN model, nosotros believe that helpful results will exist obtained. Consequently, the PPCNN based training model will be the focus of our future work. Another farther interesting investigation looks toward the other image processing applications of PPCNN. Nosotros will investigate these problems in future work.
Acknowledgments
The authors are grateful to the editor and bearding reviewers for their helpful and valuable suggestions. We too appreciate Jie Li for her back up and help.
Author Contributions
All the authors have contributed substantially to the manuscript. X.L. and H.Y. proposed the methodology. 10.Fifty. and Due west.10. performed the experiments and software. 50.Yard. and Y.T. wrote the newspaper. H.Y. and Y.T. analyzed the data. All authors take read and agreed to the published version of the manuscript.
Funding
This research was jointly funded by National Key R&D Program of China (No. 2017YFB0504201), National Natural Science Foundation of Prc (Nos. 41861055, 41671447 and 41761082), Communist china Postdoctoral Science Foundation (No. 2019M653795), and lzjtu EP Program (No. 201806).
Conflicts of Interest
The authors declare no conflict of involvement.
References
ane. Huang B., Zhao B., Song Y. Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery. Remote Sens. Environ. 2018;214:73–86. doi: 10.1016/j.rse.2018.04.050. [CrossRef] [Google Scholar]
2. Fu H., Zhou T., Sun C. Object-based shadow alphabetize via illumination intensity from high resolution satellite images over urban areas. Sensors. 2020;20:1077. doi: x.3390/s20041077. [PMC free article] [PubMed] [CrossRef] [Google Scholar]
3. Frick A., Tervooren Southward. A framework for the long-term monitoring of urban green volume based on multi-temporal and multi-sensoral remote sensing data. J. Geovis. Spat. Anal. 2019;3:6–sixteen. doi: ten.1007/s41651-019-0030-5. [CrossRef] [Google Scholar]
4. Wang M., Zhang X., Niu X., Wang F., Zhang X. Scene nomenclature of loftier-resolution remotely sensed epitome based on resnet. J. Geovis. Spat. Anal. 2019;3:one–ix. doi: ten.1007/s41651-019-0039-9. [CrossRef] [Google Scholar]
5. Bovolo F., Bruzzone L., Capobianco Fifty., Garzelli A., Marchesi South. Analysis of the furnishings of pansharpening in change detection on VHR images. IEEE Geosci. Remote Sens. Lett. 2010;vii:53–57. doi: 10.1109/LGRS.2009.2029248. [CrossRef] [Google Scholar]
6. Mohammadzadeh A., Tavakoli A., Zoej M.J.V. Road extraction based on fuzzy logic and mathematical morphology from pan-sharpened IKONOS images. Photogramm. Rec. 2006;21:44–lx. doi: 10.1111/j.1477-9730.2006.00353.ten. [CrossRef] [Google Scholar]
7. Gašparović M., Jogun T. The effect of fusing Watch-2 bands on country-encompass classification. Int. J. Remote Sens. 2018;39:822–841. doi: x.1080/01431161.2017.1392640. [CrossRef] [Google Scholar]
eight. Laporterie-Déjean F., Boissezon H., Flouzat G., Lefèvre-Fonollosaa M. Thematic and statistical evaluations of five panchromatic/multispectral fusion methods on faux PLEIADES-HR images. Inf. Fusion. 2005;vi:193–212. doi: 10.1016/j.inffus.2004.06.006. [CrossRef] [Google Scholar]
ix. Vivone Thou., Alparone L., Chanussot J., Mura M.D., Garzelli A. A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 2015;53:2565–2586. doi: 10.1109/TGRS.2014.2361734. [CrossRef] [Google Scholar]
10. Amro I., Mateos J., Vega M., Molina R., Katsaggelos A.K. A survey of classical methods and new trends in pansharpening of multispectral images. EURASIP J. Adv. Signal Process. 2011;2011:1–22. doi: 10.1186/1687-6180-2011-79. [CrossRef] [Google Scholar]
11. Thomas C., Ranchin T., Wald L., Chanussot J. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Trans. Geosci. Remote Sens. 2008;46:1301–1312. doi: x.1109/TGRS.2007.912448. [CrossRef] [Google Scholar]
12. Wang Z., Ziou D., Armenakis C., Li D., Li Q. A comparative analysis of epitome fusion methods. IEEE Trans. Geosci. Remote Sens. 2005;43:1391–1402. doi: 10.1109/TGRS.2005.846874. [CrossRef] [Google Scholar]
13. Tu T.Thou., Su S.C., Shyu H.C., Huang P.S. A new expect at IHS similar epitome fusion methods. Inf. Fusion. 2001;2:177–186. doi: 10.1016/S1566-2535(01)00036-7. [CrossRef] [Google Scholar]
14. Carper W., Lillesand T., Kiefer R. The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral epitome data. Photogramm. Eng. Remote Sens. 1990;56:459–467. [Google Scholar]
15. Psjr C., Sides S.C., Anderson J.A. Comparison of three unlike methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogramm. Eng. Remote Sens. 1991;57:265–303. [Google Scholar]
16. Chavez P.Due south., Kwarteng A.Y. Extracting spectral contrast in Landsat thematic mapper image data using selective principal component assay. Photogramm. Eng. Remote Sens. 1989;55:339–348. [Google Scholar]
17. Laben C.A., Brower B.V. Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-Sharpening. 6011875. U.Southward. Patent. 2000 January 4;
18. Mallat S.Grand. A Theoryf Multiresolution Indicate Decomposition: The Wavelet Representation. IEEE Reckoner Society; Washington, DC, USA: 1989. [Google Scholar]
nineteen. Nason G.P., Silverman B.W. The Stationary Wavelet Transform and Some Statistical Applications. Springer; New York, NY, USA: 1995. [Google Scholar]
twenty. Vivone G., Restaino R., Mura M.D., Licciardi G., Chanussot J. Contrast and error-based fusion schemes for multispectral epitome pansharpening. IEEE Geosci. Remote Sens. Lett. 2013;eleven:930–934. doi: 10.1109/LGRS.2013.2281996. [CrossRef] [Google Scholar]
21. Burt P.J., Adelson Due east.H. The Laplacian pyramid as a meaty epitome code. IEEE Trans. Commun. 2003;31:532–540. doi: 10.1109/TCOM.1983.1095851. [CrossRef] [Google Scholar]
22. Restaino R., Vivone G., Mura M.D., Chanussot J. Fusion of multispectral and panchromatic images based on morphological operators. IEEE Trans. Image Procedure. 2016;25:2882–2895. doi: ten.1109/TIP.2016.2556944. [PubMed] [CrossRef] [Google Scholar]
23. Scarpa G., Vitale S., Cozzolino D. Target-adaptive cnn-based pansharpening. IEEE Trans. Geosci. Remote Sens. 2018;56:1–15. doi: ten.1109/TGRS.2018.2817393. [CrossRef] [Google Scholar]
24. Masi G., Cozzolino D., Verdoliva L., Scarpa G. Pansharpening by convolutional neural networks. Remote Sens. 2016;8:594. doi: ten.3390/rs8070594. [CrossRef] [Google Scholar]
25. Liu J., Ma J., Fei R., Li H., Zhang J. Enhanced back-projection as postprocessing for pansharpening. Remote Sens. 2019;11:712. doi: 10.3390/rs11060712. [CrossRef] [Google Scholar]
26. Li H., Jing Fifty., Tang Y., Ding H. An improved pansharpening method for misaligned panchromatic and multispectral data. Sensors. 2018;xviii:557. doi: x.3390/s18020557. [PMC free article] [PubMed] [CrossRef] [Google Scholar]
27. Aiazzi B., Alparone L., Baronti S. Signal and Paradigm Processing for Remote Sensing. CRC Press; Boca Raton, FL, Us: 2012. 20-Five Years of Pansharpening: A disquisitional Review and New Developments; pp. 533–548. [Google Scholar]
28. Wang N., Jiang W., Lei C., Qin S., Wang J. A robust paradigm fusion method based on local spectral and spatial correlation. IEEE Geosci. Remote Sens. Lett. 2014;11:454–458. doi: 10.1109/LGRS.2013.2265915. [CrossRef] [Google Scholar]
29. Ma Y., Zhan K., Wang Z. Applications of Pulse-Coupled Neural Networks. College Education Printing; Beijing, China: 2010. [Google Scholar]
30. Wang Z., Ma Y. Medical image fusion using m-PCNN. Inf. Fusion. 2008;ix:176–185. doi: 10.1016/j.inffus.2007.04.003. [CrossRef] [Google Scholar]
31. Zhu South., Wang 50., Duan S. Memristive pulse coupled neural network with applications in medical image processing. Neurocomputing. 2017;227:149–157. doi: ten.1016/j.neucom.2016.07.068. [CrossRef] [Google Scholar]
32. Dong Z., Lai C., Qi D., Zhao 10., Li C. A general memristor-based pulse coupled neural network with variable linking coefficient for multi-focus image fusion. Neurocomputing. 2018;308:172–183. doi: ten.1016/j.neucom.2018.04.066. [CrossRef] [Google Scholar]
33. Huang C., Tian G., Lan Y., Peng Y., Ng E.Y.K. A new pulse coupled neural network (pcnn) for brain medical paradigm fusion empowered by shuffled frog leaping algorithm. Front end. Neurosci. 2019;13:1–10. doi: 10.3389/fnins.2019.00210. [PMC free article] [PubMed] [CrossRef] [Google Scholar]
34. Cheng B., Jin Fifty., Li G. Infrared and visual image fusion using LNSST and an adaptive dual-channel PCNN with triple-linking strength. Neurocomputing. 2018;310:135–147. doi: 10.1016/j.neucom.2018.05.028. [CrossRef] [Google Scholar]
35. Ma J., Xu H., Jiang J., Mei 10., Zhang 10. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 2020;29:4980–4995. doi: ten.1109/TIP.2020.2977573. [PubMed] [CrossRef] [Google Scholar]
36. Ma J., Yu W., Liang P., Li C., Jiang J. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion. 2019;48:11–26. doi: 10.1016/j.inffus.2018.09.004. [CrossRef] [Google Scholar]
37. Ma J., Liang P., Yu Westward., Chen C., Guo X. Infrared and visible image fusion via item preserving adversarial learning. Inf. Fusion. 2020;54:85–98. doi: 10.1016/j.inffus.2019.07.005. [CrossRef] [Google Scholar]
38. Shi C., Miao Q.G., Xu P.F. A novel algorithm of remote sensing epitome fusion based on Shearlets and PCNN. Neurocomputing. 2013;117:47–53. [Google Scholar]
39. Johnson J.L., Padgett 1000.L. PCNN models and applications. IEEE Trans. Neural Netw. 1999;10:480–498. doi: 10.1109/72.761706. [PubMed] [CrossRef] [Google Scholar]
40. Lindblad T., Kinser J.M. Image Processing Using Pulse-Coupled Neural Networks. Springer; Berlin/Heidelberg, Federal republic of germany: 2005. [Google Scholar]
41. Aiazzi B., Baronti S., Selva M., Alparone L. Bi-cubic interpolation for shift-gratis pan-sharpening. ISPRS J. Photogramm. Remote Sens. 2013;86:65–76. doi: 10.1016/j.isprsjprs.2013.09.007. [CrossRef] [Google Scholar]
42. Aiazzi B., Alparone L., Baronti S., Garzelli A. MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogramm. Eng. Remote Sens. 2006;72:591–596. doi: 10.14358/PERS.72.5.591. [CrossRef] [Google Scholar]
43. Goetz A., Boardman W., Yunas R. Discrimination among semi-arid mural endmembers using the Spectral Bending Mapper (SAM) algorithm; Proceedings of the Summaries tertiary Annual JPL Airborne Geoscience Workshop; Pasadena, CA, USA. i–five June 1992; pp. 147–149. [Google Scholar]
44. Wald L. Data Fusion: Definitions and Architectures: Fusion of Images of Different Spatial Resolutions. Presses Des MINES; Paris, France: 2002. [Google Scholar]
45. Alparone L., Baronti Due south., Garzelli A., Nencini F. A global quality measurement of pan-sharpened multispectral imagery. IEEE Geosci. Remote Sens. Lett. 2004;1:313–317. doi: 10.1109/LGRS.2004.836784. [CrossRef] [Google Scholar]
46. Garzelli A., Nencini F. Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geosci. Remote Sens. Lett. 2009;6:662–665. doi: 10.1109/LGRS.2009.2022650. [CrossRef] [Google Scholar]
47. Otazu X., Gonzalez-Audicana One thousand., Fors O., Nunez J. Introduction of sensor spectral response into image fusion methods. Awarding to wavelet-based methods. IEEE Trans. Geosci. Remote Sens. 2005;43:2376–2385. doi: 10.1109/TGRS.2005.856106. [CrossRef] [Google Scholar]
48. Wald L., Ranchin T., Mangolini M. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogramm. Eng. Remote Sens. 1997;63:691–699. [Google Scholar]
49. Gillespie A.R., Kahle A.B., Walker R.Eastward. Color enhancement of highly correlated images. II. Channel ratio and "chromaticity" transformation techniques. Remote Sens. Environ. 1987;22:343–365. doi: 10.1016/0034-4257(87)90088-5. [CrossRef] [Google Scholar]
50. Vivone Thou., Luciano A., Andrea One thousand., Simone L. Fast reproducible pansharpening based on instrument and acquisition modeling: AWLP revisited. Remote Sens. 2019;11:2315. doi: 10.3390/rs11192315. [CrossRef] [Google Scholar]
51. Vivone K., Rocco R., Jocelyn C. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Trans. Paradigm Process. 2018;27:3418–3431. doi: 10.1109/TIP.2018.2819501. [PubMed] [CrossRef] [Google Scholar]
Articles from Sensors (Basel, Switzerland) are provided here courtesy of Multidisciplinary Digital Publishing Plant (MDPI)
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7294424/
0 Response to "Computational Mechanisms of Pulse-coupled Neural Networks a Comprehensive Review"
Post a Comment