The accurate estimation of Surface Solar Irradiance (SSI) is crucial in domains as diverse as climatology, solar energy, agriculture, and architecture. Traditional SSI estimation methods are primarily based on physical models and cloud-index models. These approaches rely on the Independent Pixel Approximation (IPA) and neglect the intricate inter-pixel interactions, 3D effects of clouds, or parallax effects. This reliance on IPA and oversight of spatial dynamics could introduce limitations to traditional methods. These limitations are expected to increase with the advent of third-generation geostationary satellites like the GOES series, which offer enhanced spatial resolution. This work introduces a deep learning framework leveraging the increased spectral, spatial, and temporal resolution offered by third-generation geostationary satellites, without IPA, to improve SSI estimation.We developed a method using convolutional neural networks (CNNs) to analyze large satellite imagery, high-dimensional in spatial, spectral, and temporal domains, using contextual and multispectral image for SSI estimation. A comprehensive dataset, combining GOES-16 satellite imagery with 5-min global horizontal irradiance (GHI) in-situ measurements from 31 pyranometric stations in the U.S.A. over three years, was constructed and used for model training and validation, allowing for a direct comparison with PSM3, a state-of-the-art physical SSI-satellite-retrieval model from NREL. Our approach combines CNNs for image analysis and fully connected neural networks (FCNs) for processing tabular auxiliary data such as solar angles and positions, exploring various data fusion techniques. We thoroughly assess the model performance using a broad set of metrics, across various conditions and test stations, as well as the influence of varying image sizes on performance.Results demonstrate the potential of deep learning to outperform traditional models like PSM3 with traditional comparison metrics, especially under cloudy conditions, showing a 25% RMSE improvement. Our analysis highlights the importance of spatial context and the influence of image size in model performance, challenging the adequacy of IPA in traditional methods. A significant improvement is the effect of rotating input images, which substantially enhanced test performance and spatial generalization.For 5-min GHI estimation, our models achieved a test RMSE of 80 W/m^2, compared to 97 W/m^2 for PSM3, and demonstrated their robustness across diverse evaluation metrics, in most test stations and under various sky conditions. However, the mixed performance in MBE across all sky conditions, as well as other metrics under clear sky conditions and at specific test stations, indicate areas for further improvements in the representativity of the underlying physical process of SSI.While initial results are promising, further research is needed to refine model architectures and enhance generalization capabilities across different geographical locations Exploring physically informed and probabilistic deep learning methods could be a valuable direction for future research to enhance the spatial generalization, reliability, and interpretability of SSI estimation with deep learning.