Short-term forecasting is key to the safe, reliable, and economic operation of modern power systems. As the majority of modern forecasting tools are purely data-driven, their performance relies heavily on the quality and availability of data. In this work, we examine forecasting when a subset of features used during model training becomes unavailable (deleted or missing) in an operational setting, a subject that has been largely overlooked by previous works. Several reasons could lead to feature deletion, including malicious data-integrity attacks, network latency, and equipment malfunctions, among others. We leverage tools from robust optimization and machine learning and formulate a linear regression model that is optimally resilient to the deletion of features at test time. The robust counterpart of the proposed model is a linear program whose size grows polynomially with the number of training observations and the number of features; we further provide a decomposition algorithm based on the alternating direction method of multipliers to deal with large problem instances that are typically found in energy forecasting applications. We further extend to the case of Probabilistic Forecasting by robustifying the standard linear quantile regression model. To validate empirically the proposed approach, we examine several prevalent forecasting practices in power systems, namely electricity prices, load, wind production, and solar production forecasting. We compare against regularized and randomization-based models and benchmark their performance for the case of feature deletion. The results show that the proposed solution successfully mitigates the adverse effects of missing features, leading to the lowest overall performance degradation. Further, it successfully hedges against the most adverse scenario of deleting the most important feature from the test set. The results persist both for point and probabilistic forecasts and across the different series. Overall, this work highlights the benefits of leveraging robust optimization and provides a new perspective on how to deal with feature uncertainty in energy forecasting applications.