Failed jobs do not become expired to allow you to re-queue them without any
time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete)
attribute to delete them automatically.
An exception occurred during performance of the job. Fewer details...
10/07/2026 08:05:36
Oracle.ManagedDataAccess.Client.OracleException
Connection request timed out
Oracle.ManagedDataAccess.Client.OracleException (0x80004005): Connection request timed out
at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionStringcsWithDiffOrNewPwd, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionStringcsWithNewPassword, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionStringcs, PMconPM, ConnectionStringpmCS, SecureStringsecuredPassword, SecureStringsecuredProxyPassword, OracleConnectionconnRefForCriteria)
at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.CainiaoEventPusher.Execute(PerformContextcontext, WebhookCustomercustomer, StringcourierDatabaseConnectionString) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/CainiaoEventPusher.cs:line 24
at EP.BackgroundJobs.Engine.Jobs.CainiaoWebhookJob.ProcessCustomer(PerformContextcontext, StringcustomerName, StringaccountNumber, WebhookCustomercustomer, StringcourierDatabaseConnectionString) in /src/EP.BackgroundJobs.Engine/Jobs/CainiaoWebhookJob.cs:line 37
An exception occurred during performance of the job. More details...
10/07/2026 08:05:36
Oracle.ManagedDataAccess.Client.OracleException
Connection request timed out
Oracle.ManagedDataAccess.Client.OracleException (0x80004005): Connection request timed out
at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionStringcsWithDiffOrNewPwd, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionStringcsWithNewPassword, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionStringcs, PMconPM, ConnectionStringpmCS, SecureStringsecuredPassword, SecureStringsecuredProxyPassword, OracleConnectionconnRefForCriteria)
at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.EthiopianConsulateEventPusher.Execute(PerformContextcontext, WebhookCustomercustomer, StringcourierDatabaseConnectionString) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/EthiopianConsulateEventPusher.cs:line 50
An exception occurred during performance of the job. More details...
10/07/2026 08:15:36
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookDeemJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookDeemJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during performance of the job. More details...
10/07/2026 08:29:21
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during performance of the job. More details...
10/07/2026 08:33:21
Oracle.ManagedDataAccess.Client.OracleException
Connection request timed out
Oracle.ManagedDataAccess.Client.OracleException (0x80004005): Connection request timed out
at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionStringcsWithDiffOrNewPwd, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionStringcsWithNewPassword, BooleanbGetForApp, OracleConnectionconnRefForCriteria, StringaffinityInstanceName, BooleanbForceMatch)
at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionStringcs, PMconPM, ConnectionStringpmCS, SecureStringsecuredPassword, SecureStringsecuredProxyPassword, OracleConnectionconnRefForCriteria)
at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.BrandsForLessEventPusher.Execute(PerformContextcontext, WebhookCustomercustomer, StringcourierDatabaseConnectionString) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/BrandsForLessEventPusher.cs:line 64
An exception occurred during performance of the job. More details...
10/07/2026 08:37:06
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during performance of the job. More details...
10/07/2026 08:39:21
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during performance of the job. More details...
10/07/2026 08:44:21
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
An exception occurred during performance of the job. More details...
10/07/2026 08:48:15
System.Net.Http.HttpRequestException
The SSL connection could not be established, see inner exception.
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolTokenmessage, ExceptionDispatchInfoexception)
at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptionssslAuthenticationOptions)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapteradapter, BooleanreceiveFirst, Byte[]reAuthenticationData, BooleanisApm)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptionssslOptions, HttpRequestMessagerequest, Booleanasync, Streamstream, CancellationTokencancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptionssslOptions, HttpRequestMessagerequest, Booleanasync, Streamstream, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessagerequest)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellation(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessagerequest, CancellationTokencancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessagerequest, CancellationTokencancellationToken)
at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessagerequest, CancellationTokencancellationToken)
at System.Net.Http.HttpClientHandler.Send(HttpRequestMessagerequest, CancellationTokencancellationToken)
at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessagerequest, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.Send(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokencancellationToken)
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.PTSEventPusher.GetToken(PerformContextcontext, Stringapitokenurl, Stringusername, Stringpassword) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/PTSEventPusher.cs:line 325
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.PTSEventPusher.SendCustomerUpdate(OracleConnectionconnection, PerformContextcontext, WebhookCustomercustomer, DataTablerecords) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/PTSEventPusher.cs:line 122
at EP.BackgroundJobs.Engine.BusinessLogic.Implementations.PTSEventPusher.Execute(PerformContextcontext, WebhookCustomercustomer, StringcourierDatabaseConnectionString) in /src/EP.BackgroundJobs.Engine/BusinessLogic/Implementations/PTSEventPusher.cs:line 54
An exception occurred during performance of the job. More details...
10/07/2026 08:50:51
Hangfire.Storage.DistributedLockTimeoutException
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'WebhookJob.ProcessCustomer' resource.
at Hangfire.InMemory.InMemoryConnection.AcquireDistributedLock(Stringresource, TimeSpantimeout)
at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContextfilterContext)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1tuple)
at Hangfire.Profiling.SlowLogProfiler.InvokeMeasured[TInstance,TResult](TInstanceinstance, Func`2action, Stringmessage)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfilerprofiler, TInstanceinstance, Action`1action, Stringmessage)
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)