Failed to retrieve credentials from EC2 Instance Metadata Service is misleading

[jamiuaz]

Description

When make GetCredentials() while there is already existing request with TryEnterWriteLock, it fails with Failed to retrieve credentials from EC2 Instance Metadata Service instead should be something like There is request to FetchCredentials in progress

Reproduction Steps

Logs

Environment

• SDK Version: "AWSSDK.RDS" Version="3.7.6.8"
• Package Version: AWSSDK.RDS
• OS Info: EC2 - Microsoft Windows Server 2019 Datacenter
• Build Environment: Visual Studio 2019
• Targeted .NET Platform: .NET CORE 3.1

Resolution

---

This is a 🐛 bug-report

[aaoswal]

Hi @jamiuaz,
Could you please share a sample code and reproduction steps? It would help analyze the issue faster.

Thanks,
Anish.

[jamiuaz]

Hi @aaoswal

To reproduce with aws sdk class : DefaultInstanceProfileAWSCredentials.cs

1. I created a .net core console app.
2. Make concurrent call to GetCredentials with 100 threads concurrently.

using Microsoft.Extensions.Configuration;
using NLog;
using NLog.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;

namespace AwsInstanceMetadataDebug
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
            .Build();      

            LogManager.Configuration = new NLogLoggingConfiguration(config.GetSection("NLog"));
            var logger = LogManager.GetCurrentClassLogger();

            var tasks = new List<Task>();

            var credential = Amazon.Runtime.DefaultInstanceProfileAWSCredentials.Instance;

            for (var i = 0; i < 100; i++)
            {
                var task = Task.Factory.StartNew(() =>
                {
                    try
                    {
                        var cred = credential.GetCredentials();
                        logger.Info($"key={cred.AccessKey}, secrete={cred.SecretKey}");
                    }
                    catch (Exception ex)
                    {
                        logger.Error(ex, ex.Message);
                    }
                });
                tasks.Add(task);
            }

            Task.WaitAll(tasks.ToArray());

            Console.ReadKey();
        }

    }
}

To reproduce with focus on TryEnterWriteLock

1. I created a .net core console app.
2. Simulate the GetCredentials with TryEnterWriteLock.
3. Make concurrent call to GetCredentials with 2 threads: first processing delayed than second thread.

using System;
using System.Threading;
using System.Threading.Tasks;

namespace IssueReproduce
{
    class DefaultInstanceProfileAWSCredentialsIssue
    {
        private static readonly TimeSpan credentialsLockTimeout = TimeSpan.FromSeconds(20);
        public ReaderWriterLockSlim credentialsLock = new ReaderWriterLockSlim(); // Lock to control getting credentials across multiple threads.

        public string GetCredentials(int delay)
        {
            string result = string.Empty;

            // If there's no credentials cached, hit IMDS directly. Try to acquire write lock.
            if (credentialsLock.TryEnterWriteLock(credentialsLockTimeout))
            {
                try
                {
                    Thread.Sleep(TimeSpan.FromMinutes(delay));
                    result = "returned metadata";
                }
                finally
                {
                    credentialsLock.ExitWriteLock();
                }
            }

            if (string.IsNullOrWhiteSpace(result))
                throw new Exception("Failed to retrieve credentials from EC2 Instance Metadata Service.");

            return result;
        }

    }

    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                var issue = new DefaultInstanceProfileAWSCredentialsIssue();
                var first = Task.Factory.StartNew(() => { issue.GetCredentials(3); });
                var second = Task.Factory.StartNew(() => { issue.GetCredentials(6); });
                Task.WaitAll(first, second);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }

            Console.ReadKey();
        }
    }
}

[iarekk]

Folks, any updates here? Seeing some intermittent AWS credential fetch failures when trying to upgrade to the newest AWS sdk

[TandukarRajan]

Is this issue fixed in new version? We are constantly getting this issue after upgrading to 3.7.10.5